[00:00] Crash dump logging enabled: /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/comfy-test-logs/SAM3-2028/dev/linux-cpu/crash_dump.log [00:00] [00:00] [1/5] SYNTAX [00:00] ---------------------------------------- [00:00] [DEBUG] server=None, server_url=None, api=None [00:00] Found pyproject.toml (modern format) [00:00] Found requirements.txt (legacy format) [00:00] Unicode check: OK (all characters cp1252-safe) [00:00] Forbidden patterns check: OK [00:00] [SYNTAX] PASSED [00:00] [00:00] [2/5] INSTALL [00:00] ---------------------------------------- [00:00] [DEBUG] server=None, server_url=None, api=None [00:00] Setting up ComfyUI... [00:00] Creating virtual environment... [00:00] Running: uv venv /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/.venv --python 3.10 [00:01] Cloning ComfyUI (latest)... [00:01] Running: git clone --depth 1 https://github.com/comfyanonymous/ComfyUI.git /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI [00:02] Installing ComfyUI requirements... [00:02] Running: uv pip install --python /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/.venv/bin/python -r /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/requirements.txt [00:23] Installing custom node... [00:23] Copying ComfyUI-SAM3 to custom_nodes/... [00:23] Installing node requirements... [00:23] Running: uv pip install --python /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/.venv/bin/python -r /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/requirements.txt [00:23] Running install.py... [00:23] Running: /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/.venv/bin/python /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/install.py [00:24] [00:24] Installing 2 node dependencies... [00:24] Cloning ComfyUI-VideoHelperSuite... [00:24] Installing requirements for ComfyUI-VideoHelperSuite... [00:25] Cloning ComfyUI-Multiband... [00:25] Installing requirements for ComfyUI-Multiband... [00:26] [00:26] [requirements] Re-installing main package requirements... [00:26] Installing requirements for ComfyUI-SAM3... [00:26] [comfy-env] _discover: scanning /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes [00:26] [comfy-env] _discover: ComfyUI-Multiband: no comfy-env.toml found [00:26] [comfy-env] _discover: ComfyUI-SAM3 -> sam3-nodes (custom_nodes/ComfyUI-SAM3/nodes/comfy-env.toml) [00:26] [comfy-env] _discover: ComfyUI-VideoHelperSuite: no comfy-env.toml found [00:26] [comfy-env] Workspace: /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce [00:26] [comfy-env] ComfyUI: /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI [00:26] [comfy-env] Found 1 node config(s): [00:26] - sam3-nodes <- custom_nodes/ComfyUI-SAM3/nodes/comfy-env.toml (python=host) [00:26] [comfy-env] Bootstrap interpreter has python 3.10 + torch 2.11.0 (cpu). Looking for cuda-only wheels matching cpu/torch2.11/cp310. [00:28] [comfy-env] GPU: No GPU detected [00:29] [comfy-env] No CUDA detected -- pinning comfyui torch to CPU index [00:29] [comfy-env] Comfyui feature pin: torch ==2.11.0, torchvision ==0.26.*, torchaudio ==2.11.* [00:29] [comfy-env] Host glibc 2.39 -> system-requirements [00:29] [comfy-env] comfyui env: python 3.10, torch from https://download.pytorch.org/whl/cpu [00:29] Generated /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/pixi.toml [00:29] [comfy-env] Installing 1 environment(s): [00:29] - sam3-nodes (python=host, 19 pypi deps, 3 cuda wheels) [00:29] [comfy-env] Running `pixi install --all`... [00:34] WARN Skipped running the post-link scripts because `run-post-link-scripts` = `false` [00:34] - bin/.librsvg-pre-unlink.sh [00:34] To enable them, run: [00:34] pixi config set --local run-post-link-scripts insecure [00:34] More info: [00:34] https://pixi.sh/latest/reference/pixi_configuration/#run-post-link-scripts [00:44] ✔ The following environments have been installed: default, comfyui, sam3-nodes. [00:44] [comfy-env] Note: env `default` is on disk but no node declares it in this run. Leaving as-is. Remove via `pixi clean --environment default` if intended. [00:44] [comfy-env] Install log: /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/install.log [00:44] [00:44] Installation complete! [00:45] Installing 2 node dependency(ies)... [00:45] ComfyUI-VideoHelperSuite from Kosinkadink/ComfyUI-VideoHelperSuite [00:45] ComfyUI-VideoHelperSuite already exists, skipping... [00:45] ComfyUI-Multiband from PozzettiAndrea/ComfyUI-Multiband [00:45] ComfyUI-Multiband already exists, skipping... [00:45] Installing validation endpoint... [00:45] Cloning PozzettiAndrea/ComfyUI-validate-endpoint... [00:45] Running: git clone --depth 1 https://github.com/PozzettiAndrea/ComfyUI-validate-endpoint.git /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-validate-endpoint [00:45] CUDA packages absent (will mock): cc_torch, torch_generic_nms, flash_attn [00:45] [INSTALL] PASSED [00:45] [00:45] [3/5] REGISTRATION [00:45] ---------------------------------------- [00:45] [DEBUG] server=None, server_url=None, api=None [00:45] Starting ComfyUI server... [00:45] Starting ComfyUI server on port 41886... [00:45] CUDA mock packages: cc_torch, torch_generic_nms, flash_attn [00:45] Starting ComfyUI server on port 41886... [00:45] Waiting for server to be ready (timeout: 600s)... [00:46] [ComfyUI] setup plugin alembic.autogenerate.schemas [00:46] [ComfyUI] setup plugin alembic.autogenerate.tables [00:46] [ComfyUI] setup plugin alembic.autogenerate.types [00:46] [ComfyUI] setup plugin alembic.autogenerate.constraints [00:46] [ComfyUI] setup plugin alembic.autogenerate.defaults [00:46] [ComfyUI] setup plugin alembic.autogenerate.comments [00:46] [ComfyUI] [comfy-env] ComfyUI-SAM3: 1 isolation env(s): [00:46] [ComfyUI] [comfy-env] nodes -> /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes [00:46] [ComfyUI] [comfy-env] prestartup complete [00:46] [ComfyUI] [00:46] [ComfyUI] Prestartup times for custom nodes: [00:46] [ComfyUI] 0.2 seconds: /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3 [00:46] [ComfyUI] [00:50] [ComfyUI] Found comfy_kitchen backend triton: {'available': False, 'disabled': True, 'unavailable_reason': 'Neither CUDA nor XPU available on this system', 'capabilities': []} [00:50] [ComfyUI] Found comfy_kitchen backend cuda: {'available': False, 'disabled': False, 'unavailable_reason': 'CUDA not available on this system', 'capabilities': []} [00:50] [ComfyUI] Found comfy_kitchen backend eager: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['apply_rope', 'apply_rope1', 'dequantize_mxfp8', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'scaled_mm_mxfp8', 'scaled_mm_nvfp4']} [00:50] [ComfyUI] Checkpoint files will always be loaded safely. [00:50] [ComfyUI] Total VRAM 15990 MB, total RAM 15990 MB [00:50] [ComfyUI] pytorch version: 2.11.0+cu130 [00:50] [ComfyUI] Set vram state to: DISABLED [00:50] [ComfyUI] Device: cpu [00:51] [ComfyUI] Using sub quadratic optimization for attention, if you have memory or speed issues try using: --use-split-cross-attention [00:55] [ComfyUI] Python version: 3.10.20 (main, May 8 2026, 18:32:33) [Clang 22.1.3 ] [00:55] [ComfyUI] ComfyUI version: 0.20.1 [00:55] [ComfyUI] comfy-aimdo version: 0.3.0 [00:55] [ComfyUI] comfy-kitchen version: 0.2.8 [00:55] [ComfyUI] ****** User settings have been changed to be stored on the server instead of browser storage. ****** [00:55] [ComfyUI] ****** For multi-user setups add the --multi-user CLI argument to enable multiple user profiles. ****** [00:55] [ComfyUI] ComfyUI frontend version: 1.43.18 [00:55] [ComfyUI] [Prompt Server] web root: /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/.venv/lib/python3.10/site-packages/comfyui_frontend_package/static [00:55] [ComfyUI] Asset seeder disabled [00:57] [ComfyUI] ComfyUI-Multiband loaded: 15 nodes registered [00:57] [ComfyUI] [comfy-env] Version: 0.3.80 [00:57] [ComfyUI] [comfy-env] register_nodes: ComfyUI-SAM3 [00:57] [ComfyUI] [comfy-env] ComfyUI source dir: /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI [00:57] [ComfyUI] [comfy-env] Found 1 isolation env(s) [00:57] [ComfyUI] [comfy-env] Scanning nodes metadata (isolation root)... [00:57] [ComfyUI] [comfy-env] Metadata scan: /home/runner/.pixi/bin/pixi run --as-is --manifest-path /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/pixi.toml -e sam3-nodes python /tmp/comfy_meta_m23k3f22.py /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3 nodes [00:57] [ComfyUI] [comfy-env] Scan env PATH for nodes: [00:57] [ComfyUI] [comfy-env] [0] /opt/hostedtoolcache/Python/3.11.15/x64/bin [00:57] [ComfyUI] [comfy-env] [1] /opt/hostedtoolcache/Python/3.11.15/x64 [00:57] [ComfyUI] [comfy-env] [2] /snap/bin [00:57] [ComfyUI] [comfy-env] [3] /home/runner/.local/bin [00:57] [ComfyUI] [comfy-env] [4] /opt/pipx_bin [00:57] [ComfyUI] [comfy-env] [5] /home/runner/.cargo/bin [00:57] [ComfyUI] [comfy-env] [6] /home/runner/.config/composer/vendor/bin [00:57] [ComfyUI] [comfy-env] [7] /usr/local/.ghcup/bin [00:57] [ComfyUI] [comfy-env] [8] /home/runner/.dotnet/tools [00:57] [ComfyUI] [comfy-env] [9] /usr/local/sbin [00:57] [ComfyUI] [comfy-env] [10] /usr/local/bin [00:57] [ComfyUI] [comfy-env] [11] /usr/sbin [00:57] [ComfyUI] [comfy-env] [12] /usr/bin [00:57] [ComfyUI] [comfy-env] [13] /sbin [00:57] [ComfyUI] [comfy-env] [14] /bin [00:57] [ComfyUI] [comfy-env] [15] /usr/games [00:57] [ComfyUI] [comfy-env] [16] /usr/local/games [00:57] [ComfyUI] [comfy-env] [17] /snap/bin [01:03] [ComfyUI] [comfy-env] Metadata scan stderr for nodes: [01:03] [ComfyUI] [comfy-env] [meta-scan] importing nodes from /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3 [01:04] [ComfyUI] [comfy-env] [meta-scan] import OK [01:04] [ComfyUI] [comfy-env] Scanned nodes: 15 nodes (6.0s) [01:04] [ComfyUI] [comfy-env] Scanned nodes root: 15 nodes (6.2s) [01:04] [ComfyUI] [comfy-env] Registered proxy route: POST /sam3/interactive_segment_one -> nodes.sam3_interactive._api_segment_one [01:04] [ComfyUI] [comfy-env] Imported nodes root: 15 nodes (isolation) [01:04] [ComfyUI] [comfy-env] Registered 15 total nodes [01:04] [ComfyUI] [01:04] [ComfyUI] Import times for custom nodes: [01:04] [ComfyUI] 0.0 seconds: /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/websocket_image_save.py [01:04] [ComfyUI] 0.0 seconds: /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-validate-endpoint [01:04] [ComfyUI] 0.0 seconds: /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-VideoHelperSuite [01:04] [ComfyUI] 0.1 seconds: /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-Multiband [01:04] [ComfyUI] 6.4 seconds: /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3 [01:04] [ComfyUI] [01:04] [ComfyUI] Context impl SQLiteImpl. [01:04] [ComfyUI] Will assume non-transactional DDL. [01:04] [ComfyUI] Context impl SQLiteImpl. [01:04] [ComfyUI] Will assume non-transactional DDL. [01:04] [ComfyUI] Running upgrade -> 0001_assets, Initial assets schema [01:04] [ComfyUI] Revision ID: 0001_assets [01:04] [ComfyUI] Revises: None [01:04] [ComfyUI] Create Date: 2025-12-10 00:00:00 [01:04] [ComfyUI] Running upgrade 0001_assets -> 0002_merge_to_asset_references, Merge AssetInfo and AssetCacheState into unified asset_references table. [01:04] [ComfyUI] Running upgrade 0002_merge_to_asset_references -> 0003_add_metadata_job_id, Add system_metadata and job_id columns to asset_references. [01:04] [ComfyUI] Change preview_id FK from assets.id to asset_references.id. [01:04] [ComfyUI] Database upgraded from None to 0003_add_metadata_job_id [01:04] [ComfyUI] Starting server [01:04] [ComfyUI] [01:04] [ComfyUI] To see the GUI go to: http://127.0.0.1:41886 [01:04] Server responding, waiting for nodes to load... [01:24] Server is ready! [01:24] Checking for import errors in server logs... [01:24] No import errors detected [01:24] Found 775 registered nodes [01:24] [REGISTRATION] PASSED [01:24] [01:24] [4/5] INSTANTIATION [01:24] ---------------------------------------- [01:24] [DEBUG] server=, server_url=None, api= [01:24] Testing node constructors... [01:27] All 15 node(s) instantiated successfully! [01:27] [INSTANTIATION] PASSED [01:27] [01:27] [5/5] EXECUTION [01:27] ---------------------------------------- [01:27] [DEBUG] server=, server_url=None, api= [01:27] CPU runner - will execute 4 workflow(s) [01:27] Running 5 workflow(s) (all with videos)... [01:27] Installing screenshot dependencies (playwright, pillow)... [01:27] Packages installed, downloading chromium browser... [01:36] Screenshot dependencies installed successfully [01:36] Starting headless browser... [01:36] Installing Mesa llvmpipe (apt)... [01:48] Mesa install failed: E: Unable to locate package libegl1-mesa [01:48] No GPU, no Mesa -- using SwiftShader software rendering [01:49] Capturing execution frames: image_seg_point.json [01:51] [ComfyUI] [DEPRECATION WARNING] Detected import of deprecated legacy API: /extensions/core/widgetInputs.js. This is likely caused by a custom node extension using outdated APIs. Please update your extensions or contact the extension author for an updated version. [01:55] Frame 1 saved (frame_000.jpg, t=0.0s) [01:55] Validating workflow... [01:55] [ComfyUI] === /validate endpoint called === [01:55] [ComfyUI] Received JSON with keys: ['prompt'] [01:55] [ComfyUI] Prompt has 6 nodes [01:55] [ComfyUI] Node 1: LoadImage [01:55] [ComfyUI] Inputs: {"image": "example_image.jpg"} [01:55] [ComfyUI] Node 15: SAM3PointCollector [01:55] Queuing workflow for execution... [01:55] [ComfyUI] Inputs: {"points_store": "{\"positive\":[{\"x\":426.27735719657574,\"y\":321.42611685978835},{\"x\":417.13743597288953,\"y\":407.3413796907987},{\"x\":431.76130993078743,\"y\":549.9241563039649}],\"negative\" [01:55] [ComfyUI] Node 28: PreviewImage [01:55] [ComfyUI] Inputs: {"images": ["31", 2]} [01:55] [ComfyUI] Node 29: MaskPreview [01:55] [ComfyUI] Inputs: {"mask": ["31", 0]} [01:55] [ComfyUI] Node 30: LoadSAM3Model [01:55] [ComfyUI] Inputs: {"precision": "auto", "compile": false} [01:55] [ComfyUI] Node 31: SAM3Segmentation [01:55] [ComfyUI] Inputs: {"refinement_iterations": 0, "use_multimask": true, "output_best_mask": true, "sam3_model_config": ["30", 0], "image": ["1", 0], "positive_points": ["15", 0], "negative_points": ["15", 1]} [01:55] [ComfyUI] Generated prompt_id: 44f9d2d3-94a1-4c46-836e-9063769054dd [01:55] [ComfyUI] Calling execution.validate_prompt()... [01:55] [ComfyUI] Validation result: valid=True [01:55] [ComfyUI] === Validation PASSED === [01:55] [ComfyUI] got prompt [01:55] [ComfyUI] [comfy-env] Running SAM3PointCollector: points_store: '{"positive":[{"x":426.27735719657574,"y":321.42611685978..., coordinates: '[{"x":426.27735719657574,"y":321.42611685978835},{"x":41..., neg_coordinates: '[]', image: torch.float32 [1,720,1280,3] cpu [01:55] [ComfyUI] [comfy-env] SubprocessWorker: /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/bin/python [01:55] [ComfyUI] [SubprocessWorker] call_method: nodes.sam3_interactive.SAM3PointCollector.EXECUTE_NORMALIZED [01:55] [ComfyUI] [comfy-env] env_vars: {'COMFYUI_BASE': '/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI'} [01:55] [ComfyUI] [SubprocessWorker] acquired lock, ensuring started... [01:55] [ComfyUI] [ComfyUI-SAM3] python: /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/bin/python [01:55] [ComfyUI] [ComfyUI-SAM3] sys_path sent to worker: ['/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI', '/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3', '/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages', '/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3'] [01:55] [ComfyUI] [SubprocessWorker] is_pixi=True, python=/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/bin/python [01:55] [ComfyUI] [SubprocessWorker] launching cmd=['/home/runner/.pixi/bin/pixi', 'run', '--as-is', '--manifest-path', '/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/pixi.toml', '-e', 'sam3-nodes', 'python', '/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py', 'abstract://comfy_worker_fece7ceb785a']... [01:55] [ComfyUI] [SubprocessWorker] socket_addr=abstract://comfy_worker_fece7ceb785a (abstract, no filesystem path) [01:58] [ComfyUI] [SubprocessWorker] worker started/confirmed [01:58] [ComfyUI] [comfy-env] PRE-SERIALIZE 'image' shape: torch.Size([1, 720, 1280, 3]) [01:58] [ComfyUI] [SubprocessWorker] serializing kwargs to shm... [01:58] [ComfyUI] [SubprocessWorker] created 0 shm blocks [01:58] [ComfyUI] [SubprocessWorker] sending request via socket... [01:58] [ComfyUI] [SubprocessWorker] call_id=1 sending call_method [01:58] [ComfyUI] [worker:ComfyUI-SAM3] [comfy-env] _from_shm got dict with keys: ['__type__', 'strategy', 'parent_pid', 'fd', 'storage_size'] [01:58] [ComfyUI] [worker:ComfyUI-SAM3] [comfy-env] DESERIALIZE TensorRef: tensor_size=[1, 720, 1280, 3] [01:58] [ComfyUI] [worker:ComfyUI-SAM3] [comfy-env] DESERIALIZED tensor shape: torch.Size([1, 720, 1280, 3]) [01:58] Frame 2 saved (frame_001.jpg, t=2.3s) [01:58] [capture-periodic] freeze=2ms shot=109ms unfreeze=3ms saved=True [01:58] [ComfyUI] [SubprocessWorker] got response: ok [01:58] [ComfyUI] [comfy-env] SAM3PointCollector done (2.53s): result: NodeOutput [01:58] [ComfyUI] [comfy-env] Running LoadSAM3Model: precision: 'auto', compile: False [01:58] [ComfyUI] [SubprocessWorker] call_method: nodes.load_model.LoadSAM3Model.EXECUTE_NORMALIZED [01:58] [ComfyUI] [SubprocessWorker] acquired lock, ensuring started... [01:58] [ComfyUI] [ComfyUI-SAM3] Health check: ping (timeout=5.0s)... [01:58] [ComfyUI] [ComfyUI-SAM3] Health check: ok [01:58] [ComfyUI] [SubprocessWorker] worker started/confirmed [01:58] [ComfyUI] [SubprocessWorker] serializing kwargs to shm... [01:58] [ComfyUI] [SubprocessWorker] created 0 shm blocks [01:58] [ComfyUI] [SubprocessWorker] sending request via socket... [01:58] [ComfyUI] [SubprocessWorker] call_id=2 sending call_method [01:58] [ComfyUI] [worker:ComfyUI-SAM3] [SAM3] Model not found at /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/models/sam3/sam3.safetensors, downloading from HuggingFace... [01:58] Node executed (1 total), capturing... [01:58] [ComfyUI] [worker:ComfyUI-SAM3] [SAM3] Downloading sam3.safetensors from apozz/sam3-safetensors... [01:58] Frame 3 saved (frame_002.jpg, t=3.0s) [01:58] [capture-node] freeze=2ms shot=120ms unfreeze=4ms saved=True [01:58] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [01:58] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [01:59] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [01:59] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [01:59] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [01:59] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [01:59] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [01:59] [ComfyUI] [worker:ComfyUI-SAM3] [SAM3] Download progress: 10% [01:59] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [01:59] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [01:59] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [01:59] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:00] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:00] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:00] [ComfyUI] [worker:ComfyUI-SAM3] [SAM3] Download progress: 20% [02:00] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:00] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:00] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:00] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:00] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:01] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:01] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:01] Frame 4 saved (frame_003.jpg, t=5.4s) [02:01] [capture-periodic] freeze=11ms shot=127ms unfreeze=3ms saved=True [02:01] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:01] [ComfyUI] [worker:ComfyUI-SAM3] [SAM3] Download progress: 30% [02:01] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:01] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:01] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:01] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:01] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:02] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:02] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:02] [capture-loop] iter=50 t=6.7s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=12 ram=2.1/15.6GB (14%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=0.89GB [02:02] [ComfyUI] [worker:ComfyUI-SAM3] [SAM3] Download progress: 40% [02:02] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:02] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:03] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:03] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:03] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:03] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:03] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:03] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:03] [ComfyUI] [worker:ComfyUI-SAM3] [SAM3] Download progress: 50% [02:03] Frame 5 saved (frame_004.jpg, t=7.7s) [02:03] [capture-periodic] freeze=3ms shot=279ms unfreeze=17ms saved=True [02:03] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:03] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:04] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:04] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:04] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:04] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:04] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:04] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:04] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:05] [ComfyUI] [worker:ComfyUI-SAM3] [SAM3] Download progress: 60% [02:05] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:05] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:05] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:05] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:05] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:05] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:05] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:05] Frame 6 saved (frame_005.jpg, t=10.1s) [02:06] [ComfyUI] [worker:ComfyUI-SAM3] [SAM3] Download progress: 70% [02:06] [capture-periodic] freeze=9ms shot=134ms unfreeze=3ms saved=True [02:06] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:06] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:06] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:06] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:06] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:06] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:06] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:07] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:07] [ComfyUI] [worker:ComfyUI-SAM3] [SAM3] Download progress: 80% [02:07] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:07] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:07] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:07] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:07] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:08] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:08] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:08] [ComfyUI] [worker:ComfyUI-SAM3] [SAM3] Download progress: 90% [02:08] Frame 7 saved (frame_006.jpg, t=12.5s) [02:08] [capture-periodic] freeze=2ms shot=120ms unfreeze=3ms saved=True [02:08] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:08] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:08] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:08] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:08] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:09] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:09] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:09] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:09] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:09] [ComfyUI] [worker:ComfyUI-SAM3] [SAM3] Download progress: 100% [02:09] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=2 [02:09] [capture-loop] iter=100 t=14.1s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=2.1/15.6GB (14%) swap=0.0/3.0GB (0%) cpu=43% browser_rss=0.94GB [02:09] [ComfyUI] [worker:ComfyUI-SAM3] [SAM3] Downloaded sam3.safetensors [02:09] [ComfyUI] [worker:ComfyUI-SAM3] [SAM3] Model downloaded successfully [02:09] [ComfyUI] [SubprocessWorker] got response: ok [02:09] [ComfyUI] [comfy-env] LoadSAM3Model done (11.45s): result: NodeOutput [02:09] [ComfyUI] [comfy-env] Running SAM3Segmentation: refinement_iterations: 0, use_multimask: True, output_best_mask: True, sam3_model_config: dict, image: torch.float32 [1,720,1280,3] cpu, positive_points: dict, negative_points: dict [02:09] [ComfyUI] [SubprocessWorker] call_method: nodes.segmentation.SAM3Segmentation.EXECUTE_NORMALIZED [02:10] [ComfyUI] [SubprocessWorker] acquired lock, ensuring started... [02:10] [ComfyUI] [ComfyUI-SAM3] Health check: ping (timeout=5.0s)... [02:10] [ComfyUI] [ComfyUI-SAM3] Health check: ok [02:10] [ComfyUI] [SubprocessWorker] worker started/confirmed [02:10] [ComfyUI] [comfy-env] PRE-SERIALIZE 'image' shape: torch.Size([1, 720, 1280, 3]) [02:10] [ComfyUI] [SubprocessWorker] serializing kwargs to shm... [02:10] [ComfyUI] [SubprocessWorker] created 0 shm blocks [02:10] [ComfyUI] [SubprocessWorker] sending request via socket... [02:10] [ComfyUI] [SubprocessWorker] call_id=3 sending call_method [02:10] [ComfyUI] [worker:ComfyUI-SAM3] [comfy-env] _from_shm got dict with keys: ['__type__', 'strategy', 'parent_pid', 'fd', 'storage_size'] [02:10] [ComfyUI] [worker:ComfyUI-SAM3] [comfy-env] DESERIALIZE TensorRef: tensor_size=[1, 720, 1280, 3] [02:10] [ComfyUI] [worker:ComfyUI-SAM3] [comfy-env] DESERIALIZED tensor shape: torch.Size([1, 720, 1280, 3]) [02:10] Frame 8 saved (frame_007.jpg, t=14.8s) [02:10] [capture-periodic] freeze=3ms shot=113ms unfreeze=5ms saved=True [02:12] [ComfyUI] [SubprocessWorker] callback 'request_vram_budget' from call_id=3 [02:12] [ComfyUI] [comfy-env] VRAM request: 3.50GB, free before eviction: 14.32GB [02:12] [ComfyUI] [comfy-env] VRAM after eviction: 14.32GB [02:15] [capture-loop] iter=150 t=19.8s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=2.5/15.6GB (16%) swap=0.0/3.0GB (0%) cpu=36% browser_rss=0.93GB [02:20] [capture-loop] iter=200 t=25.3s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=3 ram=2.4/15.6GB (16%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=0.85GB [02:26] [capture-loop] iter=250 t=30.9s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=3 ram=2.4/15.6GB (16%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=0.84GB [02:32] [capture-loop] iter=300 t=36.4s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=3 ram=3.7/15.6GB (23%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=0.85GB [02:37] [capture-loop] iter=350 t=42.1s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=2.4/15.6GB (16%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=0.85GB [02:43] [capture-loop] iter=400 t=47.6s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=2.4/15.6GB (16%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=0.85GB [02:48] [capture-loop] iter=450 t=53.2s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=3 ram=2.5/15.6GB (16%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=0.85GB [02:54] [capture-loop] iter=500 t=58.7s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=2.4/15.6GB (15%) swap=0.0/3.0GB (0%) cpu=34% browser_rss=0.82GB [02:56] [ComfyUI] [worker:ComfyUI-SAM3] [02:56] [ComfyUI] === WATCHDOG TICK 1 === [02:56] [ComfyUI] [worker:ComfyUI-SAM3] Current thread 0x00007fbcba3a16c0 (most recent call first): [02:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 52 in _watchdog [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 953 in run [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 1016 in _bootstrap_inner [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 973 in _bootstrap [02:56] [ComfyUI] [02:56] [ComfyUI] Thread 0x00007fbcbae75740 (most recent call first): [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ops.py", line 475 in forward_comfy_cast_weights [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ops.py", line 482 in forward [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 144 in forward [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 823 in forward [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 983 in forward [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 2041 in forward [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 3321 in _forward_image_no_act_ckpt [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 3314 in forward_image [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/utils.py", line 1324 in set_image [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124 in decorate_context [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/segmentation.py", line 496 in execute [02:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy_api/latest/_io.py", line 1833 in EXECUTE_NORMALIZED [02:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 1762 in main [02:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 1807 in [02:56] [ComfyUI] [02:56] [ComfyUI] [worker:ComfyUI-SAM3] === END === [02:56] [ComfyUI] [02:59] [capture-loop] iter=550 t=64.3s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=2.4/15.6GB (15%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=0.81GB [03:05] [capture-loop] iter=600 t=69.8s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=2.5/15.6GB (16%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=0.81GB [03:11] [capture-loop] iter=650 t=75.4s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=2.4/15.6GB (15%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=0.82GB [03:16] [capture-loop] iter=700 t=80.9s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=2.6/15.6GB (17%) swap=0.0/3.0GB (0%) cpu=34% browser_rss=0.81GB [03:22] [capture-loop] iter=750 t=86.5s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=2.6/15.6GB (17%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=0.81GB [03:27] [capture-loop] iter=800 t=92.0s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=2.8/15.6GB (18%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=0.81GB [03:33] [capture-loop] iter=850 t=97.6s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=3 ram=4.6/15.6GB (30%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=0.81GB [03:38] [capture-loop] iter=900 t=103.1s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=2.7/15.6GB (17%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=0.81GB [03:42] [ComfyUI] [SubprocessWorker] got response: ok [03:42] [ComfyUI] [comfy-env] SAM3Segmentation done (92.28s): result: NodeOutput [03:42] Node executed (2 total), capturing... [03:42] [ComfyUI] Prompt executed in 106.40 seconds [03:42] Frame 9 saved (frame_008.jpg, t=106.7s) [03:42] [capture-node] freeze=3ms shot=124ms unfreeze=3ms saved=True [03:42] Node executed (3 total), capturing... [03:42] Frame 10 saved (frame_009.jpg, t=107.2s) [03:42] [capture-node] freeze=2ms shot=118ms unfreeze=0ms saved=True [03:42] Execution complete (t=107.1s) [03:42] Captured 10 unique frames over 107.31s [03:42] [debug] iframes: 0 [03:42] [timing] trigger_3d_previews: 0.0s [03:45] Saved high-quality screenshot: image_seg_point_executed.png [03:45] [timing] final_screenshot: 0.3s [03:45] Capturing execution frames: image_seg_text.json [03:50] Frame 1 saved (frame_000.jpg, t=0.0s) [03:50] Validating workflow... [03:50] [ComfyUI] === /validate endpoint called === [03:50] [ComfyUI] Received JSON with keys: ['prompt'] [03:50] [ComfyUI] Prompt has 5 nodes [03:50] [ComfyUI] Node 1: LoadImage [03:50] [ComfyUI] Inputs: {"image": "example_image.jpg"} [03:50] [ComfyUI] Node 3: PreviewImage [03:50] [ComfyUI] Inputs: {"images": ["15", 1]} [03:50] [ComfyUI] Node 4: MaskPreview [03:50] Queuing workflow for execution... [03:50] [ComfyUI] Inputs: {"mask": ["15", 0]} [03:50] [ComfyUI] Node 12: LoadSAM3Model [03:50] [ComfyUI] Inputs: {"precision": "fp32", "compile": false} [03:50] [ComfyUI] Node 15: SAM3Grounding [03:50] [ComfyUI] Inputs: {"confidence_threshold": 0.2, "text_prompt": "child", "max_detections": -1, "sam3_model_config": ["12", 0], "image": ["1", 0]} [03:50] [ComfyUI] Generated prompt_id: 4b97a3e2-ee99-4df9-b40b-fbe9322c859c [03:50] [ComfyUI] Calling execution.validate_prompt()... [03:50] [ComfyUI] Validation result: valid=True [03:50] [ComfyUI] === Validation PASSED === [03:50] [ComfyUI] got prompt [03:50] [ComfyUI] [comfy-env] Running LoadSAM3Model: precision: 'fp32', compile: False [03:50] [ComfyUI] [SubprocessWorker] call_method: nodes.load_model.LoadSAM3Model.EXECUTE_NORMALIZED [03:50] [ComfyUI] [SubprocessWorker] acquired lock, ensuring started... [03:50] [ComfyUI] [ComfyUI-SAM3] Health check: ping (timeout=5.0s)... [03:50] [ComfyUI] [ComfyUI-SAM3] Health check: ok [03:50] [ComfyUI] [SubprocessWorker] worker started/confirmed [03:50] [ComfyUI] [SubprocessWorker] serializing kwargs to shm... [03:50] [ComfyUI] [SubprocessWorker] created 0 shm blocks [03:50] [ComfyUI] [SubprocessWorker] sending request via socket... [03:50] [ComfyUI] [SubprocessWorker] call_id=4 sending call_method [03:50] [ComfyUI] [SubprocessWorker] got response: ok [03:50] [ComfyUI] [comfy-env] LoadSAM3Model done (0.01s): result: NodeOutput [03:50] [ComfyUI] [comfy-env] Running SAM3Grounding: confidence_threshold: 0.2, text_prompt: 'child', max_detections: -1, sam3_model_config: dict, image: torch.float32 [1,720,1280,3] cpu [03:50] [ComfyUI] [SubprocessWorker] call_method: nodes.segmentation.SAM3Grounding.EXECUTE_NORMALIZED [03:50] [ComfyUI] [SubprocessWorker] acquired lock, ensuring started... [03:50] [ComfyUI] [ComfyUI-SAM3] Health check: ping (timeout=5.0s)... [03:50] [ComfyUI] [ComfyUI-SAM3] Health check: ok [03:50] [ComfyUI] [SubprocessWorker] worker started/confirmed [03:50] [ComfyUI] [comfy-env] PRE-SERIALIZE 'image' shape: torch.Size([1, 720, 1280, 3]) [03:50] [ComfyUI] [SubprocessWorker] serializing kwargs to shm... [03:50] [ComfyUI] [SubprocessWorker] created 0 shm blocks [03:50] [ComfyUI] [SubprocessWorker] sending request via socket... [03:50] [ComfyUI] [SubprocessWorker] call_id=5 sending call_method [03:50] [ComfyUI] [worker:ComfyUI-SAM3] [comfy-env] _from_shm got dict with keys: ['__type__', 'strategy', 'parent_pid', 'fd', 'storage_size'] [03:50] [ComfyUI] [worker:ComfyUI-SAM3] [comfy-env] DESERIALIZE TensorRef: tensor_size=[1, 720, 1280, 3] [03:50] [ComfyUI] [worker:ComfyUI-SAM3] [comfy-env] DESERIALIZED tensor shape: torch.Size([1, 720, 1280, 3]) [03:51] [ComfyUI] [SubprocessWorker] callback 'request_vram_budget' from call_id=5 [03:51] [ComfyUI] [comfy-env] VRAM request: 3.50GB, free before eviction: 14.03GB [03:51] [ComfyUI] [comfy-env] VRAM after eviction: 14.03GB [03:53] Frame 2 saved (frame_001.jpg, t=2.2s) [03:53] [capture-periodic] freeze=2ms shot=93ms unfreeze=4ms saved=True [03:55] Frame 3 saved (frame_002.jpg, t=4.4s) [03:55] [capture-periodic] freeze=2ms shot=109ms unfreeze=3ms saved=True [03:56] [ComfyUI] [worker:ComfyUI-SAM3] [03:56] [ComfyUI] === WATCHDOG TICK 2 === [03:56] [ComfyUI] [worker:ComfyUI-SAM3] Thread 0x00007fbc3e27b6c0 (most recent call first): [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/socket.py", line 293 in accept [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/multiprocessing/connection.py", line 609 in accept [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/multiprocessing/connection.py", line 463 in accept [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/multiprocessing/resource_sharer.py", line 138 in _serve [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 953 in run [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 1016 in _bootstrap_inner [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 973 in _bootstrap [03:56] [ComfyUI] [03:56] [ComfyUI] Current thread 0x00007fbcba3a16c0 (most recent call first): [03:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 52 in _watchdog [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 953 in run [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 1016 in _bootstrap_inner [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 973 in _bootstrap [03:56] [ComfyUI] [03:56] [ComfyUI] Thread 0x00007fbcbae75740 (most recent call first): [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ops.py", line 475 in forward_comfy_cast_weights [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ops.py", line 482 in forward [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 147 in forward [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 823 in forward [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 983 in forward [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 2041 in forward [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 3321 in _forward_image_no_act_ckpt [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 3314 in forward_image [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/utils.py", line 1324 in set_image [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124 in decorate_context [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/segmentation.py", line 133 in _segment_grounding [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/segmentation.py", line 111 in execute [03:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy_api/latest/_io.py", line 1833 in EXECUTE_NORMALIZED [03:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 1762 in main [03:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 1807 in [03:56] [ComfyUI] [03:56] [ComfyUI] [worker:ComfyUI-SAM3] === END === [03:56] [ComfyUI] [03:56] [capture-loop] iter=50 t=5.6s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=2 ram=2.6/15.6GB (16%) swap=0.0/3.0GB (0%) cpu=26% browser_rss=0.82GB [04:01] [capture-loop] iter=100 t=11.1s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=2 ram=2.6/15.6GB (17%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=0.82GB [04:07] [capture-loop] iter=150 t=16.6s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=2.7/15.6GB (17%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=0.82GB [04:12] [capture-loop] iter=200 t=22.1s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=2.8/15.6GB (18%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=0.81GB [04:18] [capture-loop] iter=250 t=27.7s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=2.6/15.6GB (17%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=0.73GB [04:23] [capture-loop] iter=300 t=33.2s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=2 ram=2.4/15.6GB (16%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=0.73GB [04:29] [capture-loop] iter=350 t=38.7s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=3.9/15.6GB (25%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=0.72GB [04:35] [capture-loop] iter=400 t=44.3s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=2.6/15.6GB (17%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=0.71GB [04:40] [capture-loop] iter=450 t=49.7s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=2.6/15.6GB (16%) swap=0.0/3.0GB (0%) cpu=31% browser_rss=0.72GB [04:46] [capture-loop] iter=500 t=55.3s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=2.7/15.6GB (17%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=0.72GB [04:51] [capture-loop] iter=550 t=60.7s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=3.2/15.6GB (20%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=0.72GB [04:56] [ComfyUI] [worker:ComfyUI-SAM3] [04:56] [ComfyUI] === WATCHDOG TICK 3 === [04:56] [ComfyUI] [worker:ComfyUI-SAM3] Thread 0x00007fbc3e27b6c0 (most recent call first): [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/socket.py", line 293 in accept [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/multiprocessing/connection.py", line 609 in accept [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/multiprocessing/connection.py", line 463 in accept [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/multiprocessing/resource_sharer.py", line 138 in _serve [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 953 in run [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 1016 in _bootstrap_inner [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 973 in _bootstrap [04:56] [ComfyUI] [04:56] [ComfyUI] Current thread 0x00007fbcba3a16c0 (most recent call first): [04:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 52 in _watchdog [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 953 in run [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 1016 in _bootstrap_inner [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 973 in _bootstrap [04:56] [ComfyUI] [04:56] [ComfyUI] Thread 0x00007fbcbae75740 (most recent call first): [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ldm/modules/sub_quadratic_attention.py", line 170 in _get_attention_scores_no_kv_chunking [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ldm/modules/sub_quadratic_attention.py", line 259 in efficient_dot_product_attention [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ldm/modules/attention.py", line 275 in attention_sub_quad [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ldm/modules/attention.py", line 139 in wrapper [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/attention.py", line 120 in sam3_attention [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 729 in forward [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 819 in forward [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 983 in forward [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 2041 in forward [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 3321 in _forward_image_no_act_ckpt [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 3314 in forward_image [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/utils.py", line 1324 in set_image [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124 in decorate_context [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/segmentation.py", line 133 in _segment_grounding [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/segmentation.py", line 111 in execute [04:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy_api/latest/_io.py", line 1833 in EXECUTE_NORMALIZED [04:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 1762 in main [04:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 1807 in [04:56] [ComfyUI] [04:56] [ComfyUI] [worker:ComfyUI-SAM3] === END === [04:56] [ComfyUI] [04:57] [capture-loop] iter=600 t=66.3s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=2 ram=2.6/15.6GB (17%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=0.72GB [05:02] [capture-loop] iter=650 t=71.8s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=2.6/15.6GB (16%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=0.72GB [05:08] [capture-loop] iter=700 t=77.4s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=2.6/15.6GB (17%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=0.72GB [05:13] [capture-loop] iter=750 t=82.8s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=2.7/15.6GB (17%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=0.72GB [05:19] [capture-loop] iter=800 t=88.4s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=2 ram=2.8/15.6GB (18%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=0.72GB [05:24] [capture-loop] iter=850 t=94.0s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=3.1/15.6GB (20%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=0.72GB [05:28] [ComfyUI] [SubprocessWorker] got response: ok [05:28] [ComfyUI] [comfy-env] SAM3Grounding done (97.64s): result: NodeOutput [05:28] Node executed (1 total), capturing... [05:28] [ComfyUI] Prompt executed in 97.91 seconds [05:29] Frame 4 saved (frame_003.jpg, t=98.2s) [05:29] [capture-node] freeze=3ms shot=113ms unfreeze=4ms saved=True [05:29] Node executed (2 total), capturing... [05:29] Frame 5 saved (frame_004.jpg, t=98.6s) [05:29] [capture-node] freeze=2ms shot=108ms unfreeze=0ms saved=True [05:29] Execution complete (t=98.6s) [05:29] Captured 5 unique frames over 98.72s [05:29] [debug] iframes: 0 [05:29] [timing] trigger_3d_previews: 0.0s [05:31] Saved high-quality screenshot: image_seg_text_executed.png [05:31] [timing] final_screenshot: 0.3s [05:32] Capturing execution frames: scene_seg.json [05:37] Frame 1 saved (frame_000.jpg, t=0.0s) [05:37] Validating workflow... [05:37] [ComfyUI] === /validate endpoint called === [05:37] [ComfyUI] Received JSON with keys: ['prompt'] [05:37] [ComfyUI] Prompt has 10 nodes [05:37] [ComfyUI] Node 1: LoadImage [05:37] [ComfyUI] Inputs: {"image": "image.png"} [05:37] [ComfyUI] Node 3: PreviewImage [05:37] [ComfyUI] Inputs: {"images": ["27", 1]} [05:37] [ComfyUI] Node 4: MaskPreview [05:37] [ComfyUI] Inputs: {"mask": ["27", 0]} [05:37] [ComfyUI] Node 12: LoadSAM3Model [05:37] [ComfyUI] Inputs: {"precision": "fp32", "compile": false} [05:37] Queuing workflow for execution... [05:37] [ComfyUI] Node 20: SAM3MultiRegionCollector [05:37] [ComfyUI] Inputs: {"multi_prompts_store": "[{\"positive_points\":[{\"x\":5052.74953932266,\"y\":2847.0600693869637},{\"x\":4932.510591392486,\"y\":2953.9413227753366},{\"x\":4718.752461738845,\"y\":2967.301479448883},{ [05:37] [ComfyUI] Node 22: MultibandFromBatch [05:37] [ComfyUI] Inputs: {"images": ["1", 0], "masks": ["27", 0]} [05:37] [ComfyUI] Node 23: MultibandSave [05:37] [ComfyUI] Inputs: {"file_path": "output/multiband", "format": "npz", "multiband": ["26", 0]} [05:37] [ComfyUI] Node 24: MultibandPreview [05:37] [ComfyUI] Inputs: {"channel_index": 0, "multiband": ["26", 0]} [05:37] [ComfyUI] Node 26: MultibandResize [05:37] [ComfyUI] Inputs: {"upscale_method": "nearest-exact", "width": 900, "height": 600, "crop": "center", "multiband": ["22", 0]} [05:37] [ComfyUI] Node 27: SAM3MultipromptSegmentation [05:37] [ComfyUI] Inputs: {"refinement_iterations": 0, "use_multimask": false, "sam3_model_config": ["12", 0], "image": ["1", 0], "multi_prompts": ["20", 0]} [05:37] [ComfyUI] Generated prompt_id: 018aa872-e9b8-430b-a8f8-205d977e94bf [05:37] [ComfyUI] Calling execution.validate_prompt()... [05:37] [ComfyUI] Validation result: valid=True [05:37] [ComfyUI] === Validation PASSED === [05:37] [ComfyUI] got prompt [05:38] [ComfyUI] [comfy-env] Running SAM3MultiRegionCollector: multi_prompts_store: '[{"positive_points":[{"x":5052.74953932266,"y":2847.0600..., image: torch.float32 [1,4480,6720,3] cpu [05:38] [ComfyUI] [SubprocessWorker] call_method: nodes.sam3_interactive.SAM3MultiRegionCollector.EXECUTE_NORMALIZED [05:38] [ComfyUI] [SubprocessWorker] acquired lock, ensuring started... [05:38] [ComfyUI] [ComfyUI-SAM3] Health check: ping (timeout=5.0s)... [05:38] [ComfyUI] [ComfyUI-SAM3] Health check: ok [05:38] [ComfyUI] [SubprocessWorker] worker started/confirmed [05:38] [ComfyUI] [comfy-env] PRE-SERIALIZE 'image' shape: torch.Size([1, 4480, 6720, 3]) [05:38] [ComfyUI] [SubprocessWorker] serializing kwargs to shm... [05:38] [ComfyUI] [SubprocessWorker] created 0 shm blocks [05:38] [ComfyUI] [SubprocessWorker] sending request via socket... [05:38] [ComfyUI] [SubprocessWorker] call_id=6 sending call_method [05:38] [ComfyUI] [worker:ComfyUI-SAM3] [comfy-env] _from_shm got dict with keys: ['__type__', 'strategy', 'parent_pid', 'fd', 'storage_size'] [05:38] [ComfyUI] [worker:ComfyUI-SAM3] [comfy-env] DESERIALIZE TensorRef: tensor_size=[1, 4480, 6720, 3] [05:38] [ComfyUI] [worker:ComfyUI-SAM3] [comfy-env] DESERIALIZED tensor shape: torch.Size([1, 4480, 6720, 3]) [05:38] [ComfyUI] [SubprocessWorker] got response: ok [05:38] [ComfyUI] [comfy-env] SAM3MultiRegionCollector done (0.43s): result: NodeOutput [05:38] [ComfyUI] [comfy-env] Running LoadSAM3Model: precision: 'fp32', compile: False [05:38] [ComfyUI] [SubprocessWorker] call_method: nodes.load_model.LoadSAM3Model.EXECUTE_NORMALIZED [05:38] [ComfyUI] [SubprocessWorker] acquired lock, ensuring started... [05:38] [ComfyUI] [ComfyUI-SAM3] Health check: ping (timeout=5.0s)... [05:38] [ComfyUI] [ComfyUI-SAM3] Health check: ok [05:38] [ComfyUI] [SubprocessWorker] worker started/confirmed [05:38] [ComfyUI] [SubprocessWorker] serializing kwargs to shm... [05:38] [ComfyUI] [SubprocessWorker] created 0 shm blocks [05:38] [ComfyUI] [SubprocessWorker] sending request via socket... [05:38] [ComfyUI] [SubprocessWorker] call_id=7 sending call_method [05:38] [ComfyUI] [SubprocessWorker] got response: ok [05:38] [ComfyUI] [comfy-env] LoadSAM3Model done (0.04s): result: NodeOutput [05:38] [ComfyUI] [comfy-env] Running SAM3MultipromptSegmentation: refinement_iterations: 0, use_multimask: False, sam3_model_config: dict, image: torch.float32 [1,4480,6720,3] cpu, multi_prompts: list[7] [05:38] [ComfyUI] [SubprocessWorker] call_method: nodes.segmentation.SAM3MultipromptSegmentation.EXECUTE_NORMALIZED [05:38] [ComfyUI] [SubprocessWorker] acquired lock, ensuring started... [05:38] [ComfyUI] [ComfyUI-SAM3] Health check: ping (timeout=5.0s)... [05:38] [ComfyUI] [ComfyUI-SAM3] Health check: ok [05:38] [ComfyUI] [SubprocessWorker] worker started/confirmed [05:38] [ComfyUI] [comfy-env] PRE-SERIALIZE 'image' shape: torch.Size([1, 4480, 6720, 3]) [05:38] [ComfyUI] [SubprocessWorker] serializing kwargs to shm... [05:38] [ComfyUI] [SubprocessWorker] created 0 shm blocks [05:38] [ComfyUI] [SubprocessWorker] sending request via socket... [05:38] [ComfyUI] [SubprocessWorker] call_id=8 sending call_method [05:38] [ComfyUI] [worker:ComfyUI-SAM3] [comfy-env] _from_shm got dict with keys: ['__type__', 'strategy', 'parent_pid', 'fd', 'storage_size'] [05:38] [ComfyUI] [worker:ComfyUI-SAM3] [comfy-env] DESERIALIZE TensorRef: tensor_size=[1, 4480, 6720, 3] [05:38] [ComfyUI] [worker:ComfyUI-SAM3] [comfy-env] DESERIALIZED tensor shape: torch.Size([1, 4480, 6720, 3]) [05:38] [ComfyUI] [SubprocessWorker] callback 'request_vram_budget' from call_id=8 [05:38] [ComfyUI] [comfy-env] VRAM request: 3.50GB, free before eviction: 13.01GB [05:38] [ComfyUI] [comfy-env] VRAM after eviction: 13.01GB [05:39] Node executed (1 total), capturing... [05:39] Frame 2 saved (frame_001.jpg, t=1.9s) [05:39] [capture-node] freeze=12ms shot=201ms unfreeze=6ms saved=True [05:41] Frame 3 saved (frame_002.jpg, t=4.2s) [05:41] [capture-periodic] freeze=2ms shot=149ms unfreeze=6ms saved=True [05:43] [capture-loop] iter=50 t=6.3s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=3 ram=3.7/15.6GB (24%) swap=0.0/3.0GB (0%) cpu=29% browser_rss=2.05GB [05:49] [capture-loop] iter=100 t=12.1s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=3.6/15.6GB (23%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=2.04GB [05:55] [capture-loop] iter=150 t=17.6s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=3.5/15.6GB (22%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=1.97GB [05:56] [ComfyUI] [worker:ComfyUI-SAM3] [05:56] [ComfyUI] === WATCHDOG TICK 4 === [05:56] [ComfyUI] [worker:ComfyUI-SAM3] Thread 0x00007fbc3e27b6c0 (most recent call first): [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/socket.py", line 293 in accept [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/multiprocessing/connection.py", line 609 in accept [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/multiprocessing/connection.py", line 463 in accept [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/multiprocessing/resource_sharer.py", line 138 in _serve [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 953 in run [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 1016 in _bootstrap_inner [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 973 in _bootstrap [05:56] [ComfyUI] [05:56] [ComfyUI] Current thread 0x00007fbcba3a16c0 (most recent call first): [05:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 52 in _watchdog [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 953 in run [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 1016 in _bootstrap_inner [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 973 in _bootstrap [05:56] [ComfyUI] [05:56] [ComfyUI] Thread 0x00007fbcbae75740 (most recent call first): [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ldm/modules/sub_quadratic_attention.py", line 159 in _get_attention_scores_no_kv_chunking [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ldm/modules/sub_quadratic_attention.py", line 269 in [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ldm/modules/sub_quadratic_attention.py", line 268 in efficient_dot_product_attention [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ldm/modules/attention.py", line 275 in attention_sub_quad [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ldm/modules/attention.py", line 139 in wrapper [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/attention.py", line 120 in sam3_attention [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 729 in forward [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 819 in forward [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 983 in forward [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 2041 in forward [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 3321 in _forward_image_no_act_ckpt [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 3314 in forward_image [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/utils.py", line 1324 in set_image [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124 in decorate_context [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/segmentation.py", line 733 in execute [05:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy_api/latest/_io.py", line 1833 in EXECUTE_NORMALIZED [05:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 1762 in main [05:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 1807 in [05:56] [ComfyUI] [05:56] [ComfyUI] [worker:ComfyUI-SAM3] === END === [05:56] [ComfyUI] [06:00] [capture-loop] iter=200 t=23.4s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=3 ram=3.7/15.6GB (24%) swap=0.0/3.0GB (0%) cpu=35% browser_rss=1.97GB [06:06] [capture-loop] iter=250 t=29.1s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=3 ram=3.4/15.6GB (22%) swap=0.0/3.0GB (0%) cpu=34% browser_rss=1.89GB [06:12] [capture-loop] iter=300 t=34.7s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=3.6/15.6GB (23%) swap=0.0/3.0GB (0%) cpu=34% browser_rss=1.89GB [06:18] [capture-loop] iter=350 t=40.5s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=5.3/15.6GB (34%) swap=0.0/3.0GB (0%) cpu=35% browser_rss=1.89GB [06:23] [capture-loop] iter=400 t=46.1s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=3.3/15.6GB (21%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=1.89GB [06:29] [capture-loop] iter=450 t=51.9s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=3 ram=3.4/15.6GB (22%) swap=0.0/3.0GB (0%) cpu=34% browser_rss=1.89GB [06:34] [capture-loop] iter=500 t=57.4s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=3.3/15.6GB (21%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=1.89GB [06:40] [capture-loop] iter=550 t=63.2s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=3 ram=3.5/15.6GB (23%) swap=0.0/3.0GB (0%) cpu=34% browser_rss=1.89GB [06:46] [capture-loop] iter=600 t=68.8s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=3.6/15.6GB (23%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=1.89GB [06:51] [capture-loop] iter=650 t=74.5s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=3 ram=3.6/15.6GB (23%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=1.89GB [06:56] [ComfyUI] [worker:ComfyUI-SAM3] [06:56] [ComfyUI] === WATCHDOG TICK 5 === [06:56] [ComfyUI] [worker:ComfyUI-SAM3] Thread 0x00007fbc3e27b6c0 (most recent call first): [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/socket.py", line 293 in accept [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/multiprocessing/connection.py", line 609 in accept [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/multiprocessing/connection.py", line 463 in accept [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/multiprocessing/resource_sharer.py", line 138 in _serve [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 953 in run [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 1016 in _bootstrap_inner [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 973 in _bootstrap [06:56] [ComfyUI] [06:56] [ComfyUI] Current thread 0x00007fbcba3a16c0 (most recent call first): [06:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 52 in _watchdog [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 953 in run [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 1016 in _bootstrap_inner [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 973 in _bootstrap [06:56] [ComfyUI] [06:56] [ComfyUI] Thread 0x00007fbcbae75740 (most recent call first): [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ops.py", line 475 in forward_comfy_cast_weights [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ops.py", line 482 in forward [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 740 in forward [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 819 in forward [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 983 in forward [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 2041 in forward [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 3321 in _forward_image_no_act_ckpt [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 3314 in forward_image [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/utils.py", line 1324 in set_image [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124 in decorate_context [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/segmentation.py", line 733 in execute [06:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy_api/latest/_io.py", line 1833 in EXECUTE_NORMALIZED [06:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 1762 in main [06:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 1807 in [06:56] [ComfyUI] [06:56] [ComfyUI] [worker:ComfyUI-SAM3] === END === [06:56] [ComfyUI] [06:57] [capture-loop] iter=700 t=80.1s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=3 ram=3.5/15.6GB (23%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=1.89GB [07:03] [capture-loop] iter=750 t=85.8s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=3.5/15.6GB (23%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=1.89GB [07:08] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=8 [07:08] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=8 [07:08] [capture-loop] iter=800 t=91.4s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=10 ram=4.0/15.6GB (25%) swap=0.0/3.0GB (0%) cpu=34% browser_rss=1.89GB [07:09] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=8 [07:09] Frame 4 saved (frame_003.jpg, t=91.5s) [07:09] [capture-periodic] freeze=2ms shot=192ms unfreeze=4ms saved=True [07:09] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=8 [07:09] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=8 [07:09] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=8 [07:10] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=8 [07:11] Frame 5 saved (frame_004.jpg, t=93.8s) [07:11] [capture-periodic] freeze=3ms shot=162ms unfreeze=9ms saved=True [07:14] [capture-loop] iter=850 t=97.3s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=6.3/15.6GB (41%) swap=0.0/3.0GB (0%) cpu=41% browser_rss=2.12GB [07:18] [ComfyUI] [SubprocessWorker] got response: ok [07:18] [ComfyUI] [comfy-env] SAM3MultipromptSegmentation done (99.69s): result: NodeOutput [07:20] [capture-loop] iter=900 t=102.9s state={'complete': False, 'error': None, 'executedCount': 1, 'wsState': 1} eval_ms=2 ram=4.6/15.6GB (30%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=2.12GB [07:20] Frame 6 saved (frame_005.jpg, t=102.9s) [07:20] [capture-periodic] freeze=3ms shot=142ms unfreeze=10ms saved=True [07:23] Node executed (2 total), capturing... [07:24] Frame 7 saved (frame_006.jpg, t=105.9s) [07:24] [capture-node] freeze=552ms shot=284ms unfreeze=31ms saved=True [07:26] [ComfyUI] BatchToMultiband: Processed 1 images with 3 channels each -> 3 channels [07:26] [ComfyUI] BatchToMultiband: Processed 7 masks -> 7 channels [07:26] Node executed (3 total), capturing... [07:26] [ComfyUI] BatchToMultiband: Output shape torch.Size([1, 10, 4480, 6720]), channels: ['img_01_r', 'img_01_g', 'img_01_b', 'mask_01', 'mask_02', 'mask_03', 'mask_04', 'mask_05', 'mask_06', 'mask_07'] [07:26] [ComfyUI] ResizeMultiband: 4480x6720 -> 600x900 [07:26] [ComfyUI] PreviewMultibandImage: Batch=1, Channels=10, Names=['img_01_r', 'img_01_g', 'img_01_b', 'mask_01', 'mask_02', 'mask_03', 'mask_04', 'mask_05', 'mask_06', 'mask_07'] [07:27] [ComfyUI] SaveMultibandImage: Saved to output/multiband.npz [07:27] [ComfyUI] Prompt executed in 109.74 seconds [07:27] Frame 8 saved (frame_007.jpg, t=109.0s) [07:28] [ComfyUI] Shape: (1, 10, 600, 900) [07:28] [capture-node] freeze=154ms shot=1009ms unfreeze=6ms saved=True [07:28] [ComfyUI] Format: npz [07:28] Node executed (4 total), capturing... [07:29] Frame 9 saved (frame_008.jpg, t=111.5s) [07:29] [capture-node] freeze=2ms shot=180ms unfreeze=0ms saved=True [07:29] Execution complete (t=111.5s) [07:29] Captured 9 unique frames over 111.68s [07:29] [debug] iframes: 0 [07:29] [timing] trigger_3d_previews: 0.0s [07:31] Saved high-quality screenshot: scene_seg_executed.png [07:31] [timing] final_screenshot: 0.3s [07:31] Capturing execution frames: scene_seg_interactive.json [07:38] Frame 1 saved (frame_000.jpg, t=0.0s) [07:38] Validating workflow... [07:38] [ComfyUI] === /validate endpoint called === [07:38] [ComfyUI] Received JSON with keys: ['prompt'] [07:38] [ComfyUI] Prompt has 4 nodes [07:38] [ComfyUI] Node 1: LoadImage [07:38] [ComfyUI] Inputs: {"image": "image.png"} [07:38] [ComfyUI] Node 4: MaskPreview [07:38] [ComfyUI] Inputs: {"mask": ["26", 0]} [07:38] [ComfyUI] Node 12: LoadSAM3Model [07:38] [ComfyUI] Inputs: {"precision": "auto", "compile": false} [07:38] [ComfyUI] Node 26: SAM3InteractiveCollector [07:38] [ComfyUI] Inputs: {"multi_prompts_store": "[{\"positive_points\":[{\"x\":3348.288109906738,\"y\":2452.2208696844737}],\"negative_points\":[],\"positive_boxes\":[],\"negative_boxes\":[],\"name\":\"Prompt 1\"},{\"positiv [07:38] [ComfyUI] Generated prompt_id: ffdbf462-d0ad-4ca6-b470-162045d8f652 [07:38] [ComfyUI] Calling execution.validate_prompt()... [07:38] [ComfyUI] Validation result: valid=True [07:38] [ComfyUI] === Validation PASSED === [07:38] Queuing workflow for execution... [07:38] [ComfyUI] got prompt [07:38] [ComfyUI] [comfy-env] Running LoadSAM3Model: precision: 'auto', compile: False [07:38] [ComfyUI] [SubprocessWorker] call_method: nodes.load_model.LoadSAM3Model.EXECUTE_NORMALIZED [07:38] [ComfyUI] [SubprocessWorker] acquired lock, ensuring started... [07:38] [ComfyUI] [ComfyUI-SAM3] Health check: ping (timeout=5.0s)... [07:38] [ComfyUI] [ComfyUI-SAM3] Health check: ok [07:38] [ComfyUI] [SubprocessWorker] worker started/confirmed [07:38] [ComfyUI] [SubprocessWorker] serializing kwargs to shm... [07:38] [ComfyUI] [SubprocessWorker] created 0 shm blocks [07:38] [ComfyUI] [SubprocessWorker] sending request via socket... [07:38] [ComfyUI] [SubprocessWorker] call_id=9 sending call_method [07:38] [ComfyUI] [SubprocessWorker] got response: ok [07:38] [ComfyUI] [comfy-env] LoadSAM3Model done (0.02s): result: NodeOutput [07:38] [ComfyUI] [comfy-env] Running SAM3InteractiveCollector: multi_prompts_store: '[{"positive_points":[{"x":3348.288109906738,"y":2452.220..., sam3_model_config: dict, image: torch.float32 [1,4480,6720,3] cpu, unique_id: '26' [07:38] [ComfyUI] [SubprocessWorker] call_method: nodes.sam3_interactive.SAM3InteractiveCollector.EXECUTE_NORMALIZED [07:38] [ComfyUI] [SubprocessWorker] acquired lock, ensuring started... [07:38] [ComfyUI] [ComfyUI-SAM3] Health check: ping (timeout=5.0s)... [07:38] [ComfyUI] [ComfyUI-SAM3] Health check: ok [07:38] [ComfyUI] [SubprocessWorker] worker started/confirmed [07:38] [ComfyUI] [comfy-env] PRE-SERIALIZE 'image' shape: torch.Size([1, 4480, 6720, 3]) [07:38] [ComfyUI] [SubprocessWorker] serializing kwargs to shm... [07:38] [ComfyUI] [SubprocessWorker] created 0 shm blocks [07:38] [ComfyUI] [SubprocessWorker] sending request via socket... [07:38] [ComfyUI] [SubprocessWorker] call_id=10 sending call_method [07:38] [ComfyUI] [worker:ComfyUI-SAM3] [comfy-env] _from_shm got dict with keys: ['__type__', 'strategy', 'parent_pid', 'fd', 'storage_size'] [07:38] [ComfyUI] [worker:ComfyUI-SAM3] [comfy-env] DESERIALIZE TensorRef: tensor_size=[1, 4480, 6720, 3] [07:38] [ComfyUI] [worker:ComfyUI-SAM3] [comfy-env] DESERIALIZED tensor shape: torch.Size([1, 4480, 6720, 3]) [07:39] [ComfyUI] [SubprocessWorker] callback 'request_vram_budget' from call_id=10 [07:39] [ComfyUI] [comfy-env] VRAM request: 3.50GB, free before eviction: 12.26GB [07:39] [ComfyUI] [comfy-env] VRAM after eviction: 12.26GB [07:40] Frame 2 saved (frame_001.jpg, t=2.2s) [07:40] [capture-periodic] freeze=2ms shot=96ms unfreeze=3ms saved=True [07:42] Frame 3 saved (frame_002.jpg, t=4.5s) [07:42] [capture-periodic] freeze=2ms shot=115ms unfreeze=3ms saved=True [07:43] [capture-loop] iter=50 t=5.7s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=4.5/15.6GB (29%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=1.44GB [07:49] [capture-loop] iter=100 t=11.4s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=2 ram=4.3/15.6GB (27%) swap=0.0/3.0GB (0%) cpu=34% browser_rss=1.34GB [07:54] [capture-loop] iter=150 t=16.9s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=4.2/15.6GB (27%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=1.30GB [07:56] [ComfyUI] [worker:ComfyUI-SAM3] [07:56] [ComfyUI] === WATCHDOG TICK 6 === [07:56] [ComfyUI] [worker:ComfyUI-SAM3] Thread 0x00007fbc3e27b6c0 (most recent call first): [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/socket.py", line 293 in accept [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/multiprocessing/connection.py", line 609 in accept [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/multiprocessing/connection.py", line 463 in accept [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/multiprocessing/resource_sharer.py", line 138 in _serve [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 953 in run [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 1016 in _bootstrap_inner [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 973 in _bootstrap [07:56] [ComfyUI] [07:56] [ComfyUI] Current thread 0x00007fbcba3a16c0 (most recent call first): [07:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 52 in _watchdog [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 953 in run [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 1016 in _bootstrap_inner [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 973 in _bootstrap [07:56] [ComfyUI] [07:56] [ComfyUI] Thread 0x00007fbcbae75740 (most recent call first): [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ldm/modules/sub_quadratic_attention.py", line 159 in _get_attention_scores_no_kv_chunking [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ldm/modules/sub_quadratic_attention.py", line 269 in [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ldm/modules/sub_quadratic_attention.py", line 268 in efficient_dot_product_attention [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ldm/modules/attention.py", line 275 in attention_sub_quad [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ldm/modules/attention.py", line 139 in wrapper [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/attention.py", line 120 in sam3_attention [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 729 in forward [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 819 in forward [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 983 in forward [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 2041 in forward [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 3321 in _forward_image_no_act_ckpt [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 3314 in forward_image [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/utils.py", line 1324 in set_image [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124 in decorate_context [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3_interactive.py", line 687 in execute [07:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy_api/latest/_io.py", line 1833 in EXECUTE_NORMALIZED [07:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 1762 in main [07:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 1807 in [07:56] [ComfyUI] [07:56] [ComfyUI] [worker:ComfyUI-SAM3] === END === [07:56] [ComfyUI] [08:00] [capture-loop] iter=200 t=22.4s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=2 ram=4.4/15.6GB (28%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=1.30GB [08:05] [capture-loop] iter=250 t=28.0s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=4.4/15.6GB (28%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=1.30GB [08:11] [capture-loop] iter=300 t=33.5s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=2 ram=4.3/15.6GB (28%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=1.30GB [08:16] [capture-loop] iter=350 t=39.0s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=4.2/15.6GB (27%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=1.30GB [08:22] [capture-loop] iter=400 t=44.6s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=4.3/15.6GB (27%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=1.30GB [08:27] [capture-loop] iter=450 t=50.1s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=2 ram=4.2/15.6GB (27%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=1.30GB [08:33] [capture-loop] iter=500 t=55.7s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=2 ram=4.3/15.6GB (27%) swap=0.0/3.0GB (0%) cpu=33% browser_rss=1.30GB [08:39] [capture-loop] iter=550 t=61.2s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=5.6/15.6GB (36%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=1.30GB [08:44] [capture-loop] iter=600 t=66.8s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=2 ram=4.3/15.6GB (28%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=1.30GB [08:50] [capture-loop] iter=650 t=72.2s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=4.4/15.6GB (28%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=1.30GB [08:55] [capture-loop] iter=700 t=77.8s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=2 ram=4.4/15.6GB (28%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=1.30GB [08:56] [ComfyUI] [worker:ComfyUI-SAM3] [08:56] [ComfyUI] === WATCHDOG TICK 7 === [08:56] [ComfyUI] [worker:ComfyUI-SAM3] Thread 0x00007fbc3e27b6c0 (most recent call first): [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/socket.py", line 293 in accept [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/multiprocessing/connection.py", line 609 in accept [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/multiprocessing/connection.py", line 463 in accept [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/multiprocessing/resource_sharer.py", line 138 in _serve [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 953 in run [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 1016 in _bootstrap_inner [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 973 in _bootstrap [08:56] [ComfyUI] [08:56] [ComfyUI] Current thread 0x00007fbcba3a16c0 (most recent call first): [08:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 52 in _watchdog [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 953 in run [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 1016 in _bootstrap_inner [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/threading.py", line 973 in _bootstrap [08:56] [ComfyUI] [08:56] [ComfyUI] Thread 0x00007fbcbae75740 (most recent call first): [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ops.py", line 475 in forward_comfy_cast_weights [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy/ops.py", line 482 in forward [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 147 in forward [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 823 in forward [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 983 in forward [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 2041 in forward [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 3321 in _forward_image_no_act_ckpt [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/model.py", line 3314 in forward_image [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3/utils.py", line 1324 in set_image [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/.ce/.pixi/envs/sam3-nodes/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124 in decorate_context [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/custom_nodes/ComfyUI-SAM3/nodes/sam3_interactive.py", line 687 in execute [08:56] [ComfyUI] File "/home/runner/work/comfy-gpu-ci/comfy-gpu-ci/test_workspaces/SAM3-2028/ComfyUI/comfy_api/latest/_io.py", line 1833 in EXECUTE_NORMALIZED [08:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 1762 in main [08:56] [ComfyUI] File "/tmp/comfyui_pvenv_bmca7xhm/persistent_worker.py", line 1807 in [08:56] [ComfyUI] [08:56] [ComfyUI] [worker:ComfyUI-SAM3] === END === [08:56] [ComfyUI] [09:01] [capture-loop] iter=750 t=83.3s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=4.3/15.6GB (28%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=1.30GB [09:06] [capture-loop] iter=800 t=88.9s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=4.4/15.6GB (28%) swap=0.0/3.0GB (0%) cpu=32% browser_rss=1.30GB [09:07] [ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=10 [09:07] Frame 4 saved (frame_003.jpg, t=89.8s) [09:07] [capture-periodic] freeze=3ms shot=115ms unfreeze=5ms saved=True [09:09] [ComfyUI] [SubprocessWorker] got response: ok [09:09] [ComfyUI] [comfy-env] SAM3InteractiveCollector done (90.62s): result: NodeOutput [09:09] [ComfyUI] Prompt executed in 91.86 seconds [09:10] Node executed (1 total), capturing... [09:11] Frame 5 saved (frame_004.jpg, t=92.5s) [09:11] [capture-node] freeze=141ms shot=504ms unfreeze=5ms saved=True [09:11] Node executed (2 total), capturing... [09:11] [capture-node] freeze=2ms shot=158ms unfreeze=0ms saved=False [09:11] Execution complete (t=93.4s) [09:11] Captured 5 unique frames over 93.6s [09:11] [debug] iframes: 0 [09:11] [timing] trigger_3d_previews: 0.0s [09:13] Saved high-quality screenshot: scene_seg_interactive_executed.png [09:13] [timing] final_screenshot: 0.3s [09:14] [5/5] SKIPPED (not in cpu list) video_point_prompt.json [09:14] Results saved to /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/comfy-test-logs/SAM3-2028/dev/linux-cpu/results.json [09:14] Model report: 1 files, 3.2 GB -> /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/comfy-test-logs/SAM3-2028/dev/linux-cpu/models.json [09:14] Saved: /home/runner/work/comfy-gpu-ci/comfy-gpu-ci/comfy-test-logs/SAM3-2028/dev/linux-cpu/index.html [09:14] [EXECUTION] PASSED [09:14] linux: PASSED [09:14] Stopping ComfyUI server...