sammykumar-ComfyUI-SwissArmyKnife/dev-docs/web-js/RESOURCE_MONITOR.md

Resource Monitor - Floating Heads-Up Monitor Bar

Status: Phase 2 frontend + restart flow complete Component: Web JavaScript Extension (web/js/resource_monitor.js)

Overview

The Resource Monitor extension draws a floating glass "heads-up" bar in the ComfyUI viewport that streams live CPU/RAM/GPU telemetry and exposes an always-visible Restart button. The widget detaches from the legacy dropdown entry in swiss-army-knife.js so that operators can trigger a restart while simultaneously watching resource pressure build in real time.

Key Capabilities:

• Fixed, centered monitor bar injected directly into document.body (not the top toolbar)
• Inline CPU, RAM, temperature, GPU utilization, and VRAM stats rendered with gradient fills
• Automatic discovery of the hardware that is actually available from the backend
• Fully wired restart workflow: confirmation toasts, /swissarmyknife/restart POST, /system_stats health polling, and cache-busting reload
• WebSocket live updates via api.addEventListener("swissarmyknife.monitor", handler) so the UI stays in sync after the initial REST fetch

File Layout

web/
├── js/
│   ├── resource_monitor.js   # Floating bar + restart workflow (this document)
│   └── swiss-army-knife.js   # Legacy dropdown restart entry (still shipped but redundant)
└── css/
    └── restart-button.css    # Historical styles, no longer referenced

Extension Registration & Setup Flow

1. app.registerExtension({ name: "comfyui_swissarmyknife.resource_monitor", setup }) wires the feature into ComfyUI.
2. During setup() the extension:
   • Injects the style tag #swissarmyknife-restart-button-styles that defines the floating glassmorphism wrapper, gradient logic, and button micro-interactions.
   • Performs an initial fetch("/swissarmyknife/monitor/status") to understand which monitors should render.
   • Dynamically builds monitor cells through createMonitorDisplay(label, id) based on backend capabilities (psutil / pynvml availability, detected GPUs, exposed temperatures, etc.).
   • Appends the Restart button returned from createRestartButton() after the telemetry cells.
   • Appends the completed wrapper to document.body so it floats independently of the toolbar DOM hierarchy.
   • Subscribes to swissarmyknife.monitor events on the ComfyUI API bridge, which surface the backend push updates every ~2 seconds.

UI Layout & Styling

The wrapper #swissarmyknife-resource-monitor is positioned fixed at top: 50px and horizontally centered with left: 50% + transform: translateX(-50%). It uses a translucent background, blur filter, rounded 16px corners, and an inset highlight so it reads like a lightweight HUD.

Each telemetry cell (.swissarmyknife-monitor) displays LABEL VALUE pairs:

• CPU % and RAM (GB used) render whenever psutil is available.
• CPU temperature renders when exposed by the backend sensors.
• For every GPU device the backend reports as available, the UI renders utilization %, VRAM usage, and (optionally) GPU temp rows. Labels include a compact per-GPU suffix (e.g., GPU 0 (3090Ti), VRAM 0 (A6000)), generated by extractGPUModel().
• Cells call updateMonitorValue(id, value, percent) whenever new data arrives. The helper paints a linear gradient from left to right using thresholds <50% (green #22c55e), <70% (yellow #eab308), <90% (orange #f97316), and >=90% (red #ef4444) to immediately communicate pressure.

Workflow Execution Glow

When a workflow starts running (execution_start event) the monitor wrapper gains the swissarmyknife-is-executing class. CSS registered via @property --swissarmyknife-gradient-angle and the swissarmyknife-resource-monitor-rotation keyframes animates a rotating multi-color gradient on paired pseudo-elements: ::before renders a soft blurred plume outside the pill and ::after masks the gradient into a thin border ring so the fill stays untouched. The gradient now cycles across six hues (blue → purple → magenta → orange → gold → jade) to avoid any single color dominating the rim. The telemetry tiles live inside #swissarmyknife-resource-monitor-inner, an inset flex wrapper that introduces a small gutter so the animated halo never collides with the per-metric progress bars. The class is removed on execution_success, execution_error, execution_cancelled, or execution_end so the glow shuts off the instant a workflow finishes or fails.

The Restart button shares the wrapper so the operator can watch the monitors while the restart is underway. It follows the ComfyUI .comfyui-button base class with custom danger colors, disabled state, and a divider pseudo-element (::before) to visually separate it from the monitors.

Restart Workflow

createRestartButton() wires the full restart experience:

1. Disable + relabel the button to "Restarting..." and push a Toast (app.extensionManager.toast.add) describing what is happening.
2. Fire fetch("/swissarmyknife/restart", { method: "POST" }). Any network failure is ignored because the backend tears down the HTTP connection during restart.
3. Wait 2 seconds for shutdown to start, then poll /system_stats every second (up to 60s) until a 200 OK proves the server is alive again.
4. When the health check succeeds, pop a success toast and hard reload window.location.reload(true) to pick up new JS bundles without a stale cache.
5. If the timeout expires, re-enable the button and warn the operator that manual refresh might be required.
6. All errors propagate to the console with the [SwissArmyKnife][ResourceMonitor] prefix and surface a Toast that explains the failure.

The behavior mirrors the menu command restart logic, but the inline placement plus stateful button make the process much more discoverable.

Data Sources & Update Loop

• Initial snapshot: GET /swissarmyknife/monitor/status returns { hardware, gpu }. The UI uses this payload to decide which tiles to render (e.g., no psutil => only GPU tiles, no pynvml => skip utilization tiles, etc.).
• Live updates: The backend broadcasts swissarmyknife.monitor events. handleMonitorUpdate() consumes the payload, runs the formatters (formatGB, formatPercent, formatTemp, formatGHz) and updates every matching tile.
• Health checks: Restart logic pings /system_stats because it is small, unauthenticated, and always hosted by ComfyUI itself.
• Debug logs: Controlled by SwissArmyKnife.debug_mode. When enabled the console prints setup/start/end banners, AJAX failures, and periodic polling output (every 5 attempts during restart polling).

Dependency Matrix & Graceful Degradation

Backend Dependency	UI Impact
psutil	CPU %, RAM, CPU Temp tiles render. Without it, those tiles are simply not created.
py-cpuinfo	Used server-side to enrich CPU labels; UI automatically falls back to -- when a value is missing.
torch	Enables VRAM reporting when pynvml is absent. The UI still displays VRAM rows because the backend formats the bytes.
pynvml	Unlocks GPU utilization and temperature metrics. Without it only VRAM tiles (and restart) appear.

No JavaScript errors are thrown when a capability is missing; the UI just renders the tiles that make sense for the current machine.

Browser Refresh Requirements

Because this is a pure JavaScript widget, no ComfyUI restart is required. Developers should perform a hard refresh (Ctrl/Cmd + Shift + R) whenever resource_monitor.js changes to force the new bundle to load and to re-inject the CSS.

Roadmap

Near-Term Enhancements

• Clear VRAM button and backend endpoint wiring
• Per-tile context menus (click to copy usage, open GPU inspector, etc.)
• Visibility presets so laptop users can hide GPU tiles
• Historical sparklines per metric
• Settings surface for sampling interval + toast verbosity

ProfilerX Parity Plan

To reach the level of workflow profiling offered by ComfyUI ProfilerX, we will deliver the following phases with concrete implementation hooks learned from their codebase.

1. Instrumentation Layer

   • Mirror ProfilerX’s prestartup.py shim: wrap execution.execute, PromptExecutor.execute, and ExecutionList.__init__ so we can call a ProfilerManager analogue that records start_workflow, start_node, end_node, and end_workflow events.
   • Track cache hits by checking caches.outputs[current_item] before/after execution just like ProfilerX’s wrappers do, and collect tensor shapes via helpers similar to _get_tensor_sizes.
   • Reset CUDA peak stats per node (torch.cuda.reset_peak_memory_stats) to capture accurate VRAM deltas, persist RAM via psutil.Process().memory_info().rss, and roll running averages for nodes/workflows.
   • Keep a short-lived active_profiles dict plus a disk-backed history array, batching writes (ProfilerX saves every 5 workflows) so we don’t thrash the filesystem.

2. Profiler API Surface

   • Expose REST endpoints modeled after ProfilerX’s server.py: /swissarmyknife/profiler/stats, /archives, /archive/:id/load, /archive/:id (DELETE). Responses should include current, latest, node_averages, workflow_averages, and a bounded history.
   • Support archive rotation by dumping JSON snapshots into data/archives/ and deleting them after loading—this matches ProfilerX’s archive_history, load_archive, and delete_archive.
   • Emit monitor broadcasts (existing swissarmyknife.monitor) plus a new profiler-specific event whenever a workflow completes so the UI can refresh instantly.

3. Profiler Dashboard UI

   • Attach a menu button the way ProfilerX does in web/index.ts: locate .comfyui-menu-right, create a .comfyui-button-group, and inject a button that toggles a detailed popup.
   • Reuse their approach of listening to app.addEventListener("executed") and custom profiler:historyLoaded events so tabs stay in sync after archives load.
   • Implement tabbed content similar to web/ui/tabs.ts, with menu-level mini stats, per-node sortable tables, cache hit/miss breakdown, and inline charts (SVG or canvas) fed from the REST payloads.

4. Historical Storage & Analytics

   • Persist up to ~10k workflow entries on disk (same order-of-magnitude as ProfilerX’s max_history) with auto-archiving when limits are hit, plus asynchronous saves via a background executor.
   • Augment each record with executionOrder, averages, tensor sizes, and cacheHits/Misses so we can compute the same aggregates ProfilerX exposes.
   • Build archive management UI (list + load + delete) mirroring ProfilerX’s profilerx-stats-popup so operators can prune or revisit past runs without leaving the browser.

5. Advanced Execution Tracking

   • Once Profiler parity lands, add an opt-in tracker patterned after execution_core.ExecutionTracker: wrap internal ComfyUI methods with decorators that push method names, durations, and stack depth into method_traces.json.
   • Stream summarized trace metadata to the UI so a “Deep Trace” tab can highlight slow internals without forcing users to open raw JSON files.
   • Surface a settings toggle (Swiss Army Knife config) that sets ExecutionTracker.ENABLED instead of requiring code edits, matching ProfilerX’s optional tracing workflow.

Keep the file-level documentation synchronized with each feature drop so operators know which telemetry to expect.

Profiler Implementation Status

Last Updated: November 28, 2025 Status: ✅ Phase 1-3 Complete | 🚧 Phase 4-5 In Progress

Implemented Features

✅ Phase 1: Backend Instrumentation

• Prestartup hook system (nodes/utils/resource_monitor/prestartup.py)
• ProfilerManager singleton with workflow/node tracking
• VRAM/RAM measurement with graceful degradation
• Cache hit detection and tensor shape tracking
• Batched async saves (every 5 workflows)
• Auto-archiving at 10k workflow limit

✅ Phase 2: REST API & WebSocket

• GET /swissarmyknife/profiler/stats - Latest stats and history
• GET /swissarmyknife/profiler/archives - List archives
• POST /swissarmyknife/profiler/archive - Create archive
• POST /swissarmyknife/profiler/archive/:filename/load - Load archive
• DELETE /swissarmyknife/profiler/archive/:filename - Delete archive
• WebSocket broadcasts on workflow completion

✅ Phase 3: Profiler UI

• Profiler button (📊) in floating resource monitor
• Glassmorphic popup with:
  • Circular progress gauge
  • 6-card stats grid (Time, VRAM, RAM, Cache, Nodes). The Total Time card now renders long runs in minutes+seconds (e.g., 5m 50.9s) while sub-minute workflows stay in seconds for better readability.
  • Top 10 slowest nodes table
  • "View Full History" and "Clear History" buttons
• Full-screen modal with 4 tabs (Latest, Previous, Analytics, Settings)
• Real-time WebSocket updates
• Auto-refresh on workflow execution

Zero-Config Design

Profiler is enabled by default with:

• No user configuration required
• Automatic startup integration
• Graceful degradation when dependencies missing
• < 1% execution overhead

Usage

1. Click 📊 profiler button in floating monitor for quick stats
2. Click "View Full History" for detailed modal analysis
3. Use Settings tab for archive management

Known Limitations

VRAM Tracking

• Requires NVIDIA CUDA: VRAM tracking only works on systems with CUDA-enabled GPUs (PyTorch CUDA API)
• CPU-only mode: No VRAM tracking available when running on CPU-only systems (logs warning at startup)
• AMD/Intel GPUs: VRAM tracking not supported (PyTorch CUDA-specific implementation)
• Allocation timing: Captures memory at node execution boundaries only, not internal allocations during node execution
• Shared memory: Cannot distinguish between node-specific and globally shared VRAM usage across the process

RAM Tracking

• Requires psutil: RAM tracking disabled if psutil package not installed
• Process-level only: Measures total process memory, not per-node isolation
• System overhead: Includes Python interpreter and ComfyUI framework overhead in measurements

Cache Hit Detection

• Not yet implemented: Currently always reports cache_hit=False for all node executions
• Planned enhancement: Future integration with ComfyUI's cache system to detect actual cache hits

Tensor Shape Tracking

• Requires PyTorch: Only available when torch package is installed
• Limited types: Only tracks torch.Tensor objects in inputs/outputs
• Nested structures: May miss tensors deeply nested in complex data structures

Timing Accuracy

• Async operations: May not capture accurate timing for nodes with async GPU operations that return before completion
• Python GIL: Subject to Python Global Interpreter Lock overhead
• System scheduling: Wall-clock timing affected by system scheduling and other processes

History & Storage

• Auto-archive limit: Automatically archives after 10,000 workflows to prevent unbounded disk growth
• Disk I/O: Batched saves (every 5 workflows) may delay persistence during crashes
• Memory footprint: Active profiling adds ~500KB-2MB per workflow in memory before archiving

Hardware Detection

• Best-effort detection: Dependency availability checked at startup only, not dynamically
• Silent degradation: Missing dependencies log warnings but don't fail loudly to user
• CUDA initialization: First CUDA call may add latency during ProfilerManager initialization

Troubleshooting

VRAM shows as 0 or null:

• Verify CUDA is available: Check console for "CUDA available: True" message
• Ensure GPU is NVIDIA with CUDA support
• Confirm PyTorch installed with CUDA support: python -c "import torch; print(torch.cuda.is_available())"

RAM tracking not working:

• Install psutil: pip install psutil
• Restart ComfyUI server after installation

Profiler not capturing data:

• Check console for "[SwissArmyKnife][Profiler]" messages
• Verify profiler enabled in settings (enabled by default)
• Look for warnings about dependency availability

Next Steps

• Enhanced archive management UI (Phase 4)
• Historical charts and sparklines
• Advanced execution tracking (Phase 5)
• Implement cache hit detection
• OOM diagnosis and prevention (Phase 6)

---

OOM Diagnosis & Prevention

Status: 🚧 Phase 6 - Planned Purpose: Detect, capture, and prevent Out of Memory errors through unified profiler architecture

Overview

The OOM diagnosis system extends the profiler with pre-OOM warnings, exception capture, historical analysis, and model unload recommendations. This prevents memory exhaustion failures by combining threshold-based alerts with actionable insights.

Key Capabilities

• Pre-OOM Detection: Threshold warnings at 85% (⚠️ yellow) and 95% (🔴 red) VRAM usage
• OOM Event Capture: Automatic detection with full context (VRAM state, loaded models, timeline)
• Historical Analysis: Persistent tracking with node type ranking and model correlation
• Model Attribution: Correlate OOM failures with specific loaded model combinations
• Unload Recommendations: Suggest which models to unload based on size and usage patterns

Architecture Data Flow

1. PRE-NODE EXECUTION (start_node)
   ↓ Check free VRAM via torch.cuda.mem_get_info()
   ↓ Calculate vram_percent_before = (total-free)/total*100
   ↓ Log ⚠️  if >85%, 🔴 if >95%
   ↓ Store vram_free_before, vram_percent_before

2. NODE EXECUTION (map_node_with_profiling)
   ↓ Execute node with full profiling
   ↓ Track VRAM delta, peak, timeline
   ↓ Catch RuntimeError exceptions
   ↓ (if OOM occurs)

3. OOM CAPTURE (exception handler)
   ↓ Detect "out of memory" in error message
   ↓ Capture torch.cuda.memory_summary()
   ↓ Snapshot loaded models via _scan_gpu_models()
   ↓ Record vram_at_oom, models_at_oom, oom_error
   ↓ Mark node.oom_occurred = True

4. HISTORICAL TRACKING (end_workflow)
   ↓ Append OOM event to oom_history list
   ↓ Update OOM frequency stats per node type
   ↓ Calculate model correlation patterns
   ↓ Archive OOM data with workflow history

5. ANALYSIS & RECOMMENDATIONS
   ↓ GET /swissarmyknife/profiler/oom_stats
   ↓ Node type OOM ranking (frequency, avg VRAM)
   ↓ Model correlation (which models present at OOM)
   ↓ Unload recommendations (by size + last-used)

Extended Data Models

NodeProfile OOM Fields

class NodeProfile:
    # Existing fields...
    oom_occurred: bool = False              # True if this node caused OOM
    oom_error: Optional[str] = None         # Full exception message
    vram_free_before: Optional[int] = None  # Free VRAM before execution (bytes)
    vram_percent_before: Optional[float] = None  # VRAM usage % before execution
    vram_at_oom: Optional[int] = None       # Allocated VRAM when OOM occurred
    models_at_oom: Optional[List[Dict]] = None   # Loaded models snapshot at OOM

WorkflowProfile OOM Fields

class WorkflowProfile:
    # Existing fields...
    oom_occurred: bool = False           # True if any node OOM'd
    oom_node_id: Optional[str] = None    # Which node caused OOM

ProfilerManager OOM Tracking

class ProfilerManager:
    # Existing fields...
    oom_history: List[Dict] = []         # Separate OOM event log
    oom_stats: Dict[str, Dict] = {}      # Per-node-type OOM statistics
    max_oom_history: int = 1000          # Limit OOM history size

Pre-OOM Threshold Warnings

Thresholds:

• 85% - ⚠️ Warning (Yellow): VRAM approaching limits, monitor closely
• 95% - 🔴 Critical (Red): VRAM critically high, OOM highly likely

Implementation in ProfilerManager.start_node():

# Get free VRAM before node execution
free_vram, total_vram = torch.cuda.mem_get_info()
node_profile.vram_free_before = free_vram
node_profile.vram_percent_before = (total_vram - free_vram) / total_vram * 100

# Threshold warnings
if node_profile.vram_percent_before >= 95:
    logger.error(f"🔴 CRITICAL: VRAM at {vram_percent:.1f}% before {node_type} - OOM likely!")
elif node_profile.vram_percent_before >= 85:
    logger.warning(f"⚠️  WARNING: VRAM at {vram_percent:.1f}% before {node_type}")

Example Console Output:

⚠️  WARNING: VRAM at 87.3% before WanVideoWrapper (free: 2.8 GB / 22.0 GB)
🔴 CRITICAL: VRAM at 96.1% before WanVideoWrapper (free: 0.9 GB / 22.0 GB) - OOM likely!

OOM Exception Capture

Detection in map_node_with_profiling() exception handler:

try:
    result = await original_map_node_over_list(...)
    profiler.end_node(prompt_id, unique_id, outputs=result)
    return result
except RuntimeError as e:
    # Detect OOM specifically
    is_oom = "out of memory" in str(e).lower()
    
    if is_oom and profiler_enabled:
        node = profile.nodes[unique_id]
        node.oom_occurred = True
        node.oom_error = str(e)
        node.vram_at_oom = torch.cuda.memory_allocated()
        node.models_at_oom = profiler._scan_gpu_models()
        
        # Log models present at OOM
        logger.error(f"💥 OOM in {class_type}: {vram_at_oom / 1024**3:.1f} GB allocated")
        for gpu_id, models in node.models_at_oom.items():
            for model in models:
                logger.error(f"  - {model['name']} ({model['type']}): {model['vram_mb']} MB")
    raise

Captured Context:

• Exception message (full traceback)
• VRAM allocated at OOM moment
• GPU memory summary (PyTorch internal state)
• Complete loaded models snapshot with VRAM per model
• Node execution timeline
• VRAM headroom before node started

OOM Event Structure

{
    "timestamp": "2025-11-29T14:23:45.123Z",
    "prompt_id": "f8ad0b31-1717-4878-aad1-064d5df212d9",
    "node_id": "42",
    "node_type": "WanVideoWrapper",
    "execution_time_before_oom": 135.41,
    "vram_percent_before": 87.3,
    "vram_free_before": 2899102720,
    "vram_at_oom": 23622320128,
    "models_at_oom": {
        "0": [
            {
                "name": "Wan2_2-T2V-A14B-HIGH",
                "type": "checkpoint",
                "vram_mb": 8704
            },
            {
                "name": "sd3_medium",
                "type": "checkpoint",
                "vram_mb": 6144
            }
        ]
    },
    "error_message": "torch.OutOfMemoryError: Allocation on device",
    "total_models_vram_mb": 14848
}

New API Endpoint

GET /swissarmyknife/profiler/oom_stats

Returns OOM statistics and analysis:

{
    "success": true,
    "data": {
        "total_oom_count": 47,
        "oom_rate": 12.3,
        "node_type_ranking": [
            {
                "node_type": "WanVideoWrapper",
                "oom_count": 23,
                "oom_rate": 48.9,
                "avg_vram_at_oom_gb": 21.8,
                "avg_vram_percent_before": 92.1,
                "common_models": ["Wan2_2-T2V-A14B-HIGH (8.5 GB)", "sd3_medium (6.0 GB)"]
            }
        ],
        "recent_oom_events": [...],
        "model_correlation": {
            "Wan2_2-T2V-A14B-HIGH": {
                "present_in_oom_count": 23,
                "avg_vram_mb": 8704,
                "correlation_score": 0.89
            }
        },
        "recommendations": [
            {
                "type": "unload_model",
                "priority": "high",
                "message": "Consider unloading 'sd3_medium' (6.0 GB) before WanVideoWrapper nodes",
                "models_to_unload": ["sd3_medium"],
                "expected_vram_freed_gb": 6.0
            }
        ]
    }
}

Frontend OOM UI Enhancements

Pre-OOM Warning Indicators (Popup Stats Cards)

// Show VRAM% with color-coded warnings
const vramPercent = latest.vramPercentBefore || 0;
let warningClass = vramPercent >= 95 ? 'stat-critical' : 
                   vramPercent >= 85 ? 'stat-warning' : '';
let warningIcon = vramPercent >= 95 ? '🔴' : 
                  vramPercent >= 85 ? '⚠️' : '🎮';

statsGrid.innerHTML += `
    <div class="profiler-stat-card ${warningClass}">
        <div class="profiler-stat-label">${warningIcon} VRAM Before</div>
        <div class="profiler-stat-value">${vramPercent.toFixed(1)}%</div>
    </div>
`;

OOM Indicators in Node Table

// Mark OOM nodes with 💥 icon
tbody.innerHTML = nodeArray.map(node => `
    <tr class="${node.oomOccurred ? 'oom-node' : ''}">
        <td>${node.oomOccurred ? '💥 ' : ''}${node.nodeType}</td>
        <td>${formatMs(node.executionTime)}</td>
        <td>${formatBytes(node.vramDelta)}</td>
        <td>${node.oomOccurred ? '💥' : node.cacheHit ? '🟢' : '⚡'}</td>
    </tr>
`).join('');

New Modal Tab: "💥 OOM Analysis"

<div class="profiler-tab-panel" id="profiler-tab-oom">
    <h3>💥 Out of Memory Analysis</h3>
    
    <!-- Summary Cards -->
    <div class="oom-summary-grid">
        <div class="oom-summary-card">
            <div class="oom-summary-label">Total OOMs</div>
            <div class="oom-summary-value">47</div>
        </div>
        <div class="oom-summary-card">
            <div class="oom-summary-label">OOM Rate</div>
            <div class="oom-summary-value">12.3%</div>
        </div>
    </div>
    
    <!-- Node Type Ranking Table -->
    <h4>Node Types by OOM Frequency</h4>
    <table class="oom-ranking-table">
        <thead>
            <tr>
                <th>Node Type</th>
                <th>OOM Count</th>
                <th>OOM Rate</th>
                <th>Avg VRAM Before</th>
            </tr>
        </thead>
        <tbody id="oom-ranking-body"></tbody>
    </table>
    
    <!-- Model Correlation -->
    <h4>Models Present at OOM</h4>
    <div id="oom-models-correlation"></div>
    
    <!-- Recommendations -->
    <h4>💡 Recommendations</h4>
    <div id="oom-recommendations-list"></div>
</div>

Model Unload Recommendations

Algorithm:

1. Sort loaded models by VRAM usage (descending)
2. Check last-used timestamps via node execution tracking
3. Calculate VRAM headroom needed to prevent OOM
4. Generate specific recommendations

Example Output:

💡 RECOMMENDATION: Unload these models before WanVideoWrapper
  1. sd3_medium_incl_clips (6.0 GB) - Last used 45s ago
  2. flux_vae (0.5 GB) - Last used 62s ago
  → Expected VRAM freed: 6.5 GB
  → Usage would drop from 92% to 63%

💡 RECOMMENDATION: Enable --lowvram mode
  Reason: Workflows with WanVideoWrapper consistently use >85% VRAM
  Impact: Slower execution but prevents OOM

Archive Enhancement with OOM Data

Archives now include OOM-specific data:

{
    "archived_at": "2025-11-29T15:30:00.000Z",
    "history": [...],
    "oom_history": [...],
    "oom_summary": {
        "total_oom_count": 47,
        "oom_rate": 12.3,
        "most_problematic_node": "WanVideoWrapper",
        "most_problematic_model": "Wan2_2-T2V-A14B-HIGH"
    }
}

Archives with OOM events show a badge in the UI:

<span class="oom-badge">💥 ${archive.oom_count} OOMs</span>

Real-World Example: WanVideoWrapper OOM

Before (no OOM detection):

torch.OutOfMemoryError: Allocation on device
Got an OOM, unloading all loaded models.

After (with OOM detection):

⚠️  WARNING: VRAM at 87.3% before WanVideoWrapper (free: 2.8 GB / 22.0 GB)
🔴 CRITICAL: VRAM at 96.1% before next node (free: 0.9 GB / 22.0 GB) - OOM likely!
💥 OOM detected in node WanVideoWrapper (ID: 42)

Models loaded at OOM:
  GPU 0: 4 models
    - Wan2_2-T2V-A14B-HIGH (checkpoint): 8704 MB
    - sd3_medium_incl_clips (checkpoint): 6144 MB
    - flux_vae (vae): 512 MB
    - controlnet_union (controlnet): 1280 MB
  Total model VRAM: 16.3 GB
  System/Framework overhead: 5.7 GB

💡 RECOMMENDATION: Unload 'sd3_medium_incl_clips' (6.0 GB) before WanVideoWrapper
   This would bring pre-execution VRAM from 87.3% to 60.1%

Common OOM Scenarios & Solutions

Scenario 1: Large Checkpoint + Large Video Model

Symptoms: VRAM >90% before video generation, OOM during forward pass Solution:

import comfy.model_management as mm
mm.unload_all_models()  # Before WanVideoWrapper node
mm.soft_empty_cache()

Scenario 2: Batch Size Too Large

Symptoms: OOM during KSampler with high batch size Solution: Reduce batch_size or enable --lowvram mode

Scenario 3: Memory Fragmentation

Symptoms: Free VRAM available but allocation fails, OOM with <80% usage Solution: Restart ComfyUI to defragment GPU memory

OOM Prevention Best Practices

1. Monitor Pre-OOM Warnings: Act on ⚠️ (85%) and 🔴 (95%) warnings
2. Unload Models Proactively: Clear models between workflow stages
3. Use --lowvram Mode: For workflows consistently using >80% VRAM
4. Reduce Resolution First: Lower image/video resolution before quality
5. Check Model Combinations: Use OOM analysis to identify problematic combos
6. Archive OOM History: Review patterns to optimize workflows

Phase 6.4: Frontend OOM UI Features

Visual OOM Indicators

Stats Card Warning Borders

• Stats cards in the profiler popup show warning borders when VRAM usage is high
• ⚠️ Yellow Border (85-95% VRAM): Warning threshold reached
  • Border: 2px solid #f59e0b with glow shadow
  • Applied to Time, VRAM Peak, and RAM Peak cards
• 🔴 Red Border (>95% VRAM): Critical threshold - OOM imminent
  • Border: 2px solid #ef4444 with red glow shadow
  • High risk of OOM in next node execution

Node Table 💥 Column

• New column in the "Top 10 Slowest Nodes" table shows OOM status
• 💥 icon appears for nodes that experienced Out of Memory errors
• Empty cell for nodes that executed successfully
• Tooltip shows "Out of Memory Error" or "No OOM" on hover

💥 OOM Analysis Tab

The profiler modal now includes a dedicated "💥 OOM Analysis" tab showing comprehensive OOM statistics:

1. Summary Cards

• 💥 Total OOMs: Count of all OOM events (red if >0)
• 📈 Total Workflows: Number of workflow executions tracked
• 📉 OOM Rate: Percentage of workflows with OOMs (yellow if >10%)

2. Recent OOM Events

• Chronological list of recent OOM occurrences
• Each event shows:
  • Node type that failed (e.g., "💥 WanVideoSampler")
  • Timestamp of the OOM
  • 🎮 VRAM at OOM and percentage used
  • 📈 VRAM peak during execution
  • ⏱️ Execution time before OOM
  • 📦 Models loaded at time of OOM with VRAM usage
  • ⚠️ Error message from the exception

3. Node Type Ranking

• Table showing which node types are most OOM-prone
• Columns:
  • Node Type: Name of the node
  • OOMs: Total OOM count for this node type
  • Total Executions: How many times the node ran
  • OOM Rate: Percentage (color-coded: >50% red, >10% yellow)
  • Avg VRAM at OOM: Average VRAM usage when OOMs occur
  • Last OOM: Date of most recent OOM

4. Model Correlation

• Shows which model combinations correlate with OOMs
• Identifies patterns like "BiRefNet + WanVideoSampler = OOM"
• Displays:
  • Node type
  • Models loaded during OOM
  • OOM count for this combination
  • Total VRAM of the model set

5. Recommendations

• Auto-generated actionable recommendations based on OOM patterns
• Severity-coded with icons:
  • 🛑 Critical: Node OOM rate >50% - immediate action needed
  • ⚠️ Warning: Model combination caused ≥2 OOMs
  • 💡 Info: General recommendations for >10% OOM rate
• Each recommendation includes:
  • Title describing the issue
  • Detailed message with specific actions

CSS Classes for OOM Styling

/* Warning borders for stats cards */
.profiler-stat-card-warning { /* Yellow border at 85% VRAM */ }
.profiler-stat-card-critical { /* Red border at 95% VRAM */ }

/* OOM icon column */
.profiler-oom-icon { /* Center-aligned 💥 icon */ }

/* OOM Analysis tab components */
.oom-summary-cards { /* 3-column grid of summary cards */ }
.oom-summary-error { /* Red text for OOM count */ }
.oom-summary-warning { /* Yellow text for high OOM rate */ }
.oom-recent-item { /* Red border for OOM event cards */ }
.oom-table { /* Styled tables for rankings */ }
.oom-rate-critical { /* Red text for >50% OOM rate */ }
.oom-rate-warning { /* Yellow text for >10% OOM rate */ }
.oom-rec-critical { /* Red left border for critical recommendations */ }
.oom-rec-warning { /* Yellow left border for warnings */ }
.oom-rec-info { /* Blue left border for info */ }
.oom-success { /* Green success message when no OOMs */ }

Accessing OOM Analysis

1. During Workflow: Watch for warning borders on stats cards in profiler popup
2. After Execution: Check 💥 column in node table to see which nodes failed
3. Full Analysis: Click profiler button → Open modal → "💥 OOM Analysis" tab
4. Review History: Analyze patterns across multiple workflows to identify problem nodes/models

Implementation Phases

Phase 6.1: Backend OOM Detection ✅ COMPLETED

• Extend NodeProfile/WorkflowProfile with OOM fields
• Add pre-OOM warnings in start_node() (85%/95% thresholds)
• Implement OOM exception capture in map_node_with_profiling()
• Add OOM event logging with model snapshots

Phase 6.2: Historical OOM Tracking ✅ COMPLETED

• Add oom_history to ProfilerManager
• Implement _update_oom_stats() method
• Extend archive structure with OOM data
• Create GET /swissarmyknife/profiler/oom_stats endpoint

Phase 6.3: Model Recommendations 🚧 PARTIALLY COMPLETED

• Track model snapshots at OOM
• Implement basic recommendation algorithm
• Track model last-used timestamps (future enhancement)
• Implement advanced unload recommendation with staleness (future enhancement)

Phase 6.4: Frontend OOM UI ✅ COMPLETED

• Add pre-OOM warning indicators to stats cards (yellow at 85%, red at 95%)
• Add 💥 OOM column to node table in profiler popup
• Create "💥 OOM Analysis" modal tab
• Display OOM summary cards (total OOMs, rate, workflows)
• Show recent OOM events with VRAM context
• Display node type ranking by OOM frequency
• Display model correlation table
• Show recommendations with severity indicators