# Resource Monitor - Floating Heads-Up Monitor Bar

**Status**: Phase 2 frontend + restart flow complete
**Component**: Web JavaScript Extension (`web/js/resource_monitor.js`)

## Overview

The Resource Monitor extension draws a floating glass "heads-up" bar in the ComfyUI viewport that streams live CPU/RAM/GPU telemetry **and** exposes an always-visible Restart button. The widget detaches from the legacy dropdown entry in `swiss-army-knife.js` so that operators can trigger a restart while simultaneously watching resource pressure build in real time.

Key Capabilities:
- Fixed, centered monitor bar injected directly into `document.body` (not the top toolbar)
- Inline CPU, RAM, temperature, GPU utilization, and VRAM stats rendered with gradient fills
- Automatic discovery of the hardware that is actually available from the backend
- Fully wired restart workflow: confirmation toasts, `/swissarmyknife/restart` POST, `/system_stats` health polling, and cache-busting reload
- WebSocket live updates via `api.addEventListener("swissarmyknife.monitor", handler)` so the UI stays in sync after the initial REST fetch

## File Layout

```
web/
├── js/
│   ├── resource_monitor.js   # Floating bar + restart workflow (this document)
│   └── swiss-army-knife.js   # Legacy dropdown restart entry (still shipped but redundant)
└── css/
    └── restart-button.css    # Historical styles, no longer referenced
```

## Extension Registration & Setup Flow

1. `app.registerExtension({ name: "comfyui_swissarmyknife.resource_monitor", setup })` wires the feature into ComfyUI.
2. During `setup()` the extension:
   - Injects the style tag `#swissarmyknife-restart-button-styles` that defines the floating glassmorphism wrapper, gradient logic, and button micro-interactions.
   - Performs an initial `fetch("/swissarmyknife/monitor/status")` to understand which monitors should render.
   - Dynamically builds monitor cells through `createMonitorDisplay(label, id)` based on backend capabilities (psutil / pynvml availability, detected GPUs, exposed temperatures, etc.).
   - Appends the Restart button returned from `createRestartButton()` after the telemetry cells.
   - Appends the completed wrapper to `document.body` so it floats independently of the toolbar DOM hierarchy.
   - Subscribes to `swissarmyknife.monitor` events on the ComfyUI API bridge, which surface the backend push updates every ~2 seconds.

## UI Layout & Styling

The wrapper `#swissarmyknife-resource-monitor` is positioned `fixed` at `top: 50px` and horizontally centered with `left: 50%` + `transform: translateX(-50%)`. It uses a translucent background, blur filter, rounded 16px corners, and an inset highlight so it reads like a lightweight HUD.

Each telemetry cell (`.swissarmyknife-monitor`) displays `LABEL VALUE` pairs:
- CPU % and RAM (GB used) render whenever psutil is available.
- CPU temperature renders when exposed by the backend sensors.
- For every GPU device the backend reports as `available`, the UI renders utilization %, VRAM usage, and (optionally) GPU temp rows. Labels include a compact per-GPU suffix (e.g., `GPU 0 (3090Ti)`, `VRAM 0 (A6000)`), generated by `extractGPUModel()`.
- Cells call `updateMonitorValue(id, value, percent)` whenever new data arrives. The helper paints a linear gradient from left to right using thresholds `<50% (green #22c55e)`, `<70% (yellow #eab308)`, `<90% (orange #f97316)`, and `>=90% (red #ef4444)` to immediately communicate pressure.

### Workflow Execution Glow

When a workflow starts running (`execution_start` event) the monitor wrapper gains the `swissarmyknife-is-executing` class. CSS registered via `@property --swissarmyknife-gradient-angle` and the `swissarmyknife-resource-monitor-rotation` keyframes animates a rotating multi-color gradient on paired pseudo-elements: `::before` renders a soft blurred plume outside the pill and `::after` masks the gradient into a thin border ring so the fill stays untouched. The gradient now cycles across six hues (blue → purple → magenta → orange → gold → jade) to avoid any single color dominating the rim. The telemetry tiles live inside `#swissarmyknife-resource-monitor-inner`, an inset flex wrapper that introduces a small gutter so the animated halo never collides with the per-metric progress bars. The class is removed on `execution_success`, `execution_error`, `execution_cancelled`, or `execution_end` so the glow shuts off the instant a workflow finishes or fails.

The Restart button shares the wrapper so the operator can watch the monitors while the restart is underway. It follows the ComfyUI `.comfyui-button` base class with custom danger colors, disabled state, and a divider pseudo-element (`::before`) to visually separate it from the monitors.

## Restart Workflow

`createRestartButton()` wires the full restart experience:
1. Disable + relabel the button to "Restarting..." and push a Toast (`app.extensionManager.toast.add`) describing what is happening.
2. Fire `fetch("/swissarmyknife/restart", { method: "POST" })`. Any network failure is ignored because the backend tears down the HTTP connection during restart.
3. Wait 2 seconds for shutdown to start, then poll `/system_stats` every second (up to 60s) until a 200 OK proves the server is alive again.
4. When the health check succeeds, pop a success toast and hard reload `window.location.reload(true)` to pick up new JS bundles without a stale cache.
5. If the timeout expires, re-enable the button and warn the operator that manual refresh might be required.
6. All errors propagate to the console with the `[SwissArmyKnife][ResourceMonitor]` prefix and surface a Toast that explains the failure.

The behavior mirrors the menu command restart logic, but the inline placement plus stateful button make the process much more discoverable.

## Data Sources & Update Loop

- **Initial snapshot**: `GET /swissarmyknife/monitor/status` returns `{ hardware, gpu }`. The UI uses this payload to decide which tiles to render (e.g., no psutil => only GPU tiles, no pynvml => skip utilization tiles, etc.).
- **Live updates**: The backend broadcasts `swissarmyknife.monitor` events. `handleMonitorUpdate()` consumes the payload, runs the formatters (`formatGB`, `formatPercent`, `formatTemp`, `formatGHz`) and updates every matching tile.
- **Health checks**: Restart logic pings `/system_stats` because it is small, unauthenticated, and always hosted by ComfyUI itself.
- **Debug logs**: Controlled by `SwissArmyKnife.debug_mode`. When enabled the console prints setup/start/end banners, AJAX failures, and periodic polling output (every 5 attempts during restart polling).

## Dependency Matrix & Graceful Degradation

| Backend Dependency | UI Impact |
|--------------------|-----------|
| `psutil`           | CPU %, RAM, CPU Temp tiles render. Without it, those tiles are simply not created. |
| `py-cpuinfo`       | Used server-side to enrich CPU labels; UI automatically falls back to `--` when a value is missing. |
| `torch`            | Enables VRAM reporting when pynvml is absent. The UI still displays VRAM rows because the backend formats the bytes. |
| `pynvml`           | Unlocks GPU utilization and temperature metrics. Without it only VRAM tiles (and restart) appear. |

No JavaScript errors are thrown when a capability is missing; the UI just renders the tiles that make sense for the current machine.

## Browser Refresh Requirements

Because this is a pure JavaScript widget, no ComfyUI restart is required. Developers should perform a hard refresh (Ctrl/Cmd + Shift + R) whenever `resource_monitor.js` changes to force the new bundle to load and to re-inject the CSS.

## Roadmap

### Near-Term Enhancements

- Clear VRAM button and backend endpoint wiring
- Per-tile context menus (click to copy usage, open GPU inspector, etc.)
- Visibility presets so laptop users can hide GPU tiles
- Historical sparklines per metric
- Settings surface for sampling interval + toast verbosity

### ProfilerX Parity Plan

To reach the level of workflow profiling offered by [ComfyUI ProfilerX](https://github.com/ryanontheinside/ComfyUI_ProfilerX), we will deliver the following phases with concrete implementation hooks learned from their codebase.

1. **Instrumentation Layer**
   - Mirror ProfilerX’s `prestartup.py` shim: wrap `execution.execute`, `PromptExecutor.execute`, and `ExecutionList.__init__` so we can call a `ProfilerManager` analogue that records `start_workflow`, `start_node`, `end_node`, and `end_workflow` events.
   - Track cache hits by checking `caches.outputs[current_item]` before/after execution just like ProfilerX’s wrappers do, and collect tensor shapes via helpers similar to `_get_tensor_sizes`.
   - Reset CUDA peak stats per node (`torch.cuda.reset_peak_memory_stats`) to capture accurate VRAM deltas, persist RAM via `psutil.Process().memory_info().rss`, and roll running averages for nodes/workflows.
   - Keep a short-lived `active_profiles` dict plus a disk-backed `history` array, batching writes (ProfilerX saves every 5 workflows) so we don’t thrash the filesystem.

2. **Profiler API Surface**
   - Expose REST endpoints modeled after ProfilerX’s `server.py`: `/swissarmyknife/profiler/stats`, `/archives`, `/archive/:id/load`, `/archive/:id` (DELETE). Responses should include `current`, `latest`, `node_averages`, `workflow_averages`, and a bounded `history`.
   - Support archive rotation by dumping JSON snapshots into `data/archives/` and deleting them after loading—this matches ProfilerX’s `archive_history`, `load_archive`, and `delete_archive`.
   - Emit monitor broadcasts (existing `swissarmyknife.monitor`) plus a new profiler-specific event whenever a workflow completes so the UI can refresh instantly.

3. **Profiler Dashboard UI**
   - Attach a menu button the way ProfilerX does in `web/index.ts`: locate `.comfyui-menu-right`, create a `.comfyui-button-group`, and inject a button that toggles a detailed popup.
   - Reuse their approach of listening to `app.addEventListener("executed")` and custom `profiler:historyLoaded` events so tabs stay in sync after archives load.
   - Implement tabbed content similar to `web/ui/tabs.ts`, with menu-level mini stats, per-node sortable tables, cache hit/miss breakdown, and inline charts (SVG or canvas) fed from the REST payloads.

4. **Historical Storage & Analytics**
   - Persist up to ~10k workflow entries on disk (same order-of-magnitude as ProfilerX’s `max_history`) with auto-archiving when limits are hit, plus asynchronous saves via a background executor.
   - Augment each record with `executionOrder`, `averages`, tensor sizes, and `cacheHits/Misses` so we can compute the same aggregates ProfilerX exposes.
   - Build archive management UI (list + load + delete) mirroring ProfilerX’s `profilerx-stats-popup` so operators can prune or revisit past runs without leaving the browser.

5. **Advanced Execution Tracking**
   - Once Profiler parity lands, add an opt-in tracker patterned after `execution_core.ExecutionTracker`: wrap internal ComfyUI methods with decorators that push method names, durations, and stack depth into `method_traces.json`.
   - Stream summarized trace metadata to the UI so a “Deep Trace” tab can highlight slow internals without forcing users to open raw JSON files.
   - Surface a settings toggle (Swiss Army Knife config) that sets `ExecutionTracker.ENABLED` instead of requiring code edits, matching ProfilerX’s optional tracing workflow.

Keep the file-level documentation synchronized with each feature drop so operators know which telemetry to expect.

## Profiler Implementation Status

**Last Updated**: November 28, 2025
**Status**: ✅ Phase 1-3 Complete | 🚧 Phase 4-5 In Progress

### Implemented Features

#### ✅ Phase 1: Backend Instrumentation
- Prestartup hook system (`nodes/utils/resource_monitor/prestartup.py`)
- ProfilerManager singleton with workflow/node tracking
- VRAM/RAM measurement with graceful degradation
- Cache hit detection and tensor shape tracking
- Batched async saves (every 5 workflows)
- Auto-archiving at 10k workflow limit

#### ✅ Phase 2: REST API & WebSocket
- `GET /swissarmyknife/profiler/stats` - Latest stats and history
- `GET /swissarmyknife/profiler/archives` - List archives
- `POST /swissarmyknife/profiler/archive` - Create archive
- `POST /swissarmyknife/profiler/archive/:filename/load` - Load archive
- `DELETE /swissarmyknife/profiler/archive/:filename` - Delete archive
- WebSocket broadcasts on workflow completion

#### ✅ Phase 3: Profiler UI
- Profiler button (📊) in floating resource monitor
- Glassmorphic popup with:
  - Circular progress gauge
  - 6-card stats grid (Time, VRAM, RAM, Cache, Nodes). The Total Time card now renders long runs in minutes+seconds (e.g., `5m 50.9s`) while sub-minute workflows stay in seconds for better readability.
  - Top 10 slowest nodes table
  - "View Full History" and "Clear History" buttons
- Full-screen modal with 4 tabs (Latest, Previous, Analytics, Settings)
- Real-time WebSocket updates
- Auto-refresh on workflow execution

### Zero-Config Design

Profiler is enabled by default with:
- No user configuration required
- Automatic startup integration
- Graceful degradation when dependencies missing
- < 1% execution overhead

### Usage

1. Click 📊 profiler button in floating monitor for quick stats
2. Click "View Full History" for detailed modal analysis
3. Use Settings tab for archive management

### Known Limitations

#### VRAM Tracking
- **Requires NVIDIA CUDA**: VRAM tracking only works on systems with CUDA-enabled GPUs (PyTorch CUDA API)
- **CPU-only mode**: No VRAM tracking available when running on CPU-only systems (logs warning at startup)
- **AMD/Intel GPUs**: VRAM tracking not supported (PyTorch CUDA-specific implementation)
- **Allocation timing**: Captures memory at node execution boundaries only, not internal allocations during node execution
- **Shared memory**: Cannot distinguish between node-specific and globally shared VRAM usage across the process

#### RAM Tracking
- **Requires psutil**: RAM tracking disabled if psutil package not installed
- **Process-level only**: Measures total process memory, not per-node isolation
- **System overhead**: Includes Python interpreter and ComfyUI framework overhead in measurements

#### Cache Hit Detection
- **Not yet implemented**: Currently always reports `cache_hit=False` for all node executions
- **Planned enhancement**: Future integration with ComfyUI's cache system to detect actual cache hits

#### Tensor Shape Tracking
- **Requires PyTorch**: Only available when torch package is installed
- **Limited types**: Only tracks `torch.Tensor` objects in inputs/outputs
- **Nested structures**: May miss tensors deeply nested in complex data structures

#### Timing Accuracy
- **Async operations**: May not capture accurate timing for nodes with async GPU operations that return before completion
- **Python GIL**: Subject to Python Global Interpreter Lock overhead
- **System scheduling**: Wall-clock timing affected by system scheduling and other processes

#### History & Storage
- **Auto-archive limit**: Automatically archives after 10,000 workflows to prevent unbounded disk growth
- **Disk I/O**: Batched saves (every 5 workflows) may delay persistence during crashes
- **Memory footprint**: Active profiling adds ~500KB-2MB per workflow in memory before archiving

#### Hardware Detection
- **Best-effort detection**: Dependency availability checked at startup only, not dynamically
- **Silent degradation**: Missing dependencies log warnings but don't fail loudly to user
- **CUDA initialization**: First CUDA call may add latency during ProfilerManager initialization

### Troubleshooting

**VRAM shows as 0 or null:**
- Verify CUDA is available: Check console for "CUDA available: True" message
- Ensure GPU is NVIDIA with CUDA support
- Confirm PyTorch installed with CUDA support: `python -c "import torch; print(torch.cuda.is_available())"`

**RAM tracking not working:**
- Install psutil: `pip install psutil`
- Restart ComfyUI server after installation

**Profiler not capturing data:**
- Check console for "[SwissArmyKnife][Profiler]" messages
- Verify profiler enabled in settings (enabled by default)
- Look for warnings about dependency availability

### Next Steps

- Enhanced archive management UI (Phase 4)
- Historical charts and sparklines
- Advanced execution tracking (Phase 5)
- Implement cache hit detection
- OOM diagnosis and prevention (Phase 6)

---

## OOM Diagnosis & Prevention

**Status**: 🚧 Phase 6 - Planned
**Purpose**: Detect, capture, and prevent Out of Memory errors through unified profiler architecture

### Overview

The OOM diagnosis system extends the profiler with pre-OOM warnings, exception capture, historical analysis, and model unload recommendations. This prevents memory exhaustion failures by combining threshold-based alerts with actionable insights.

### Key Capabilities

- **Pre-OOM Detection**: Threshold warnings at 85% (⚠️ yellow) and 95% (🔴 red) VRAM usage
- **OOM Event Capture**: Automatic detection with full context (VRAM state, loaded models, timeline)
- **Historical Analysis**: Persistent tracking with node type ranking and model correlation
- **Model Attribution**: Correlate OOM failures with specific loaded model combinations
- **Unload Recommendations**: Suggest which models to unload based on size and usage patterns

### Architecture Data Flow

```
1. PRE-NODE EXECUTION (start_node)
   ↓ Check free VRAM via torch.cuda.mem_get_info()
   ↓ Calculate vram_percent_before = (total-free)/total*100
   ↓ Log ⚠️  if >85%, 🔴 if >95%
   ↓ Store vram_free_before, vram_percent_before

2. NODE EXECUTION (map_node_with_profiling)
   ↓ Execute node with full profiling
   ↓ Track VRAM delta, peak, timeline
   ↓ Catch RuntimeError exceptions
   ↓ (if OOM occurs)

3. OOM CAPTURE (exception handler)
   ↓ Detect "out of memory" in error message
   ↓ Capture torch.cuda.memory_summary()
   ↓ Snapshot loaded models via _scan_gpu_models()
   ↓ Record vram_at_oom, models_at_oom, oom_error
   ↓ Mark node.oom_occurred = True

4. HISTORICAL TRACKING (end_workflow)
   ↓ Append OOM event to oom_history list
   ↓ Update OOM frequency stats per node type
   ↓ Calculate model correlation patterns
   ↓ Archive OOM data with workflow history

5. ANALYSIS & RECOMMENDATIONS
   ↓ GET /swissarmyknife/profiler/oom_stats
   ↓ Node type OOM ranking (frequency, avg VRAM)
   ↓ Model correlation (which models present at OOM)
   ↓ Unload recommendations (by size + last-used)
```

### Extended Data Models

#### NodeProfile OOM Fields

```python
class NodeProfile:
    # Existing fields...
    oom_occurred: bool = False              # True if this node caused OOM
    oom_error: Optional[str] = None         # Full exception message
    vram_free_before: Optional[int] = None  # Free VRAM before execution (bytes)
    vram_percent_before: Optional[float] = None  # VRAM usage % before execution
    vram_at_oom: Optional[int] = None       # Allocated VRAM when OOM occurred
    models_at_oom: Optional[List[Dict]] = None   # Loaded models snapshot at OOM
```

#### WorkflowProfile OOM Fields

```python
class WorkflowProfile:
    # Existing fields...
    oom_occurred: bool = False           # True if any node OOM'd
    oom_node_id: Optional[str] = None    # Which node caused OOM
```

#### ProfilerManager OOM Tracking

```python
class ProfilerManager:
    # Existing fields...
    oom_history: List[Dict] = []         # Separate OOM event log
    oom_stats: Dict[str, Dict] = {}      # Per-node-type OOM statistics
    max_oom_history: int = 1000          # Limit OOM history size
```

### Pre-OOM Threshold Warnings

**Thresholds**:
- **85% - ⚠️ Warning (Yellow)**: VRAM approaching limits, monitor closely
- **95% - 🔴 Critical (Red)**: VRAM critically high, OOM highly likely

**Implementation in `ProfilerManager.start_node()`**:

```python
# Get free VRAM before node execution
free_vram, total_vram = torch.cuda.mem_get_info()
node_profile.vram_free_before = free_vram
node_profile.vram_percent_before = (total_vram - free_vram) / total_vram * 100

# Threshold warnings
if node_profile.vram_percent_before >= 95:
    logger.error(f"🔴 CRITICAL: VRAM at {vram_percent:.1f}% before {node_type} - OOM likely!")
elif node_profile.vram_percent_before >= 85:
    logger.warning(f"⚠️  WARNING: VRAM at {vram_percent:.1f}% before {node_type}")
```

**Example Console Output**:
```
⚠️  WARNING: VRAM at 87.3% before WanVideoWrapper (free: 2.8 GB / 22.0 GB)
🔴 CRITICAL: VRAM at 96.1% before WanVideoWrapper (free: 0.9 GB / 22.0 GB) - OOM likely!
```

### OOM Exception Capture

**Detection in `map_node_with_profiling()` exception handler**:

```python
try:
    result = await original_map_node_over_list(...)
    profiler.end_node(prompt_id, unique_id, outputs=result)
    return result
except RuntimeError as e:
    # Detect OOM specifically
    is_oom = "out of memory" in str(e).lower()
    
    if is_oom and profiler_enabled:
        node = profile.nodes[unique_id]
        node.oom_occurred = True
        node.oom_error = str(e)
        node.vram_at_oom = torch.cuda.memory_allocated()
        node.models_at_oom = profiler._scan_gpu_models()
        
        # Log models present at OOM
        logger.error(f"💥 OOM in {class_type}: {vram_at_oom / 1024**3:.1f} GB allocated")
        for gpu_id, models in node.models_at_oom.items():
            for model in models:
                logger.error(f"  - {model['name']} ({model['type']}): {model['vram_mb']} MB")
    raise
```

**Captured Context**:
- Exception message (full traceback)
- VRAM allocated at OOM moment
- GPU memory summary (PyTorch internal state)
- Complete loaded models snapshot with VRAM per model
- Node execution timeline
- VRAM headroom before node started

### OOM Event Structure

```json
{
    "timestamp": "2025-11-29T14:23:45.123Z",
    "prompt_id": "f8ad0b31-1717-4878-aad1-064d5df212d9",
    "node_id": "42",
    "node_type": "WanVideoWrapper",
    "execution_time_before_oom": 135.41,
    "vram_percent_before": 87.3,
    "vram_free_before": 2899102720,
    "vram_at_oom": 23622320128,
    "models_at_oom": {
        "0": [
            {
                "name": "Wan2_2-T2V-A14B-HIGH",
                "type": "checkpoint",
                "vram_mb": 8704
            },
            {
                "name": "sd3_medium",
                "type": "checkpoint",
                "vram_mb": 6144
            }
        ]
    },
    "error_message": "torch.OutOfMemoryError: Allocation on device",
    "total_models_vram_mb": 14848
}
```

### New API Endpoint

**`GET /swissarmyknife/profiler/oom_stats`**

Returns OOM statistics and analysis:

```json
{
    "success": true,
    "data": {
        "total_oom_count": 47,
        "oom_rate": 12.3,
        "node_type_ranking": [
            {
                "node_type": "WanVideoWrapper",
                "oom_count": 23,
                "oom_rate": 48.9,
                "avg_vram_at_oom_gb": 21.8,
                "avg_vram_percent_before": 92.1,
                "common_models": ["Wan2_2-T2V-A14B-HIGH (8.5 GB)", "sd3_medium (6.0 GB)"]
            }
        ],
        "recent_oom_events": [...],
        "model_correlation": {
            "Wan2_2-T2V-A14B-HIGH": {
                "present_in_oom_count": 23,
                "avg_vram_mb": 8704,
                "correlation_score": 0.89
            }
        },
        "recommendations": [
            {
                "type": "unload_model",
                "priority": "high",
                "message": "Consider unloading 'sd3_medium' (6.0 GB) before WanVideoWrapper nodes",
                "models_to_unload": ["sd3_medium"],
                "expected_vram_freed_gb": 6.0
            }
        ]
    }
}
```

### Frontend OOM UI Enhancements

#### Pre-OOM Warning Indicators (Popup Stats Cards)

```javascript
// Show VRAM% with color-coded warnings
const vramPercent = latest.vramPercentBefore || 0;
let warningClass = vramPercent >= 95 ? 'stat-critical' : 
                   vramPercent >= 85 ? 'stat-warning' : '';
let warningIcon = vramPercent >= 95 ? '🔴' : 
                  vramPercent >= 85 ? '⚠️' : '🎮';

statsGrid.innerHTML += `
    <div class="profiler-stat-card ${warningClass}">
        <div class="profiler-stat-label">${warningIcon} VRAM Before</div>
        <div class="profiler-stat-value">${vramPercent.toFixed(1)}%</div>
    </div>
`;
```

#### OOM Indicators in Node Table

```javascript
// Mark OOM nodes with 💥 icon
tbody.innerHTML = nodeArray.map(node => `
    <tr class="${node.oomOccurred ? 'oom-node' : ''}">
        <td>${node.oomOccurred ? '💥 ' : ''}${node.nodeType}</td>
        <td>${formatMs(node.executionTime)}</td>
        <td>${formatBytes(node.vramDelta)}</td>
        <td>${node.oomOccurred ? '💥' : node.cacheHit ? '🟢' : '⚡'}</td>
    </tr>
`).join('');
```

#### New Modal Tab: "💥 OOM Analysis"

```javascript
<div class="profiler-tab-panel" id="profiler-tab-oom">
    <h3>💥 Out of Memory Analysis</h3>
    
    <!-- Summary Cards -->
    <div class="oom-summary-grid">
        <div class="oom-summary-card">
            <div class="oom-summary-label">Total OOMs</div>
            <div class="oom-summary-value">47</div>
        </div>
        <div class="oom-summary-card">
            <div class="oom-summary-label">OOM Rate</div>
            <div class="oom-summary-value">12.3%</div>
        </div>
    </div>
    
    <!-- Node Type Ranking Table -->
    <h4>Node Types by OOM Frequency</h4>
    <table class="oom-ranking-table">
        <thead>
            <tr>
                <th>Node Type</th>
                <th>OOM Count</th>
                <th>OOM Rate</th>
                <th>Avg VRAM Before</th>
            </tr>
        </thead>
        <tbody id="oom-ranking-body"></tbody>
    </table>
    
    <!-- Model Correlation -->
    <h4>Models Present at OOM</h4>
    <div id="oom-models-correlation"></div>
    
    <!-- Recommendations -->
    <h4>💡 Recommendations</h4>
    <div id="oom-recommendations-list"></div>
</div>
```

### Model Unload Recommendations

**Algorithm**:
1. Sort loaded models by VRAM usage (descending)
2. Check last-used timestamps via node execution tracking
3. Calculate VRAM headroom needed to prevent OOM
4. Generate specific recommendations

**Example Output**:
```
💡 RECOMMENDATION: Unload these models before WanVideoWrapper
  1. sd3_medium_incl_clips (6.0 GB) - Last used 45s ago
  2. flux_vae (0.5 GB) - Last used 62s ago
  → Expected VRAM freed: 6.5 GB
  → Usage would drop from 92% to 63%

💡 RECOMMENDATION: Enable --lowvram mode
  Reason: Workflows with WanVideoWrapper consistently use >85% VRAM
  Impact: Slower execution but prevents OOM
```

### Archive Enhancement with OOM Data

Archives now include OOM-specific data:

```json
{
    "archived_at": "2025-11-29T15:30:00.000Z",
    "history": [...],
    "oom_history": [...],
    "oom_summary": {
        "total_oom_count": 47,
        "oom_rate": 12.3,
        "most_problematic_node": "WanVideoWrapper",
        "most_problematic_model": "Wan2_2-T2V-A14B-HIGH"
    }
}
```

Archives with OOM events show a badge in the UI:
```javascript
<span class="oom-badge">💥 ${archive.oom_count} OOMs</span>
```

### Real-World Example: WanVideoWrapper OOM

**Before (no OOM detection)**:
```
torch.OutOfMemoryError: Allocation on device
Got an OOM, unloading all loaded models.
```

**After (with OOM detection)**:
```
⚠️  WARNING: VRAM at 87.3% before WanVideoWrapper (free: 2.8 GB / 22.0 GB)
🔴 CRITICAL: VRAM at 96.1% before next node (free: 0.9 GB / 22.0 GB) - OOM likely!
💥 OOM detected in node WanVideoWrapper (ID: 42)

Models loaded at OOM:
  GPU 0: 4 models
    - Wan2_2-T2V-A14B-HIGH (checkpoint): 8704 MB
    - sd3_medium_incl_clips (checkpoint): 6144 MB
    - flux_vae (vae): 512 MB
    - controlnet_union (controlnet): 1280 MB
  Total model VRAM: 16.3 GB
  System/Framework overhead: 5.7 GB

💡 RECOMMENDATION: Unload 'sd3_medium_incl_clips' (6.0 GB) before WanVideoWrapper
   This would bring pre-execution VRAM from 87.3% to 60.1%
```

### Common OOM Scenarios & Solutions

#### Scenario 1: Large Checkpoint + Large Video Model
**Symptoms**: VRAM >90% before video generation, OOM during forward pass
**Solution**: 
```python
import comfy.model_management as mm
mm.unload_all_models()  # Before WanVideoWrapper node
mm.soft_empty_cache()
```

#### Scenario 2: Batch Size Too Large
**Symptoms**: OOM during KSampler with high batch size
**Solution**: Reduce `batch_size` or enable `--lowvram` mode

#### Scenario 3: Memory Fragmentation
**Symptoms**: Free VRAM available but allocation fails, OOM with <80% usage
**Solution**: Restart ComfyUI to defragment GPU memory

### OOM Prevention Best Practices

1. **Monitor Pre-OOM Warnings**: Act on ⚠️ (85%) and 🔴 (95%) warnings
2. **Unload Models Proactively**: Clear models between workflow stages
3. **Use --lowvram Mode**: For workflows consistently using >80% VRAM
4. **Reduce Resolution First**: Lower image/video resolution before quality
5. **Check Model Combinations**: Use OOM analysis to identify problematic combos
6. **Archive OOM History**: Review patterns to optimize workflows

## Phase 6.4: Frontend OOM UI Features

### Visual OOM Indicators

**Stats Card Warning Borders**
- Stats cards in the profiler popup show warning borders when VRAM usage is high
- ⚠️ **Yellow Border (85-95% VRAM)**: Warning threshold reached
  - Border: 2px solid #f59e0b with glow shadow
  - Applied to Time, VRAM Peak, and RAM Peak cards
- 🔴 **Red Border (>95% VRAM)**: Critical threshold - OOM imminent
  - Border: 2px solid #ef4444 with red glow shadow
  - High risk of OOM in next node execution

**Node Table 💥 Column**
- New column in the "Top 10 Slowest Nodes" table shows OOM status
- 💥 icon appears for nodes that experienced Out of Memory errors
- Empty cell for nodes that executed successfully
- Tooltip shows "Out of Memory Error" or "No OOM" on hover

### 💥 OOM Analysis Tab

The profiler modal now includes a dedicated "💥 OOM Analysis" tab showing comprehensive OOM statistics:

**1. Summary Cards**
- **💥 Total OOMs**: Count of all OOM events (red if >0)
- **📈 Total Workflows**: Number of workflow executions tracked
- **📉 OOM Rate**: Percentage of workflows with OOMs (yellow if >10%)

**2. Recent OOM Events**
- Chronological list of recent OOM occurrences
- Each event shows:
  - Node type that failed (e.g., "💥 WanVideoSampler")
  - Timestamp of the OOM
  - 🎮 VRAM at OOM and percentage used
  - 📈 VRAM peak during execution
  - ⏱️ Execution time before OOM
  - 📦 Models loaded at time of OOM with VRAM usage
  - ⚠️ Error message from the exception

**3. Node Type Ranking**
- Table showing which node types are most OOM-prone
- Columns:
  - **Node Type**: Name of the node
  - **OOMs**: Total OOM count for this node type
  - **Total Executions**: How many times the node ran
  - **OOM Rate**: Percentage (color-coded: >50% red, >10% yellow)
  - **Avg VRAM at OOM**: Average VRAM usage when OOMs occur
  - **Last OOM**: Date of most recent OOM

**4. Model Correlation**
- Shows which model combinations correlate with OOMs
- Identifies patterns like "BiRefNet + WanVideoSampler = OOM"
- Displays:
  - Node type
  - Models loaded during OOM
  - OOM count for this combination
  - Total VRAM of the model set

**5. Recommendations**
- Auto-generated actionable recommendations based on OOM patterns
- Severity-coded with icons:
  - 🛑 **Critical**: Node OOM rate >50% - immediate action needed
  - ⚠️ **Warning**: Model combination caused ≥2 OOMs
  - 💡 **Info**: General recommendations for >10% OOM rate
- Each recommendation includes:
  - Title describing the issue
  - Detailed message with specific actions

### CSS Classes for OOM Styling

```css
/* Warning borders for stats cards */
.profiler-stat-card-warning { /* Yellow border at 85% VRAM */ }
.profiler-stat-card-critical { /* Red border at 95% VRAM */ }

/* OOM icon column */
.profiler-oom-icon { /* Center-aligned 💥 icon */ }

/* OOM Analysis tab components */
.oom-summary-cards { /* 3-column grid of summary cards */ }
.oom-summary-error { /* Red text for OOM count */ }
.oom-summary-warning { /* Yellow text for high OOM rate */ }
.oom-recent-item { /* Red border for OOM event cards */ }
.oom-table { /* Styled tables for rankings */ }
.oom-rate-critical { /* Red text for >50% OOM rate */ }
.oom-rate-warning { /* Yellow text for >10% OOM rate */ }
.oom-rec-critical { /* Red left border for critical recommendations */ }
.oom-rec-warning { /* Yellow left border for warnings */ }
.oom-rec-info { /* Blue left border for info */ }
.oom-success { /* Green success message when no OOMs */ }
```

### Accessing OOM Analysis

1. **During Workflow**: Watch for warning borders on stats cards in profiler popup
2. **After Execution**: Check 💥 column in node table to see which nodes failed
3. **Full Analysis**: Click profiler button → Open modal → "💥 OOM Analysis" tab
4. **Review History**: Analyze patterns across multiple workflows to identify problem nodes/models

### Implementation Phases

**Phase 6.1: Backend OOM Detection** ✅ COMPLETED
- [x] Extend NodeProfile/WorkflowProfile with OOM fields
- [x] Add pre-OOM warnings in start_node() (85%/95% thresholds)
- [x] Implement OOM exception capture in map_node_with_profiling()
- [x] Add OOM event logging with model snapshots

**Phase 6.2: Historical OOM Tracking** ✅ COMPLETED
- [x] Add oom_history to ProfilerManager
- [x] Implement _update_oom_stats() method
- [x] Extend archive structure with OOM data
- [x] Create GET /swissarmyknife/profiler/oom_stats endpoint

**Phase 6.3: Model Recommendations** 🚧 PARTIALLY COMPLETED
- [x] Track model snapshots at OOM
- [x] Implement basic recommendation algorithm
- [ ] Track model last-used timestamps (future enhancement)
- [ ] Implement advanced unload recommendation with staleness (future enhancement)

**Phase 6.4: Frontend OOM UI** ✅ COMPLETED
- [x] Add pre-OOM warning indicators to stats cards (yellow at 85%, red at 95%)
- [x] Add 💥 OOM column to node table in profiler popup
- [x] Create "💥 OOM Analysis" modal tab
- [x] Display OOM summary cards (total OOMs, rate, workflows)
- [x] Show recent OOM events with VRAM context
- [x] Display node type ranking by OOM frequency
- [x] Display model correlation table
- [x] Show recommendations with severity indicators