769 lines
34 KiB
Markdown
769 lines
34 KiB
Markdown
# Resource Monitor - Floating Heads-Up Monitor Bar
|
||
|
||
**Status**: Phase 2 frontend + restart flow complete
|
||
**Component**: Web JavaScript Extension (`web/js/resource_monitor.js`)
|
||
|
||
## Overview
|
||
|
||
The Resource Monitor extension draws a floating glass "heads-up" bar in the ComfyUI viewport that streams live CPU/RAM/GPU telemetry **and** exposes an always-visible Restart button. The widget detaches from the legacy dropdown entry in `swiss-army-knife.js` so that operators can trigger a restart while simultaneously watching resource pressure build in real time.
|
||
|
||
Key Capabilities:
|
||
- Fixed, centered monitor bar injected directly into `document.body` (not the top toolbar)
|
||
- Inline CPU, RAM, temperature, GPU utilization, and VRAM stats rendered with gradient fills
|
||
- Automatic discovery of the hardware that is actually available from the backend
|
||
- Fully wired restart workflow: confirmation toasts, `/swissarmyknife/restart` POST, `/system_stats` health polling, and cache-busting reload
|
||
- WebSocket live updates via `api.addEventListener("swissarmyknife.monitor", handler)` so the UI stays in sync after the initial REST fetch
|
||
|
||
## File Layout
|
||
|
||
```
|
||
web/
|
||
├── js/
|
||
│ ├── resource_monitor.js # Floating bar + restart workflow (this document)
|
||
│ └── swiss-army-knife.js # Legacy dropdown restart entry (still shipped but redundant)
|
||
└── css/
|
||
└── restart-button.css # Historical styles, no longer referenced
|
||
```
|
||
|
||
## Extension Registration & Setup Flow
|
||
|
||
1. `app.registerExtension({ name: "comfyui_swissarmyknife.resource_monitor", setup })` wires the feature into ComfyUI.
|
||
2. During `setup()` the extension:
|
||
- Injects the style tag `#swissarmyknife-restart-button-styles` that defines the floating glassmorphism wrapper, gradient logic, and button micro-interactions.
|
||
- Performs an initial `fetch("/swissarmyknife/monitor/status")` to understand which monitors should render.
|
||
- Dynamically builds monitor cells through `createMonitorDisplay(label, id)` based on backend capabilities (psutil / pynvml availability, detected GPUs, exposed temperatures, etc.).
|
||
- Appends the Restart button returned from `createRestartButton()` after the telemetry cells.
|
||
- Appends the completed wrapper to `document.body` so it floats independently of the toolbar DOM hierarchy.
|
||
- Subscribes to `swissarmyknife.monitor` events on the ComfyUI API bridge, which surface the backend push updates every ~2 seconds.
|
||
|
||
## UI Layout & Styling
|
||
|
||
The wrapper `#swissarmyknife-resource-monitor` is positioned `fixed` at `top: 50px` and horizontally centered with `left: 50%` + `transform: translateX(-50%)`. It uses a translucent background, blur filter, rounded 16px corners, and an inset highlight so it reads like a lightweight HUD.
|
||
|
||
Each telemetry cell (`.swissarmyknife-monitor`) displays `LABEL VALUE` pairs:
|
||
- CPU % and RAM (GB used) render whenever psutil is available.
|
||
- CPU temperature renders when exposed by the backend sensors.
|
||
- For every GPU device the backend reports as `available`, the UI renders utilization %, VRAM usage, and (optionally) GPU temp rows. Labels include a compact per-GPU suffix (e.g., `GPU 0 (3090Ti)`, `VRAM 0 (A6000)`), generated by `extractGPUModel()`.
|
||
- Cells call `updateMonitorValue(id, value, percent)` whenever new data arrives. The helper paints a linear gradient from left to right using thresholds `<50% (green #22c55e)`, `<70% (yellow #eab308)`, `<90% (orange #f97316)`, and `>=90% (red #ef4444)` to immediately communicate pressure.
|
||
|
||
### Workflow Execution Glow
|
||
|
||
When a workflow starts running (`execution_start` event) the monitor wrapper gains the `swissarmyknife-is-executing` class. CSS registered via `@property --swissarmyknife-gradient-angle` and the `swissarmyknife-resource-monitor-rotation` keyframes animates a rotating multi-color gradient on paired pseudo-elements: `::before` renders a soft blurred plume outside the pill and `::after` masks the gradient into a thin border ring so the fill stays untouched. The gradient now cycles across six hues (blue → purple → magenta → orange → gold → jade) to avoid any single color dominating the rim. The telemetry tiles live inside `#swissarmyknife-resource-monitor-inner`, an inset flex wrapper that introduces a small gutter so the animated halo never collides with the per-metric progress bars. The class is removed on `execution_success`, `execution_error`, `execution_cancelled`, or `execution_end` so the glow shuts off the instant a workflow finishes or fails.
|
||
|
||
The Restart button shares the wrapper so the operator can watch the monitors while the restart is underway. It follows the ComfyUI `.comfyui-button` base class with custom danger colors, disabled state, and a divider pseudo-element (`::before`) to visually separate it from the monitors.
|
||
|
||
## Restart Workflow
|
||
|
||
`createRestartButton()` wires the full restart experience:
|
||
1. Disable + relabel the button to "Restarting..." and push a Toast (`app.extensionManager.toast.add`) describing what is happening.
|
||
2. Fire `fetch("/swissarmyknife/restart", { method: "POST" })`. Any network failure is ignored because the backend tears down the HTTP connection during restart.
|
||
3. Wait 2 seconds for shutdown to start, then poll `/system_stats` every second (up to 60s) until a 200 OK proves the server is alive again.
|
||
4. When the health check succeeds, pop a success toast and hard reload `window.location.reload(true)` to pick up new JS bundles without a stale cache.
|
||
5. If the timeout expires, re-enable the button and warn the operator that manual refresh might be required.
|
||
6. All errors propagate to the console with the `[SwissArmyKnife][ResourceMonitor]` prefix and surface a Toast that explains the failure.
|
||
|
||
The behavior mirrors the menu command restart logic, but the inline placement plus stateful button make the process much more discoverable.
|
||
|
||
## Data Sources & Update Loop
|
||
|
||
- **Initial snapshot**: `GET /swissarmyknife/monitor/status` returns `{ hardware, gpu }`. The UI uses this payload to decide which tiles to render (e.g., no psutil => only GPU tiles, no pynvml => skip utilization tiles, etc.).
|
||
- **Live updates**: The backend broadcasts `swissarmyknife.monitor` events. `handleMonitorUpdate()` consumes the payload, runs the formatters (`formatGB`, `formatPercent`, `formatTemp`, `formatGHz`) and updates every matching tile.
|
||
- **Health checks**: Restart logic pings `/system_stats` because it is small, unauthenticated, and always hosted by ComfyUI itself.
|
||
- **Debug logs**: Controlled by `SwissArmyKnife.debug_mode`. When enabled the console prints setup/start/end banners, AJAX failures, and periodic polling output (every 5 attempts during restart polling).
|
||
|
||
## Dependency Matrix & Graceful Degradation
|
||
|
||
| Backend Dependency | UI Impact |
|
||
|--------------------|-----------|
|
||
| `psutil` | CPU %, RAM, CPU Temp tiles render. Without it, those tiles are simply not created. |
|
||
| `py-cpuinfo` | Used server-side to enrich CPU labels; UI automatically falls back to `--` when a value is missing. |
|
||
| `torch` | Enables VRAM reporting when pynvml is absent. The UI still displays VRAM rows because the backend formats the bytes. |
|
||
| `pynvml` | Unlocks GPU utilization and temperature metrics. Without it only VRAM tiles (and restart) appear. |
|
||
|
||
No JavaScript errors are thrown when a capability is missing; the UI just renders the tiles that make sense for the current machine.
|
||
|
||
## Browser Refresh Requirements
|
||
|
||
Because this is a pure JavaScript widget, no ComfyUI restart is required. Developers should perform a hard refresh (Ctrl/Cmd + Shift + R) whenever `resource_monitor.js` changes to force the new bundle to load and to re-inject the CSS.
|
||
|
||
## Roadmap
|
||
|
||
### Near-Term Enhancements
|
||
|
||
- Clear VRAM button and backend endpoint wiring
|
||
- Per-tile context menus (click to copy usage, open GPU inspector, etc.)
|
||
- Visibility presets so laptop users can hide GPU tiles
|
||
- Historical sparklines per metric
|
||
- Settings surface for sampling interval + toast verbosity
|
||
|
||
### ProfilerX Parity Plan
|
||
|
||
To reach the level of workflow profiling offered by [ComfyUI ProfilerX](https://github.com/ryanontheinside/ComfyUI_ProfilerX), we will deliver the following phases with concrete implementation hooks learned from their codebase.
|
||
|
||
1. **Instrumentation Layer**
|
||
- Mirror ProfilerX’s `prestartup.py` shim: wrap `execution.execute`, `PromptExecutor.execute`, and `ExecutionList.__init__` so we can call a `ProfilerManager` analogue that records `start_workflow`, `start_node`, `end_node`, and `end_workflow` events.
|
||
- Track cache hits by checking `caches.outputs[current_item]` before/after execution just like ProfilerX’s wrappers do, and collect tensor shapes via helpers similar to `_get_tensor_sizes`.
|
||
- Reset CUDA peak stats per node (`torch.cuda.reset_peak_memory_stats`) to capture accurate VRAM deltas, persist RAM via `psutil.Process().memory_info().rss`, and roll running averages for nodes/workflows.
|
||
- Keep a short-lived `active_profiles` dict plus a disk-backed `history` array, batching writes (ProfilerX saves every 5 workflows) so we don’t thrash the filesystem.
|
||
|
||
2. **Profiler API Surface**
|
||
- Expose REST endpoints modeled after ProfilerX’s `server.py`: `/swissarmyknife/profiler/stats`, `/archives`, `/archive/:id/load`, `/archive/:id` (DELETE). Responses should include `current`, `latest`, `node_averages`, `workflow_averages`, and a bounded `history`.
|
||
- Support archive rotation by dumping JSON snapshots into `data/archives/` and deleting them after loading—this matches ProfilerX’s `archive_history`, `load_archive`, and `delete_archive`.
|
||
- Emit monitor broadcasts (existing `swissarmyknife.monitor`) plus a new profiler-specific event whenever a workflow completes so the UI can refresh instantly.
|
||
|
||
3. **Profiler Dashboard UI**
|
||
- Attach a menu button the way ProfilerX does in `web/index.ts`: locate `.comfyui-menu-right`, create a `.comfyui-button-group`, and inject a button that toggles a detailed popup.
|
||
- Reuse their approach of listening to `app.addEventListener("executed")` and custom `profiler:historyLoaded` events so tabs stay in sync after archives load.
|
||
- Implement tabbed content similar to `web/ui/tabs.ts`, with menu-level mini stats, per-node sortable tables, cache hit/miss breakdown, and inline charts (SVG or canvas) fed from the REST payloads.
|
||
|
||
4. **Historical Storage & Analytics**
|
||
- Persist up to ~10k workflow entries on disk (same order-of-magnitude as ProfilerX’s `max_history`) with auto-archiving when limits are hit, plus asynchronous saves via a background executor.
|
||
- Augment each record with `executionOrder`, `averages`, tensor sizes, and `cacheHits/Misses` so we can compute the same aggregates ProfilerX exposes.
|
||
- Build archive management UI (list + load + delete) mirroring ProfilerX’s `profilerx-stats-popup` so operators can prune or revisit past runs without leaving the browser.
|
||
|
||
5. **Advanced Execution Tracking**
|
||
- Once Profiler parity lands, add an opt-in tracker patterned after `execution_core.ExecutionTracker`: wrap internal ComfyUI methods with decorators that push method names, durations, and stack depth into `method_traces.json`.
|
||
- Stream summarized trace metadata to the UI so a “Deep Trace” tab can highlight slow internals without forcing users to open raw JSON files.
|
||
- Surface a settings toggle (Swiss Army Knife config) that sets `ExecutionTracker.ENABLED` instead of requiring code edits, matching ProfilerX’s optional tracing workflow.
|
||
|
||
Keep the file-level documentation synchronized with each feature drop so operators know which telemetry to expect.
|
||
|
||
## Profiler Implementation Status
|
||
|
||
**Last Updated**: November 28, 2025
|
||
**Status**: ✅ Phase 1-3 Complete | 🚧 Phase 4-5 In Progress
|
||
|
||
### Implemented Features
|
||
|
||
#### ✅ Phase 1: Backend Instrumentation
|
||
- Prestartup hook system (`nodes/utils/resource_monitor/prestartup.py`)
|
||
- ProfilerManager singleton with workflow/node tracking
|
||
- VRAM/RAM measurement with graceful degradation
|
||
- Cache hit detection and tensor shape tracking
|
||
- Batched async saves (every 5 workflows)
|
||
- Auto-archiving at 10k workflow limit
|
||
|
||
#### ✅ Phase 2: REST API & WebSocket
|
||
- `GET /swissarmyknife/profiler/stats` - Latest stats and history
|
||
- `GET /swissarmyknife/profiler/archives` - List archives
|
||
- `POST /swissarmyknife/profiler/archive` - Create archive
|
||
- `POST /swissarmyknife/profiler/archive/:filename/load` - Load archive
|
||
- `DELETE /swissarmyknife/profiler/archive/:filename` - Delete archive
|
||
- WebSocket broadcasts on workflow completion
|
||
|
||
#### ✅ Phase 3: Profiler UI
|
||
- Profiler button (📊) in floating resource monitor
|
||
- Glassmorphic popup with:
|
||
- Circular progress gauge
|
||
- 6-card stats grid (Time, VRAM, RAM, Cache, Nodes). The Total Time card now renders long runs in minutes+seconds (e.g., `5m 50.9s`) while sub-minute workflows stay in seconds for better readability.
|
||
- Top 10 slowest nodes table
|
||
- "View Full History" and "Clear History" buttons
|
||
- Full-screen modal with 4 tabs (Latest, Previous, Analytics, Settings)
|
||
- Real-time WebSocket updates
|
||
- Auto-refresh on workflow execution
|
||
|
||
### Zero-Config Design
|
||
|
||
Profiler is enabled by default with:
|
||
- No user configuration required
|
||
- Automatic startup integration
|
||
- Graceful degradation when dependencies missing
|
||
- < 1% execution overhead
|
||
|
||
### Usage
|
||
|
||
1. Click 📊 profiler button in floating monitor for quick stats
|
||
2. Click "View Full History" for detailed modal analysis
|
||
3. Use Settings tab for archive management
|
||
|
||
### Known Limitations
|
||
|
||
#### VRAM Tracking
|
||
- **Requires NVIDIA CUDA**: VRAM tracking only works on systems with CUDA-enabled GPUs (PyTorch CUDA API)
|
||
- **CPU-only mode**: No VRAM tracking available when running on CPU-only systems (logs warning at startup)
|
||
- **AMD/Intel GPUs**: VRAM tracking not supported (PyTorch CUDA-specific implementation)
|
||
- **Allocation timing**: Captures memory at node execution boundaries only, not internal allocations during node execution
|
||
- **Shared memory**: Cannot distinguish between node-specific and globally shared VRAM usage across the process
|
||
|
||
#### RAM Tracking
|
||
- **Requires psutil**: RAM tracking disabled if psutil package not installed
|
||
- **Process-level only**: Measures total process memory, not per-node isolation
|
||
- **System overhead**: Includes Python interpreter and ComfyUI framework overhead in measurements
|
||
|
||
#### Cache Hit Detection
|
||
- **Not yet implemented**: Currently always reports `cache_hit=False` for all node executions
|
||
- **Planned enhancement**: Future integration with ComfyUI's cache system to detect actual cache hits
|
||
|
||
#### Tensor Shape Tracking
|
||
- **Requires PyTorch**: Only available when torch package is installed
|
||
- **Limited types**: Only tracks `torch.Tensor` objects in inputs/outputs
|
||
- **Nested structures**: May miss tensors deeply nested in complex data structures
|
||
|
||
#### Timing Accuracy
|
||
- **Async operations**: May not capture accurate timing for nodes with async GPU operations that return before completion
|
||
- **Python GIL**: Subject to Python Global Interpreter Lock overhead
|
||
- **System scheduling**: Wall-clock timing affected by system scheduling and other processes
|
||
|
||
#### History & Storage
|
||
- **Auto-archive limit**: Automatically archives after 10,000 workflows to prevent unbounded disk growth
|
||
- **Disk I/O**: Batched saves (every 5 workflows) may delay persistence during crashes
|
||
- **Memory footprint**: Active profiling adds ~500KB-2MB per workflow in memory before archiving
|
||
|
||
#### Hardware Detection
|
||
- **Best-effort detection**: Dependency availability checked at startup only, not dynamically
|
||
- **Silent degradation**: Missing dependencies log warnings but don't fail loudly to user
|
||
- **CUDA initialization**: First CUDA call may add latency during ProfilerManager initialization
|
||
|
||
### Troubleshooting
|
||
|
||
**VRAM shows as 0 or null:**
|
||
- Verify CUDA is available: Check console for "CUDA available: True" message
|
||
- Ensure GPU is NVIDIA with CUDA support
|
||
- Confirm PyTorch installed with CUDA support: `python -c "import torch; print(torch.cuda.is_available())"`
|
||
|
||
**RAM tracking not working:**
|
||
- Install psutil: `pip install psutil`
|
||
- Restart ComfyUI server after installation
|
||
|
||
**Profiler not capturing data:**
|
||
- Check console for "[SwissArmyKnife][Profiler]" messages
|
||
- Verify profiler enabled in settings (enabled by default)
|
||
- Look for warnings about dependency availability
|
||
|
||
### Next Steps
|
||
|
||
- Enhanced archive management UI (Phase 4)
|
||
- Historical charts and sparklines
|
||
- Advanced execution tracking (Phase 5)
|
||
- Implement cache hit detection
|
||
- OOM diagnosis and prevention (Phase 6)
|
||
|
||
---
|
||
|
||
## OOM Diagnosis & Prevention
|
||
|
||
**Status**: 🚧 Phase 6 - Planned
|
||
**Purpose**: Detect, capture, and prevent Out of Memory errors through unified profiler architecture
|
||
|
||
### Overview
|
||
|
||
The OOM diagnosis system extends the profiler with pre-OOM warnings, exception capture, historical analysis, and model unload recommendations. This prevents memory exhaustion failures by combining threshold-based alerts with actionable insights.
|
||
|
||
### Key Capabilities
|
||
|
||
- **Pre-OOM Detection**: Threshold warnings at 85% (⚠️ yellow) and 95% (🔴 red) VRAM usage
|
||
- **OOM Event Capture**: Automatic detection with full context (VRAM state, loaded models, timeline)
|
||
- **Historical Analysis**: Persistent tracking with node type ranking and model correlation
|
||
- **Model Attribution**: Correlate OOM failures with specific loaded model combinations
|
||
- **Unload Recommendations**: Suggest which models to unload based on size and usage patterns
|
||
|
||
### Architecture Data Flow
|
||
|
||
```
|
||
1. PRE-NODE EXECUTION (start_node)
|
||
↓ Check free VRAM via torch.cuda.mem_get_info()
|
||
↓ Calculate vram_percent_before = (total-free)/total*100
|
||
↓ Log ⚠️ if >85%, 🔴 if >95%
|
||
↓ Store vram_free_before, vram_percent_before
|
||
|
||
2. NODE EXECUTION (map_node_with_profiling)
|
||
↓ Execute node with full profiling
|
||
↓ Track VRAM delta, peak, timeline
|
||
↓ Catch RuntimeError exceptions
|
||
↓ (if OOM occurs)
|
||
|
||
3. OOM CAPTURE (exception handler)
|
||
↓ Detect "out of memory" in error message
|
||
↓ Capture torch.cuda.memory_summary()
|
||
↓ Snapshot loaded models via _scan_gpu_models()
|
||
↓ Record vram_at_oom, models_at_oom, oom_error
|
||
↓ Mark node.oom_occurred = True
|
||
|
||
4. HISTORICAL TRACKING (end_workflow)
|
||
↓ Append OOM event to oom_history list
|
||
↓ Update OOM frequency stats per node type
|
||
↓ Calculate model correlation patterns
|
||
↓ Archive OOM data with workflow history
|
||
|
||
5. ANALYSIS & RECOMMENDATIONS
|
||
↓ GET /swissarmyknife/profiler/oom_stats
|
||
↓ Node type OOM ranking (frequency, avg VRAM)
|
||
↓ Model correlation (which models present at OOM)
|
||
↓ Unload recommendations (by size + last-used)
|
||
```
|
||
|
||
### Extended Data Models
|
||
|
||
#### NodeProfile OOM Fields
|
||
|
||
```python
|
||
class NodeProfile:
|
||
# Existing fields...
|
||
oom_occurred: bool = False # True if this node caused OOM
|
||
oom_error: Optional[str] = None # Full exception message
|
||
vram_free_before: Optional[int] = None # Free VRAM before execution (bytes)
|
||
vram_percent_before: Optional[float] = None # VRAM usage % before execution
|
||
vram_at_oom: Optional[int] = None # Allocated VRAM when OOM occurred
|
||
models_at_oom: Optional[List[Dict]] = None # Loaded models snapshot at OOM
|
||
```
|
||
|
||
#### WorkflowProfile OOM Fields
|
||
|
||
```python
|
||
class WorkflowProfile:
|
||
# Existing fields...
|
||
oom_occurred: bool = False # True if any node OOM'd
|
||
oom_node_id: Optional[str] = None # Which node caused OOM
|
||
```
|
||
|
||
#### ProfilerManager OOM Tracking
|
||
|
||
```python
|
||
class ProfilerManager:
|
||
# Existing fields...
|
||
oom_history: List[Dict] = [] # Separate OOM event log
|
||
oom_stats: Dict[str, Dict] = {} # Per-node-type OOM statistics
|
||
max_oom_history: int = 1000 # Limit OOM history size
|
||
```
|
||
|
||
### Pre-OOM Threshold Warnings
|
||
|
||
**Thresholds**:
|
||
- **85% - ⚠️ Warning (Yellow)**: VRAM approaching limits, monitor closely
|
||
- **95% - 🔴 Critical (Red)**: VRAM critically high, OOM highly likely
|
||
|
||
**Implementation in `ProfilerManager.start_node()`**:
|
||
|
||
```python
|
||
# Get free VRAM before node execution
|
||
free_vram, total_vram = torch.cuda.mem_get_info()
|
||
node_profile.vram_free_before = free_vram
|
||
node_profile.vram_percent_before = (total_vram - free_vram) / total_vram * 100
|
||
|
||
# Threshold warnings
|
||
if node_profile.vram_percent_before >= 95:
|
||
logger.error(f"🔴 CRITICAL: VRAM at {vram_percent:.1f}% before {node_type} - OOM likely!")
|
||
elif node_profile.vram_percent_before >= 85:
|
||
logger.warning(f"⚠️ WARNING: VRAM at {vram_percent:.1f}% before {node_type}")
|
||
```
|
||
|
||
**Example Console Output**:
|
||
```
|
||
⚠️ WARNING: VRAM at 87.3% before WanVideoWrapper (free: 2.8 GB / 22.0 GB)
|
||
🔴 CRITICAL: VRAM at 96.1% before WanVideoWrapper (free: 0.9 GB / 22.0 GB) - OOM likely!
|
||
```
|
||
|
||
### OOM Exception Capture
|
||
|
||
**Detection in `map_node_with_profiling()` exception handler**:
|
||
|
||
```python
|
||
try:
|
||
result = await original_map_node_over_list(...)
|
||
profiler.end_node(prompt_id, unique_id, outputs=result)
|
||
return result
|
||
except RuntimeError as e:
|
||
# Detect OOM specifically
|
||
is_oom = "out of memory" in str(e).lower()
|
||
|
||
if is_oom and profiler_enabled:
|
||
node = profile.nodes[unique_id]
|
||
node.oom_occurred = True
|
||
node.oom_error = str(e)
|
||
node.vram_at_oom = torch.cuda.memory_allocated()
|
||
node.models_at_oom = profiler._scan_gpu_models()
|
||
|
||
# Log models present at OOM
|
||
logger.error(f"💥 OOM in {class_type}: {vram_at_oom / 1024**3:.1f} GB allocated")
|
||
for gpu_id, models in node.models_at_oom.items():
|
||
for model in models:
|
||
logger.error(f" - {model['name']} ({model['type']}): {model['vram_mb']} MB")
|
||
raise
|
||
```
|
||
|
||
**Captured Context**:
|
||
- Exception message (full traceback)
|
||
- VRAM allocated at OOM moment
|
||
- GPU memory summary (PyTorch internal state)
|
||
- Complete loaded models snapshot with VRAM per model
|
||
- Node execution timeline
|
||
- VRAM headroom before node started
|
||
|
||
### OOM Event Structure
|
||
|
||
```json
|
||
{
|
||
"timestamp": "2025-11-29T14:23:45.123Z",
|
||
"prompt_id": "f8ad0b31-1717-4878-aad1-064d5df212d9",
|
||
"node_id": "42",
|
||
"node_type": "WanVideoWrapper",
|
||
"execution_time_before_oom": 135.41,
|
||
"vram_percent_before": 87.3,
|
||
"vram_free_before": 2899102720,
|
||
"vram_at_oom": 23622320128,
|
||
"models_at_oom": {
|
||
"0": [
|
||
{
|
||
"name": "Wan2_2-T2V-A14B-HIGH",
|
||
"type": "checkpoint",
|
||
"vram_mb": 8704
|
||
},
|
||
{
|
||
"name": "sd3_medium",
|
||
"type": "checkpoint",
|
||
"vram_mb": 6144
|
||
}
|
||
]
|
||
},
|
||
"error_message": "torch.OutOfMemoryError: Allocation on device",
|
||
"total_models_vram_mb": 14848
|
||
}
|
||
```
|
||
|
||
### New API Endpoint
|
||
|
||
**`GET /swissarmyknife/profiler/oom_stats`**
|
||
|
||
Returns OOM statistics and analysis:
|
||
|
||
```json
|
||
{
|
||
"success": true,
|
||
"data": {
|
||
"total_oom_count": 47,
|
||
"oom_rate": 12.3,
|
||
"node_type_ranking": [
|
||
{
|
||
"node_type": "WanVideoWrapper",
|
||
"oom_count": 23,
|
||
"oom_rate": 48.9,
|
||
"avg_vram_at_oom_gb": 21.8,
|
||
"avg_vram_percent_before": 92.1,
|
||
"common_models": ["Wan2_2-T2V-A14B-HIGH (8.5 GB)", "sd3_medium (6.0 GB)"]
|
||
}
|
||
],
|
||
"recent_oom_events": [...],
|
||
"model_correlation": {
|
||
"Wan2_2-T2V-A14B-HIGH": {
|
||
"present_in_oom_count": 23,
|
||
"avg_vram_mb": 8704,
|
||
"correlation_score": 0.89
|
||
}
|
||
},
|
||
"recommendations": [
|
||
{
|
||
"type": "unload_model",
|
||
"priority": "high",
|
||
"message": "Consider unloading 'sd3_medium' (6.0 GB) before WanVideoWrapper nodes",
|
||
"models_to_unload": ["sd3_medium"],
|
||
"expected_vram_freed_gb": 6.0
|
||
}
|
||
]
|
||
}
|
||
}
|
||
```
|
||
|
||
### Frontend OOM UI Enhancements
|
||
|
||
#### Pre-OOM Warning Indicators (Popup Stats Cards)
|
||
|
||
```javascript
|
||
// Show VRAM% with color-coded warnings
|
||
const vramPercent = latest.vramPercentBefore || 0;
|
||
let warningClass = vramPercent >= 95 ? 'stat-critical' :
|
||
vramPercent >= 85 ? 'stat-warning' : '';
|
||
let warningIcon = vramPercent >= 95 ? '🔴' :
|
||
vramPercent >= 85 ? '⚠️' : '🎮';
|
||
|
||
statsGrid.innerHTML += `
|
||
<div class="profiler-stat-card ${warningClass}">
|
||
<div class="profiler-stat-label">${warningIcon} VRAM Before</div>
|
||
<div class="profiler-stat-value">${vramPercent.toFixed(1)}%</div>
|
||
</div>
|
||
`;
|
||
```
|
||
|
||
#### OOM Indicators in Node Table
|
||
|
||
```javascript
|
||
// Mark OOM nodes with 💥 icon
|
||
tbody.innerHTML = nodeArray.map(node => `
|
||
<tr class="${node.oomOccurred ? 'oom-node' : ''}">
|
||
<td>${node.oomOccurred ? '💥 ' : ''}${node.nodeType}</td>
|
||
<td>${formatMs(node.executionTime)}</td>
|
||
<td>${formatBytes(node.vramDelta)}</td>
|
||
<td>${node.oomOccurred ? '💥' : node.cacheHit ? '🟢' : '⚡'}</td>
|
||
</tr>
|
||
`).join('');
|
||
```
|
||
|
||
#### New Modal Tab: "💥 OOM Analysis"
|
||
|
||
```javascript
|
||
<div class="profiler-tab-panel" id="profiler-tab-oom">
|
||
<h3>💥 Out of Memory Analysis</h3>
|
||
|
||
<!-- Summary Cards -->
|
||
<div class="oom-summary-grid">
|
||
<div class="oom-summary-card">
|
||
<div class="oom-summary-label">Total OOMs</div>
|
||
<div class="oom-summary-value">47</div>
|
||
</div>
|
||
<div class="oom-summary-card">
|
||
<div class="oom-summary-label">OOM Rate</div>
|
||
<div class="oom-summary-value">12.3%</div>
|
||
</div>
|
||
</div>
|
||
|
||
<!-- Node Type Ranking Table -->
|
||
<h4>Node Types by OOM Frequency</h4>
|
||
<table class="oom-ranking-table">
|
||
<thead>
|
||
<tr>
|
||
<th>Node Type</th>
|
||
<th>OOM Count</th>
|
||
<th>OOM Rate</th>
|
||
<th>Avg VRAM Before</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody id="oom-ranking-body"></tbody>
|
||
</table>
|
||
|
||
<!-- Model Correlation -->
|
||
<h4>Models Present at OOM</h4>
|
||
<div id="oom-models-correlation"></div>
|
||
|
||
<!-- Recommendations -->
|
||
<h4>💡 Recommendations</h4>
|
||
<div id="oom-recommendations-list"></div>
|
||
</div>
|
||
```
|
||
|
||
### Model Unload Recommendations
|
||
|
||
**Algorithm**:
|
||
1. Sort loaded models by VRAM usage (descending)
|
||
2. Check last-used timestamps via node execution tracking
|
||
3. Calculate VRAM headroom needed to prevent OOM
|
||
4. Generate specific recommendations
|
||
|
||
**Example Output**:
|
||
```
|
||
💡 RECOMMENDATION: Unload these models before WanVideoWrapper
|
||
1. sd3_medium_incl_clips (6.0 GB) - Last used 45s ago
|
||
2. flux_vae (0.5 GB) - Last used 62s ago
|
||
→ Expected VRAM freed: 6.5 GB
|
||
→ Usage would drop from 92% to 63%
|
||
|
||
💡 RECOMMENDATION: Enable --lowvram mode
|
||
Reason: Workflows with WanVideoWrapper consistently use >85% VRAM
|
||
Impact: Slower execution but prevents OOM
|
||
```
|
||
|
||
### Archive Enhancement with OOM Data
|
||
|
||
Archives now include OOM-specific data:
|
||
|
||
```json
|
||
{
|
||
"archived_at": "2025-11-29T15:30:00.000Z",
|
||
"history": [...],
|
||
"oom_history": [...],
|
||
"oom_summary": {
|
||
"total_oom_count": 47,
|
||
"oom_rate": 12.3,
|
||
"most_problematic_node": "WanVideoWrapper",
|
||
"most_problematic_model": "Wan2_2-T2V-A14B-HIGH"
|
||
}
|
||
}
|
||
```
|
||
|
||
Archives with OOM events show a badge in the UI:
|
||
```javascript
|
||
<span class="oom-badge">💥 ${archive.oom_count} OOMs</span>
|
||
```
|
||
|
||
### Real-World Example: WanVideoWrapper OOM
|
||
|
||
**Before (no OOM detection)**:
|
||
```
|
||
torch.OutOfMemoryError: Allocation on device
|
||
Got an OOM, unloading all loaded models.
|
||
```
|
||
|
||
**After (with OOM detection)**:
|
||
```
|
||
⚠️ WARNING: VRAM at 87.3% before WanVideoWrapper (free: 2.8 GB / 22.0 GB)
|
||
🔴 CRITICAL: VRAM at 96.1% before next node (free: 0.9 GB / 22.0 GB) - OOM likely!
|
||
💥 OOM detected in node WanVideoWrapper (ID: 42)
|
||
|
||
Models loaded at OOM:
|
||
GPU 0: 4 models
|
||
- Wan2_2-T2V-A14B-HIGH (checkpoint): 8704 MB
|
||
- sd3_medium_incl_clips (checkpoint): 6144 MB
|
||
- flux_vae (vae): 512 MB
|
||
- controlnet_union (controlnet): 1280 MB
|
||
Total model VRAM: 16.3 GB
|
||
System/Framework overhead: 5.7 GB
|
||
|
||
💡 RECOMMENDATION: Unload 'sd3_medium_incl_clips' (6.0 GB) before WanVideoWrapper
|
||
This would bring pre-execution VRAM from 87.3% to 60.1%
|
||
```
|
||
|
||
### Common OOM Scenarios & Solutions
|
||
|
||
#### Scenario 1: Large Checkpoint + Large Video Model
|
||
**Symptoms**: VRAM >90% before video generation, OOM during forward pass
|
||
**Solution**:
|
||
```python
|
||
import comfy.model_management as mm
|
||
mm.unload_all_models() # Before WanVideoWrapper node
|
||
mm.soft_empty_cache()
|
||
```
|
||
|
||
#### Scenario 2: Batch Size Too Large
|
||
**Symptoms**: OOM during KSampler with high batch size
|
||
**Solution**: Reduce `batch_size` or enable `--lowvram` mode
|
||
|
||
#### Scenario 3: Memory Fragmentation
|
||
**Symptoms**: Free VRAM available but allocation fails, OOM with <80% usage
|
||
**Solution**: Restart ComfyUI to defragment GPU memory
|
||
|
||
### OOM Prevention Best Practices
|
||
|
||
1. **Monitor Pre-OOM Warnings**: Act on ⚠️ (85%) and 🔴 (95%) warnings
|
||
2. **Unload Models Proactively**: Clear models between workflow stages
|
||
3. **Use --lowvram Mode**: For workflows consistently using >80% VRAM
|
||
4. **Reduce Resolution First**: Lower image/video resolution before quality
|
||
5. **Check Model Combinations**: Use OOM analysis to identify problematic combos
|
||
6. **Archive OOM History**: Review patterns to optimize workflows
|
||
|
||
## Phase 6.4: Frontend OOM UI Features
|
||
|
||
### Visual OOM Indicators
|
||
|
||
**Stats Card Warning Borders**
|
||
- Stats cards in the profiler popup show warning borders when VRAM usage is high
|
||
- ⚠️ **Yellow Border (85-95% VRAM)**: Warning threshold reached
|
||
- Border: 2px solid #f59e0b with glow shadow
|
||
- Applied to Time, VRAM Peak, and RAM Peak cards
|
||
- 🔴 **Red Border (>95% VRAM)**: Critical threshold - OOM imminent
|
||
- Border: 2px solid #ef4444 with red glow shadow
|
||
- High risk of OOM in next node execution
|
||
|
||
**Node Table 💥 Column**
|
||
- New column in the "Top 10 Slowest Nodes" table shows OOM status
|
||
- 💥 icon appears for nodes that experienced Out of Memory errors
|
||
- Empty cell for nodes that executed successfully
|
||
- Tooltip shows "Out of Memory Error" or "No OOM" on hover
|
||
|
||
### 💥 OOM Analysis Tab
|
||
|
||
The profiler modal now includes a dedicated "💥 OOM Analysis" tab showing comprehensive OOM statistics:
|
||
|
||
**1. Summary Cards**
|
||
- **💥 Total OOMs**: Count of all OOM events (red if >0)
|
||
- **📈 Total Workflows**: Number of workflow executions tracked
|
||
- **📉 OOM Rate**: Percentage of workflows with OOMs (yellow if >10%)
|
||
|
||
**2. Recent OOM Events**
|
||
- Chronological list of recent OOM occurrences
|
||
- Each event shows:
|
||
- Node type that failed (e.g., "💥 WanVideoSampler")
|
||
- Timestamp of the OOM
|
||
- 🎮 VRAM at OOM and percentage used
|
||
- 📈 VRAM peak during execution
|
||
- ⏱️ Execution time before OOM
|
||
- 📦 Models loaded at time of OOM with VRAM usage
|
||
- ⚠️ Error message from the exception
|
||
|
||
**3. Node Type Ranking**
|
||
- Table showing which node types are most OOM-prone
|
||
- Columns:
|
||
- **Node Type**: Name of the node
|
||
- **OOMs**: Total OOM count for this node type
|
||
- **Total Executions**: How many times the node ran
|
||
- **OOM Rate**: Percentage (color-coded: >50% red, >10% yellow)
|
||
- **Avg VRAM at OOM**: Average VRAM usage when OOMs occur
|
||
- **Last OOM**: Date of most recent OOM
|
||
|
||
**4. Model Correlation**
|
||
- Shows which model combinations correlate with OOMs
|
||
- Identifies patterns like "BiRefNet + WanVideoSampler = OOM"
|
||
- Displays:
|
||
- Node type
|
||
- Models loaded during OOM
|
||
- OOM count for this combination
|
||
- Total VRAM of the model set
|
||
|
||
**5. Recommendations**
|
||
- Auto-generated actionable recommendations based on OOM patterns
|
||
- Severity-coded with icons:
|
||
- 🛑 **Critical**: Node OOM rate >50% - immediate action needed
|
||
- ⚠️ **Warning**: Model combination caused ≥2 OOMs
|
||
- 💡 **Info**: General recommendations for >10% OOM rate
|
||
- Each recommendation includes:
|
||
- Title describing the issue
|
||
- Detailed message with specific actions
|
||
|
||
### CSS Classes for OOM Styling
|
||
|
||
```css
|
||
/* Warning borders for stats cards */
|
||
.profiler-stat-card-warning { /* Yellow border at 85% VRAM */ }
|
||
.profiler-stat-card-critical { /* Red border at 95% VRAM */ }
|
||
|
||
/* OOM icon column */
|
||
.profiler-oom-icon { /* Center-aligned 💥 icon */ }
|
||
|
||
/* OOM Analysis tab components */
|
||
.oom-summary-cards { /* 3-column grid of summary cards */ }
|
||
.oom-summary-error { /* Red text for OOM count */ }
|
||
.oom-summary-warning { /* Yellow text for high OOM rate */ }
|
||
.oom-recent-item { /* Red border for OOM event cards */ }
|
||
.oom-table { /* Styled tables for rankings */ }
|
||
.oom-rate-critical { /* Red text for >50% OOM rate */ }
|
||
.oom-rate-warning { /* Yellow text for >10% OOM rate */ }
|
||
.oom-rec-critical { /* Red left border for critical recommendations */ }
|
||
.oom-rec-warning { /* Yellow left border for warnings */ }
|
||
.oom-rec-info { /* Blue left border for info */ }
|
||
.oom-success { /* Green success message when no OOMs */ }
|
||
```
|
||
|
||
### Accessing OOM Analysis
|
||
|
||
1. **During Workflow**: Watch for warning borders on stats cards in profiler popup
|
||
2. **After Execution**: Check 💥 column in node table to see which nodes failed
|
||
3. **Full Analysis**: Click profiler button → Open modal → "💥 OOM Analysis" tab
|
||
4. **Review History**: Analyze patterns across multiple workflows to identify problem nodes/models
|
||
|
||
### Implementation Phases
|
||
|
||
**Phase 6.1: Backend OOM Detection** ✅ COMPLETED
|
||
- [x] Extend NodeProfile/WorkflowProfile with OOM fields
|
||
- [x] Add pre-OOM warnings in start_node() (85%/95% thresholds)
|
||
- [x] Implement OOM exception capture in map_node_with_profiling()
|
||
- [x] Add OOM event logging with model snapshots
|
||
|
||
**Phase 6.2: Historical OOM Tracking** ✅ COMPLETED
|
||
- [x] Add oom_history to ProfilerManager
|
||
- [x] Implement _update_oom_stats() method
|
||
- [x] Extend archive structure with OOM data
|
||
- [x] Create GET /swissarmyknife/profiler/oom_stats endpoint
|
||
|
||
**Phase 6.3: Model Recommendations** 🚧 PARTIALLY COMPLETED
|
||
- [x] Track model snapshots at OOM
|
||
- [x] Implement basic recommendation algorithm
|
||
- [ ] Track model last-used timestamps (future enhancement)
|
||
- [ ] Implement advanced unload recommendation with staleness (future enhancement)
|
||
|
||
**Phase 6.4: Frontend OOM UI** ✅ COMPLETED
|
||
- [x] Add pre-OOM warning indicators to stats cards (yellow at 85%, red at 95%)
|
||
- [x] Add 💥 OOM column to node table in profiler popup
|
||
- [x] Create "💥 OOM Analysis" modal tab
|
||
- [x] Display OOM summary cards (total OOMs, rate, workflows)
|
||
- [x] Show recent OOM events with VRAM context
|
||
- [x] Display node type ranking by OOM frequency
|
||
- [x] Display model correlation table
|
||
- [x] Show recommendations with severity indicators
|