nodes/EnragedAntelope-comfy-modelopt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

PROJECT ON HOLD - PyTorch compatibility blocking inference

Browse Source 
Added prominent warnings to README and RESEARCH_NOTES documenting
project status and blocking issues.

## Blocking Issue
PyTorch/TorchScript compatibility prevents full end-to-end inference:
- Encode/decode operations don't work with quantized models in PyTorch
- Root cause: Outdated distributed API in public ModelOpt repo
- Workaround exists in Diffusers wrapper (not ComfyUI native)
- Fix pending: PyTorch upstream updates (PR submitted by @marduk191)

## Community Feedback (2025-11-10)
Source: @marduk191
- "encoding/decoding won't work in torch until they update it"
- "works fine in diffusers" but not in native PyTorch/ComfyUI
- Submitted PR to PyTorch for encoder script fixes
- Recommendation: "yea i'll just wait until the update lol"

## Additional Issue Identified
Storage Inefficiency:
- Current loader requires original checkpoint + quantized model = 2x storage
- Defeats purpose of quantization (should reduce storage, not increase)
- Root cause: mto.restore() needs base model architecture
- Potential fix: Save architecture metadata, create empty model from config
- Not implemented due to project hold

## What Works (v0.4.0)
✅ Model quantization - 794 modules unwrapped, 2382 quantizers inserted
✅ Save/Load pipeline - mto.save()/restore() implementation
✅ Calibration starts successfully

## What's Blocked
❌ Full end-to-end inference in ComfyUI
❌ Encode/decode operations
❌ Production use

## Documentation Updates
- README: Added prominent status warning at top
- RESEARCH_NOTES: Added "PROJECT STATUS - ON HOLD" section
- RESEARCH_NOTES: Documented blocking issues in detail
- RESEARCH_NOTES: Added community feedback references
- RESEARCH_NOTES: Linked to related PyTorch issues (#76726, #75005)
- Updated final status to "ON HOLD" with recommendations

Recommendation: Wait for PyTorch upstream fixes before resuming development.
This commit is contained in:
Claude
2025-11-10 21:09:31 +00:00
parent 4402a17ab1
commit ea4b970efe
2 changed files with 104 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+22
View File
@@ -1,10 +1,32 @@
# NVIDIA ModelOpt for ComfyUI
> **⚠️ PROJECT STATUS: ON HOLD (2025-11-10)**
>
> **Current Status**: This project is on hold pending PyTorch compatibility fixes.
>
> **Issue**: ModelOpt quantized models have known compatibility issues with PyTorch/TorchScript that prevent full end-to-end inference in ComfyUI. While quantization and save/load work correctly, encode/decode operations are currently blocked.
>
> **What Works**:
> - ✅ Model quantization (v0.3.0)
> - ✅ Save/Load pipeline (v0.4.0)
>
> **What's Blocked**:
> - ❌ Full inference in ComfyUI (pending PyTorch updates)
>
> **Alternative**: Works in Diffusers wrapper (confirmed by community testing)
>
> **Recommendation**: Wait for PyTorch upstream fixes before production use.
>
> See [RESEARCH_NOTES.md](RESEARCH_NOTES.md) for technical details.
---
Quantize and optimize Stable Diffusion models with NVIDIA ModelOpt directly in ComfyUI. Achieve **~2x faster inference** with INT8/FP8 quantization while maintaining image quality. Works with various diffusion model architectures including SDXL, SD1.5, SD3, and more.
![License](https://img.shields.io/badge/license-MIT-blue.svg)
![Python](https://img.shields.io/badge/python-3.10%2B-blue)
![CUDA](https://img.shields.io/badge/CUDA-12.0%2B-green)
![Status](https://img.shields.io/badge/status-on%20hold-yellow)
## ✨ Features
RESEARCH_NOTES.md
+82 -14
View File
@@ -2,7 +2,36 @@
**Purpose**: Track technical findings, known issues, and solutions for future development without relearning.
**Last Updated**: 2025-01-10
**Last Updated**: 2025-11-10
---
## ⚠️ PROJECT STATUS - ON HOLD
**BLOCKING ISSUE**: PyTorch/TorchScript compatibility issues with ModelOpt quantized models in ComfyUI.
**Source**: Community feedback from @marduk191 (2025-11-10):
- "encoding/decoding won't work in torch until they update it"
- "works fine in diffusers" but not in native PyTorch/ComfyUI
- Submitted PR to PyTorch for encoder script fixes
- Waiting for upstream PyTorch updates
**What Works**:
- ✅ Model quantization (v0.3.0) - 794 modules unwrapped, 2382 quantizers inserted
- ✅ Save/Load pipeline (v0.4.0) - `mto.save()`/`mto.restore()` implementation
- ✅ Calibration starts successfully
**What's Blocked**:
- ❌ Full end-to-end inference in ComfyUI (pending PyTorch updates)
- ❌ Encode/decode operations with quantized models
- ❌ Production use until compatibility issues resolved
**Alternative Path**:
- Works in Diffusers wrapper (confirmed by marduk)
- May work with TensorRT export (untested)
- Native ComfyUI support pending PyTorch fixes
**Recommendation**: Wait for PyTorch updates before continuing development.
---
@@ -280,14 +309,33 @@ base_model.load_state_dict(torch.load("weights.pth"))
## ⚠️ KNOWN LIMITATIONS
### Current Implementation
### Blocking Issues (Project on Hold)
1. **ComfyUI Model Quantization** ✅ SOLVED (v0.3.0)
1. **PyTorch/TorchScript Encode/Decode Incompatibility** ❌ BLOCKING
- **Issue**: Encode/decode operations don't work with quantized models in PyTorch
- **Source**: Community feedback (@marduk191, 2025-11-10)
- **Root Cause**: Outdated distributed API in public ModelOpt repo
- **Workaround**: Works in Diffusers wrapper (not ComfyUI native)
- **Fix**: Pending PyTorch upstream updates (PR submitted by marduk)
- **Impact**: Cannot do full end-to-end inference in ComfyUI
- **Status**: PROJECT ON HOLD until PyTorch fixes
2. **Storage Inefficiency** ❌ NOT FIXED
- **Issue**: Loader requires original checkpoint + quantized model = 2x storage
- **Problem**: Defeats purpose of quantization (should reduce storage, not increase)
- **Root Cause**: `mto.restore()` needs base model architecture to restore into
- **Potential Fix**: Save architecture metadata, create empty model from config
- **Status**: Not implemented due to project hold
- **Workaround**: Keep original checkpoint loaded in workflow session
### Resolved Issues
3. **ComfyUI Model Quantization** ✅ SOLVED (v0.3.0)
- Root cause WAS: Module type mismatch
- Solution: Module unwrapping (_unwrap_comfy_ops)
- Status: WORKING - 794 modules unwrapped, 2382 quantizers inserted
2. **Save/Load Pipeline** ✅ FIXED (v0.4.0)
4. **Save/Load Pipeline** ✅ FIXED (v0.4.0)
- **WAS BROKEN**: Lost quantizer infrastructure during save/load
- **FIX**: Complete rewrite using `mto.save()` and `mto.restore()`
- **Save**: `mto.save(diffusion_model, path)` preserves quantizers
@@ -295,11 +343,13 @@ base_model.load_state_dict(torch.load("weights.pth"))
- **Requirement**: Loader now needs base unquantized model as input
- **Status**: WORKING - Quantizer infrastructure preserved correctly
3. **No Verification of Quantization Quality**
### Future Improvements
5. **No Verification of Quantization Quality**
- Even if quantization works, no validation that output quality is acceptable
- Need to add: PSNR, SSIM, or visual comparison tests
4. **No Support for Non-UNet Components**
6. **No Support for Non-UNet Components**
- VAE: Not quantizable (per ModelOpt design)
- CLIP: Not quantizable
- Only UNet backbone can be quantized
@@ -460,15 +510,33 @@ mte.export_to_tensorrt(
- Module: `comfy.ops.disable_weight_init`
- Model: `comfy.ldm.modules.diffusionmodules.openaimodel.UNetModel`
### Related Issues
### Related Issues & Community Feedback
**PyTorch/TorchScript Compatibility** (BLOCKING - 2025-11-10):
- Source: @marduk191 community feedback
- Issue: "encoding/decoding won't work in torch until they update it"
- Status: PR submitted to PyTorch for encoder script fixes
- Workaround: Works in Diffusers wrapper, not native PyTorch/ComfyUI
- Related: [PyTorch Issue #76726](https://github.com/pytorch/pytorch/issues/76726) - TorchScript quantization runtime errors
- Related: [PyTorch Issue #75005](https://github.com/pytorch/pytorch/issues/75005) - Cannot create TorchScript from quantized models
**Storage Inefficiency** (IDENTIFIED - 2025-11-10):
- Current loader requires: Original checkpoint + Quantized model = 2x storage
- Problem: Defeats purpose of quantization (should reduce storage)
- Root Cause: `mto.restore()` needs base model architecture
- Potential Fix: Save architecture metadata, create empty model from config
- Status: Not implemented due to project hold
**ComfyUI Module Wrapping**:
- ModelOpt expects exact torch.nn types
- ComfyUI optimizes with custom wrappers
- Fundamental incompatibility requiring adapter layer
- ComfyUI optimizes with custom wrappers (`comfy.ops.disable_weight_init`)
- Solution: Module unwrapping before quantization (SOLVED v0.3.0)
---
**Status**: ✅ FULLY FUNCTIONAL - v0.4.0
- Module unwrapping: Working (794 modules, 2382 quantizers)
- Quantization: Working (FP8/INT8/INT4)
- Save/Load: Fixed with `mto.save()`/`mto.restore()`
- Complete end-to-end pipeline functional
**Status**: ⚠️ ON HOLD - v0.4.0
- Module unwrapping: ✅ Working (794 modules, 2382 quantizers)
- Quantization: ✅ Working (FP8/INT8/INT4)
- Save/Load: ✅ Fixed with `mto.save()`/`mto.restore()`
- Full Inference: ❌ BLOCKED (PyTorch compatibility pending)
- **Recommendation**: Wait for PyTorch updates before resuming