nodes/EnragedAntelope-comfyui-sdnq

Fork 0

Files

T

EnragedAntelope c0e01f4b05 chore: update dependencies and documentation for version 1.6.0

2026-04-13 08:48:55 -04:00

35 KiB

Raw Permalink Blame History

context.md - ComfyUI-SDNQ Development Context

Important

: Update this file after every significant change or discovery!

Project Status

Current Phase: PRODUCTION / MAINTENANCE Last Updated: 2026-04-12 Version: 1.5.4 Overall Progress: Production-ready. 30 SDNQ models in catalog. ~1839 lines in sampler.py.

✅ IMPLEMENTATION COMPLETE ✅

Standalone sampler node (SDNQSampler) is fully implemented and production-ready.

Upstream Review — 2026-04-12

Performed a full review of the sdnq library (v0.1.7) and the HuggingFace SDNQ collection.

Model Catalog: All 28 image generation models and 2 video models from the HuggingFace SDNQ collection are present in our catalog. The only model not included is Disty0/Qwen3-VL-32B-Instruct-SDNQ-uint4-svd-r32, which is a VLM (text output) and correctly excluded. Two legacy models (FLUX.1-dev-qint8, FLUX.1-dev-qint4) predate the SDNQ naming convention but remain valid.

SDNQ API: All v0.1.0→v0.1.7 changes are transparent runtime improvements (AMD RDNA2 Triton kernels, Intel Arc GPU support, memory-efficient large batch matmul). Our API usage is verified correct and backward compatible. No code changes required in sampler.py.

Actions taken: Updated catalog date, VLM exclusion note (with specific SDNQ repo ID), and bumped sdnq>=0.1.7 in requirements.txt and pyproject.toml.

Original Implementation (2025-12-09)

What's Been Done (This Session)

Research & Planning:
- Created ASSESSMENT.md documenting architectural incompatibility
- Created STANDALONE_SAMPLER_PLAN.md with detailed implementation steps
- Created RESEARCH_NOTES.md with verified API documentation
- Archived broken wrapper code to archive/ directory
Implementation (COMPLETE):
- Created nodes/sampler.py - Standalone SDNQ sampler node (585 lines, fully featured)
- Fixed nodes/__init__.py import error (was importing old loader)
- Updated main __init__.py to load new sampler
- All code based on verified APIs (NO assumptions)
Features Implemented: ✅ Model catalog dropdown (21 pre-configured SDNQ models from Disty0) ✅ HuggingFace auto-download with progress tracking and resume support ✅ Custom model path support ([Custom Path] option) ✅ Comprehensive error handling with helpful troubleshooting messages ✅ Graceful interruption support (InterruptedError handling) ✅ Widget tooltips (mouseover help for all parameters) ✅ ComfyUI V3 API compliance with V1 backward compatibility ✅ Pipeline caching for performance (avoids reloading same model) ✅ Detailed logging for debugging ([SDNQ Sampler] prefixed messages) ✅ Proper error categorization (ValueError, FileNotFoundError, Exception) ✅ LoRA support (local files and HuggingFace repos with strength control) ✅ Memory management modes (gpu/balanced/lowvram)
Key Design Decisions:
- Using DiffusionPipeline.from_pretrained() (auto-detects model type)
- Using enable_model_cpu_offload() for memory efficiency
- Pipeline caching to avoid reloading same model
- Proper PIL to ComfyUI IMAGE tensor conversion (NHWC, float32, 0-1 range)
- Integrated with existing core/ modules (registry.py, downloader.py, config.py)
- Verified all APIs from official docs before implementation
Bug Fixes (Post user QA feedback):
- Fixed core/__init__.py still importing wrapper.py (moved to archive)
- Added structure validation test
- Tested import chain properly before final commit
- Fixed FLUX.2 negative_prompt parameter error
- CRITICAL: Fixed CPU processing - was forcing CPU instead of GPU

Lessons Learned - Quality Assurance

CRITICAL LESSON #1: Always validate the full import chain before claiming code is ready!

What went wrong:

Moved wrapper.py to archive/ but forgot to update core/__init__.py
Didn't test imports in a clean environment
Claimed code was "production-ready" without proper validation

How to prevent:

✅ Created structure validation test (tests module imports without dependencies)
✅ Test import chain: main init → nodes.init → sampler → core modules
✅ Check for orphaned imports when moving/archiving files
✅ Never claim "ready" without running validation tests

Note: Can't test full imports in this environment (no torch/diffusers), but structure tests pass.

CRITICAL LESSON #2: Always check device placement - don't blindly follow old patterns!

What went wrong (Commit e0c1550):

Used enable_model_cpu_offload() by default, forcing CPU processing
Didn't verify device placement behavior for current versions
Failed to learn from context.md (lines 124-157 documented this EXACT issue before)
Failed to check archive code that worked correctly

The facts (December 2025, verified):

DiffusionPipeline.from_pretrained() loads to CPU by default (HuggingFace docs)
Must explicitly call .to("cuda") for GPU processing
enable_model_cpu_offload() is for low VRAM systems (12-16GB), not modern GPUs
User has torch 2.9.x + CUDA 12.8 - should use full GPU mode

How to prevent:

✅ Read context.md FIRST - lessons are documented there
✅ Check archive code to see what worked before
✅ Research current best practices (don't assume from old docs)
✅ Verify device placement in console logs
✅ Added memory_mode parameter: "gpu" (default), "balanced", "lowvram"

Sources verified:

HuggingFace Diffusers Memory Optimization
DiffusionPipeline Loading Guide
Context.md lines 124-157 (previous session documented this)

LoRA Support (COMPLETE):
- Added lora_path and lora_strength optional parameters
- Supports both local .safetensors files and HuggingFace repo IDs
- Automatic LoRA loading/unloading based on cache changes
- LoRA strength adjustment (-5.0 to +5.0, per user request)
- Integrated caching: prevents unnecessary reloads when using same LoRA
- Clears LoRA cache when model changes
Memory Management (UPDATED):
- Changed default memory_mode to "balanced" (per user request)
- Options: "gpu" (all on GPU, fastest), "balanced" (offloading, 12-16GB), "lowvram" (sequential, 8GB)
- Proper GPU placement with .to("cuda") for "gpu" mode
Scheduler Support (ENHANCED):
- Researched all available schedulers in diffusers 0.36.0 (December 2025)
- EXPANDED: Now supports 14 schedulers (1 flow-match + 13 traditional)
- Flow-based: FlowMatchEulerDiscreteScheduler (for FLUX/SD3/Qwen/Z-Image)
- Traditional: DPMSolver, UniPC, Euler, EulerAncestral, DDIM, Heun, KDPM2, DEIS, LMS, DDPM, PNDM
- Implemented swap_scheduler() method with all scheduler mappings
- Scheduler caching to avoid unnecessary swaps
- See UX_IMPROVEMENTS_RESEARCH.md for comprehensive scheduler research
Major UX Improvements (COMPLETE):
- LoRA Dropdown: Integrated with ComfyUI's folder_paths to show available LoRAs
- Default Negative Prompt: "blurry, low quality, distorted, deformed, ugly, bad anatomy, bad hands, text, watermark, signature"
- Logical Parameter Ordering: 5 clear groups (Model Selection → Prompts → Settings → Configuration → Enhancements)
- Multi-Model Scheduler Support: Both flow-based and traditional diffusion schedulers
- QA validated with test_ux_improvements.py (7/7 tests passed)

LoRA Integration Details

Dropdown Options:

[None]: No LoRA (default)
[Custom Path]: Use lora_custom_path for manual path or HuggingFace repo
Available LoRAs: Automatically populated from ComfyUI/models/loras/

Path Resolution:

Uses folder_paths.get_filename_list("loras") to discover available LoRAs
Uses folder_paths.get_folder_paths("loras") to build full paths
Supports subdirectories within loras folder
Falls back gracefully if folder_paths not available

Parameters:

lora_selection: Dropdown with available LoRAs
lora_custom_path: Custom path (only used when [Custom Path] selected)
lora_strength: -5.0 to +5.0 (negative values invert effect)

Scheduler Compatibility Matrix

Model Type	Compatible Schedulers	Default
FLUX/SD3/Qwen/Z-Image	FlowMatchEulerDiscreteScheduler	FlowMatch...
SDXL/SD1.5	DPMSolver, UniPC, Euler, EulerAncestral, DDIM, etc.	DPMSolverMultistep

CRITICAL: Wrong scheduler type produces broken/corrupted images!

Tooltip warns users about compatibility
Default changed to DPMSolverMultistepScheduler (works with SDXL, most common in catalog)

Parameter Organization

GROUP 1: MODEL SELECTION

model_selection
custom_model_path

GROUP 2: GENERATION PROMPTS 3. prompt 4. negative_prompt (now required with default value)

GROUP 3: GENERATION SETTINGS 5. steps 6. cfg 7. width 8. height 9. seed 10. scheduler (moved to settings, now visible to all users)

GROUP 4: MODEL CONFIGURATION 11. dtype 12. memory_mode 13. auto_download

GROUP 5: ENHANCEMENTS (Optional) 14. lora_selection 15. lora_custom_path 16. lora_strength

Key Research Findings (Full Scheduler List)

Flow-Based Models (FLUX, SD3, Qwen, Z-Image):

✅ FlowMatchEulerDiscreteScheduler (ONLY one that works)
❌ All traditional schedulers produce incorrect images

Traditional Diffusion Models (SDXL, SD1.5):

✅ DPMSolverMultistepScheduler (recommended, best speed/quality)
✅ UniPCMultistepScheduler (very fast, high quality)
✅ EulerDiscreteScheduler (simple, reliable)
✅ EulerAncestralDiscreteScheduler (creative results)
✅ DDIMScheduler (classic, deterministic)
✅ HeunDiscreteScheduler, KDPM2DiscreteScheduler, etc. (10+ more options)
❌ Flow-match schedulers don't work with traditional models

Sources:

SDXL Scheduler Testing
Stable Diffusion Samplers Guide
ML Guide to Schedulers
ComfyUI folder_paths.py
ComfyUI Folder Structure
All research documented in UX_IMPROVEMENTS_RESEARCH.md

CRITICAL BUG FIX (Post-UX Improvements):

Issue: FLUX.2 crashed when negative_prompt was provided (TypeError)
Root Cause: Default negative prompt added in UX improvements, but FLUX.2 doesn't support negative_prompt parameter
Previous Behavior: Hard crash with error message telling user to clear negative_prompt
New Behavior: Gracefully detects unsupported parameter, removes it, retries generation, logs warning to console
Fix Location: nodes/sampler.py:684-709 (try/except with retry logic)
Impact: With default negative prompt, FLUX.2 would crash on EVERY generation unless manually cleared
User Experience: Now seamless - FLUX.2 works out of the box despite having default negative prompt
Console Output: ⚠️ Pipeline Flux2Pipeline doesn't support negative_prompt - skipping it
QA validated with test_negative_prompt_fix.py (7/7 checks passed)

Graceful Parameter Handling

The node now implements intelligent parameter compatibility checking:

First Attempt: Try generation with all parameters (including negative_prompt if provided)
Error Detection: If TypeError about 'negative_prompt' occurs
Automatic Retry: Remove negative_prompt from kwargs and retry
User Notification: Log warning to console (non-intrusive)
Success: Generation continues successfully

This prevents crashes for:

FLUX.2 (doesn't support negative_prompt)
FLUX-schnell (cfg=0, ignores negative_prompt but accepts it)
Future models with different parameter signatures

Other unsupported parameters still raise helpful errors with troubleshooting tips.

Next Steps

User to test node in ComfyUI (all fixes applied)
Test LoRA support with real LoRA files (-5.0 to +5.0 range)
Test scheduler parameter (currently only one option, but implemented correctly)
Test model selection and auto-download with real SDNQ model
Fix any errors discovered during testing

PREVIOUS STATUS - Wrapper Approach (Archived)

Quick Reference

Key Commands

# Install dependencies
pip install -r requirements.txt

# Test in ComfyUI (restart server after changes)
# Node should appear under loaders/SDNQ

# Run a quick SDNQ test
python -c "from sdnq import SDNQConfig; print('SDNQ imported successfully')"

Key Files

CLAUDE.md - Development guide (read-only reference)
nodes/loader.py - Main loader node (✓ IMPLEMENTED)
core/wrapper.py - ComfyUI type wrappers (✓ IMPLEMENTED)
core/config.py - Configuration helpers (✓ IMPLEMENTED)
README.md - User documentation (✓ COMPLETE)

✅ FIXED ISSUES - GitHub Issue Resolution (2025-11-27 - Session 3)

Issue #12: torch.compile Compiler Error ✅ FIXED (Commits: `661db1e`, `0d9bc46`)

Problem:

RuntimeError: Compiler: cl is not found

Solution Implemented (Commit 661db1e):

Added automatic C++ compiler detection before model loading
Gracefully suppress torch.compile errors if compiler not found
Model still works with quantized weights (same memory savings, slightly slower)
Provides helpful setup instructions when compiler not available

CRITICAL BUG FIX (Commit 0d9bc46): Initial fix had a critical bug: torch._dynamo.config.disable = True was forcing CPU execution!

What was wrong:

Setting disable=True completely disabled torch.compile
SDNQ fallback path ran on CPU instead of GPU
Model loaded into system RAM instead of VRAM
User experienced slow CPU-only processing

Proper fix:

REMOVED: torch._dynamo.config.disable = True
KEPT: torch._dynamo.config.suppress_errors = True
Now torch.compile tries to compile, catches compiler error, falls back to GPU eager mode
Model still uses GPU/VRAM (same memory usage)
Only difference: No compiled optimizations (slightly slower than with compiler)

SDPA OPTIMIZATION (Commit a7bb90b): User pointed out: SDPA is better than eager and doesn't need compiler!

What was improved:

Added attn_implementation="sdpa" to pipeline loading
Model attention now uses SDPA (GPU-accelerated Flash Attention)
SDPA doesn't require C++ compiler (unlike torch.compile)
Much faster than pure eager mode

Two separate optimizations:

SDPA = Model attention (always fast, no compiler needed)
torch.compile = Weight dequantization (needs compiler, falls back to eager)

Performance without compiler:

✅ Attention: Fast (SDPA)
⚠️ Dequantization: Slower (eager fallback)
📊 Net result: Much better than full eager mode

Status: ✅ Fully optimized - Uses best available backend for each operation

Issue #9: hf-xet>=1.3.0 not found ✅ FIXED

Problem: Optional dependency marked as required, causing installation failures

Solution Implemented:

Removed hf-xet from main requirements.txt
Created requirements-optional.txt for optional dependencies
Updated README with clear optional installation instructions

Status: ✅ Fixed - hf-xet now properly optional

Issue #15: Requirements Installation Conflicts ✅ FIXED

Problem:

huggingface-hub>=1.1.0 conflicted with transformers
Duplicate transformers version specifications
diffusers 0.36.0 not on PyPI

Solution Implemented:

Relaxed huggingface-hub to >=0.20.0 (let dependencies resolve naturally)
Removed duplicate transformers entry
Changed diffusers to >=0.35.0 with GitHub install instructions in comments
Updated README with conflict resolution guide

Status: ✅ Fixed - dependencies now resolve correctly

🚨 CRITICAL: diffusers 0.36.0+ Breaking Changes (2025-11-27 - Session 3)

AutoPipeline Removed in diffusers 0.36.0.dev0 ✅ FIXED

Issue: ImportError when loading models with diffusers installed from GitHub (0.36.0.dev0)

ImportError: cannot import name 'AutoPipeline' from 'diffusers'

Root Cause:

AutoPipeline class was completely removed in diffusers 0.36.0
Only task-specific classes remain: AutoPipelineForText2Image, AutoPipelineForImage2Image, AutoPipelineForInpainting
NO video or multimodal AutoPipeline classes exist

Solution: Changed to DiffusionPipeline (base class)

DiffusionPipeline.from_pretrained() auto-detects pipeline type from model_index.json
Works with ALL model types: T2I, I2I, I2V, T2V, multimodal
Supports FLUX, Qwen, video models (Wan2.2), everything

Files Modified:

nodes/loader.py:
- Line 19: from diffusers import DiffusionPipeline (was AutoPipeline)
- Line 262: pipeline = DiffusionPipeline.from_pretrained(...) (was AutoPipeline)
README.md: Added critical section about diffusers 0.36.0+ requirement with GitHub install instructions
requirements.txt: Specifies diffusers>=0.36.0 (though 0.36.0 not yet on PyPI, must install from GitHub)

Status: ✅ Fixed and documented

Important: Users must install diffusers from GitHub until 0.36.0 is released on PyPI:

pip install git+https://github.com/huggingface/diffusers.git

✅ PROPER FIX IMPLEMENTED (2025-11-27 - Session 3)

Switched from State Dict Extraction to Wrapper Approach

What Was Wrong: Trying to force SDNQ quantized models through ComfyUI's native loaders

Extracted state dicts from diffusers pipeline components
Tried to pass to comfy.sd.load_diffusion_model_state_dict()
Added bias injection hacks to fix missing keys
All of this was fundamentally wrong for SDNQ models

The Problem:

KeyError: 'x_embedder.bias'  # ComfyUI expects standard checkpoint format
RuntimeError: tensor size mismatch  # Quantized weights incompatible with bias

PROPER FIX: Use wrapper approach (as originally planned in CLAUDE.md)

REMOVED (179 lines of hacks):
- State dict extraction from pipeline components
- Bias injection for missing keys
- ComfyUI's load_diffusion_model_state_dict() calls
- _extract_clip_state_dicts() helper method
- comfy.sd import
ADDED (32 lines of proper code):
- Use wrap_pipeline_components() from core/wrapper.py
- Keep diffusers pipeline intact with quantized weights
- Apply SDNQ optimizations directly to pipeline components
- Return wrapped MODEL/CLIP/VAE that implement ComfyUI interfaces

Why This Is Better:

✅ No monkeypatching or hacks
✅ Piggybacks on existing diffusers code (current versions)
✅ SDNQ quantized weights preserved in original format
✅ Wrappers implement proper ComfyUI interfaces (tokenize, encode_from_tokens, etc.)
✅ Minimal maintenance required
✅ Aligns with original CLAUDE.md architecture plan

Pre-load Cleanup:

Runs before EACH model load (defensive cleanup)
Clears gc, CUDA cache, torch dynamo state
Ensures clean state even after failures
Does NOT affect other workflows (only cleans before this node runs)

Status: ✅ Properly implemented, ready for testing

MAJOR REFACTOR (2025-11-27 - Session 2)

Complete Rewrite to Use ComfyUI Native Loading ✅ IN PROGRESS

Issue: Previous wrapper approach (SDNQModelWrapper, SDNQCLIPWrapper, SDNQVAEWrapper) was fundamentally broken:

Didn't create proper ComfyUI ModelPatcher objects
Missing latent_format attribute causing 'NoneType' object has no attribute 'latent_channels' error
Not compatible with ComfyUI's expected interfaces

Solution: Rewrote nodes/loader.py to use ComfyUI's native model loading functions:

Load SDNQ pipeline via diffusers (preserves pre-quantized weights)
Extract state dictionaries from transformer/unet, text_encoder(s), and VAE
Use ComfyUI's native loaders:
- comfy.sd.load_diffusion_model_state_dict() → creates proper ModelPatcher with latent_format
- comfy.sd.load_text_encoder_state_dicts() → creates proper CLIP object
- comfy.sd.VAE() → creates proper VAE object
Apply SDNQ Triton optimizations to model inside ModelPatcher (optional)

Key Changes:

Removed core/wrapper.py dependency (will delete file)
Removed cpu_offload option (ComfyUI handles model management)
Added _extract_clip_state_dicts() helper method
Now returns proper ComfyUI MODEL/CLIP/VAE objects, not custom wrappers
Quantized weights preserved through state_dict extraction
Triton optimizations applied after ComfyUI loading

Files Modified:

nodes/loader.py: Complete rewrite of load_model() method

Status: ✅ Implementation complete

AutoPipeline Fix for Video/Multimodal Models ✅ COMPLETE (2025-11-27 - Session 2)

Issue: Code used AutoPipelineForText2Image which fails for:

Video models (Wan2.2-I2V, Wan2.2-T2V)
Multimodal editing models (Qwen-Image-Edit)
Any non-T2I pipeline types

Solution: Changed to AutoPipeline.from_pretrained():

Auto-detects correct pipeline type from model_index.json
Supports all pipeline types: T2I, I2I, I2V, T2V, multimodal
FLUX.2 → Flux2Pipeline (T2I with optional image guidance)
Qwen-Image-Edit → QwenImageEditPipeline (requires input image + text)
Wan2.2-I2V/T2V → Video pipelines with temporal components

Research Findings:

FLUX.2 uses single Mistral Small 3.1 text encoder (vs FLUX.1's dual encoders)
Qwen-Image-Edit uses dual-path architecture: Qwen2.5-VL + VAE Encoder
All models still have transformer/unet components that can be extracted
ComfyUI has native support for both FLUX.2 and Qwen-Image-Edit

Files Modified:

nodes/loader.py:
- Changed import: added from diffusers import AutoPipeline
- Changed pipeline loading from AutoPipelineForText2Image to AutoPipeline
- Removed unused comfy.model_management import
- Added comprehensive comments about pipeline types

Compatibility Note:

Video and multimodal models will load correctly via AutoPipeline
Whether ComfyUI's load_diffusion_model_state_dict() recognizes all architectures needs testing
FLUX.2 and Qwen-Image-Edit should work (ComfyUI has native support)
Video model (Wan2.2) support in ComfyUI is unknown

Status: ✅ Fix implemented, needs user testing

Documentation Updates ✅ COMPLETE

Changes:

Updated README.md:
- Removed reference to CREDITS.md (now credits Disty0 directly in header)
- Added modern model examples (FLUX.2, Qwen, Z-Image, HunyuanImage3)
- Updated model count (21+ models)
- Removed cpu_offload from parameters (no longer needed)
- Updated Phase 3 status to complete
- Added SDNQ Model Quantizer documentation
- Removed inaccurate VRAM estimates
- Updated troubleshooting section
Removed CREDITS.md file
Updated context.md with complete session history

Status: ✅ Documentation complete

CRITICAL BUG FIXES (2025-11-27)

Files: nodes/loader.py

Completed Tasks

Phase 1: Project setup - Complete folder structure created
Phase 1: Core wrapper implementation - MODEL, CLIP, VAE wrappers implemented
Phase 1: Basic loader node - SDNQModelLoader fully functional
Phase 1: Test with real model - NEEDS TESTING BY USER
Phase 2: Model registry - Complete catalog with 21 SDNQ models (all verified)
Phase 2: HuggingFace downloader - Full implementation with Windows fixes
Phase 2: Catalog dropdown - Integrated into loader node with auto-download
Phase 2: Smart caching - ComfyUI models folder integration
Phase 2: Model metadata - Display VRAM, size, quality info
Phase 3: Quantization node - FULLY IMPLEMENTED (uses MODEL input)
Phase 3: V3 API schemas - V3 API via comfy_entrypoint()
Bug Fixes: All critical bugs fixed (Windows symlink, storage location, model_name)

Current Status (2025-11-27 - Session 2)

ALL PHASES COMPLETE! ✅

Completed in This Session:

✅ Rewrote loader to use ComfyUI native model loading (proper ModelPatcher/CLIP/VAE objects)
✅ Removed all inaccurate size/VRAM estimates from registry
✅ Reviewed quantizer node (no changes needed)
✅ Updated all documentation (README, context.md)
✅ Removed CREDITS.md (now credits in README header)

Ready for Testing:

Model loading with ComfyUI native integration
Proper MODEL/CLIP/VAE objects that work with KSampler and other nodes
21 pre-configured models
Auto-download and caching
Quantizer node for converting existing models

Current Blockers

NONE - All implementation complete! Ready for user testing!

Next Steps

Test model dropdown with auto-download
Verify caching works correctly
Test with various models (FLUX, SD3.5, SDXL)
Gather user feedback
Plan Phase 3 features

Lessons Learned

ComfyUI Type System

Wrapper Strategy: Created wrapper classes (SDNQModelWrapper, SDNQCLIPWrapper, SDNQVAEWrapper) that hold references to the full diffusers pipeline
Model Component: Can be either transformer (FLUX/SD3) or unet (SDXL/SD1.5)
Integration Point: Wrappers provide methods like get_model(), get_pipeline() for ComfyUI to access underlying components
Key Insight: Instead of trying to perfectly mimic ComfyUI's internal types, we wrap the diffusers components and trust that ComfyUI nodes can work with them

SDNQ Integration

Import Side-Effect: Simply importing from sdnq import SDNQConfig registers SDNQ into diffusers - no manual registration needed!
Transparent Loading: Models load via standard diffusers.AutoPipelineForText2Image.from_pretrained() - SDNQ detection is automatic
Optimization: apply_sdnq_options_to_model() applies Triton quantized matmul when available
Memory Management: enable_model_cpu_offload() provides additional VRAM savings

Diffusers Pipeline

Pipeline Components:
- transformer or unet: Main diffusion model
- text_encoder + tokenizer: Text encoding (CLIP/T5)
- vae: Image encoding/decoding
- scheduler: Noise scheduler (internal to pipeline)
Detection: Check for hasattr(pipeline, 'transformer') vs hasattr(pipeline, 'unet') to determine architecture
Local vs Remote: Use local_files_only=True for local paths, False for HuggingFace downloads

Code Snippets to Remember

Basic SDNQ Loading Pattern

from sdnq import SDNQConfig  # Registers SDNQ
import diffusers
import torch

# Load pipeline
pipe = diffusers.AutoPipelineForText2Image.from_pretrained(
    "Disty0/FLUX.1-dev-qint8",
    torch_dtype=torch.bfloat16,
    local_files_only=False
)

# Apply optimizations
from sdnq.loader import apply_sdnq_options_to_model
pipe.transformer = apply_sdnq_options_to_model(
    pipe.transformer,
    use_quantized_matmul=True
)

# Enable CPU offload
pipe.enable_model_cpu_offload()

Wrapper Usage

from core.wrapper import wrap_pipeline_components

model, clip, vae = wrap_pipeline_components(pipeline)
# Returns (SDNQModelWrapper, SDNQCLIPWrapper, SDNQVAEWrapper)

Project Structure Created

ComfyUI-SDNQ/
├── __init__.py                    ✓ V1 API with NODE_CLASS_MAPPINGS
├── nodes/
│   ├── __init__.py                ✓ Package exports
│   ├── loader.py                  ✓ SDNQModelLoader (PHASE 1 COMPLETE)
│   ├── quantizer.py               ⏳ Placeholder for Phase 3
│   └── catalog.py                 ⏳ Placeholder for Phase 2
├── core/
│   ├── __init__.py                ✓ Package exports
│   ├── config.py                  ✓ Helper functions (dtype conversion, etc.)
│   ├── wrapper.py                 ✓ ComfyUI type wrappers (MODEL, CLIP, VAE)
│   ├── registry.py                ✓ Model catalog (basic implementation)
│   └── downloader.py              ⏳ Placeholder for Phase 2
├── requirements.txt               ✓ All dependencies listed
├── install.py                     ✓ ComfyUI Manager install hook
├── pyproject.toml                 ✓ Modern Python packaging
├── LICENSE                        ✓ Apache 2.0 (already existed)
├── README.md                      ✓ Comprehensive user documentation
├── CREDITS.md                     ✓ Attribution to Disty0
├── CLAUDE.md                      ✓ Development guide (already existed)
├── SDNQ_ComfyUI_Development_Plan.md ✓ Feasibility analysis (already existed)
└── context.md                     ✓ This file!

Future TODOs

Immediate (Before Phase 2)

Test with real SDNQ model - Critical to validate wrapper approach
Test with KSampler - Verify MODEL output works with sampling
Test CLIP integration - Verify text encoding works
Test VAE integration - Verify image decoding works
Fix any type compatibility issues discovered during testing

Phase 2 Enhancements

Complete HuggingFace downloader with progress callbacks
Add model catalog dropdown to loader node
Implement local model scanning
Add model size/VRAM requirement display
Progress bar for downloads in ComfyUI UI

Phase 3 Advanced Features

Checkpoint quantization node (convert existing models)
V3 API schema support
LoRA support with SDNQ models
Batch quantization of multiple checkpoints
Memory usage reporting node
Video model support (Wan2.2, etc.)

Testing Notes

Models to Test (Priority Order)

Model	Repo ID	Priority	Status
FLUX.1-dev-qint8	Disty0/FLUX.1-dev-qint8	HIGH	⏳ Not tested
SD3.5-Large-qint8	Disty0/stable-diffusion-3.5-large-qint8	MEDIUM	⏳ Not tested
SDXL-base-qint8	Disty0/stable-diffusion-xl-base-1.0-qint8	MEDIUM	⏳ Not tested

Environments Tested

OS	Python	PyTorch	CUDA	Status
Linux	3.10+	2.0+	11.8+	⏳ Not tested

Test Checklist

Node appears in ComfyUI under loaders/SDNQ
Can input HuggingFace repo ID
Model downloads successfully (if not cached)
Model loads without errors
MODEL output connects to KSampler
CLIP output connects to CLIP Text Encode
VAE output connects to VAE Decode
Can generate an image end-to-end
Triton optimization works (if available)
CPU offload works
Error messages are helpful

Session Log

2025-11-26 - Session 1: Initial Implementation

Goal: Set up project structure and implement Phase 1 MVP

Achieved:

✅ Created complete folder structure
✅ Implemented all Phase 1 core modules:
- core/config.py - Configuration helpers
- core/wrapper.py - ComfyUI type wrappers (MODEL, CLIP, VAE)
- nodes/loader.py - SDNQModelLoader node with full functionality
✅ Created comprehensive documentation:
- README.md - User guide with installation, usage, troubleshooting
- CREDITS.md - Proper attribution to Disty0
- pyproject.toml - Modern packaging configuration
- install.py - ComfyUI Manager integration
✅ Created placeholder modules for Phase 2 & 3:
- core/registry.py - Model catalog (basic version)
- core/downloader.py - HuggingFace Hub downloader (placeholder)
- nodes/quantizer.py - Checkpoint quantization (placeholder)
- nodes/catalog.py - Model catalog display (placeholder)

Issues: None - development went smoothly!

Key Decisions Made:

Wrapper Approach: Created lightweight wrapper classes that hold pipeline references rather than trying to perfectly mimic ComfyUI's internal types
Error Handling: Added comprehensive error messages with troubleshooting hints
Phase Strategy: Implemented full Phase 1 with placeholders for Phase 2/3 to establish clear roadmap
Documentation First: Wrote comprehensive README before testing to clarify user experience

Next Session:

Deploy to a ComfyUI instance for testing
Test with real SDNQ model (Disty0/FLUX.1-dev-qint8)
Validate wrapper compatibility with KSampler and other nodes
Fix any issues discovered during integration testing
Consider Phase 2 implementation based on test results

2025-11-26 - Session 2: Phase 2 Implementation

Goal: Add convenient model dropdown with automatic downloading

Achieved:

✅ Expanded model registry to 9+ SDNQ models with complete metadata:
- FLUX variants (qint8, qint6, qint4, schnell)
- SD 3.5 models (Large, Large-Turbo, Medium)
- SDXL models (base qint8, base qint4)
- Each with VRAM requirements, download size, quality estimates
✅ Implemented full HuggingFace downloader:
- Progress tracking with size/speed display
- Smart caching (checks if model already downloaded)
- Resume support for interrupted downloads
- Parallel download threads (8 workers)
- Comprehensive error handling
✅ Updated SDNQModelLoader node with dropdown:
- model_selection dropdown with formatted names (includes VRAM info)
- Auto-download on first use
- Custom model support via --Custom Model-- option
- Model metadata display in console
- Cache detection and reuse
✅ Updated core package exports for all new functions
✅ Updated README with Phase 2 features and new workflow examples
✅ Updated documentation to reflect dropdown usage

Issues: None - implementation went smoothly!

Key Decisions Made:

Model Dropdown Format: Display as "ModelName [VRAM]" for easy selection
Caching Strategy: Use HuggingFace Hub's built-in caching via try_to_load_from_cache
Download Progress: Print to console (ComfyUI UI integration would require more work)
Model Priority: Added priority field to catalog for recommended ordering
Custom Model Option: Added --Custom Model-- dropdown entry for flexibility

New Features Summary:

📦 9+ pre-configured models in dropdown
⚡ Automatic download from HuggingFace on first use
💾 Smart caching - download once, use forever
📊 Model metadata display (VRAM, size, quality)
🔧 Custom model support for advanced users

Next Session:

Test dropdown functionality in real ComfyUI
Test auto-download with real internet connection
Verify caching works correctly
Test with multiple models
Consider Phase 3 features (quantization node, LoRA support)

Architecture Notes

Why This Approach Works

No Monkeypatching: SDNQ integrates via diffusers registration, so we don't need to modify ComfyUI core
Pipeline-Centric: We keep the full diffusers pipeline intact and expose components through wrappers
Flexible Wrappers: Wrappers provide both high-level (get_pipeline()) and low-level (get_model()) access
Type Detection: Dynamic detection of model architecture (FLUX vs SDXL) via hasattr() checks

Potential Issues to Watch For

Type Compatibility: ComfyUI may expect specific methods/attributes on MODEL/CLIP/VAE that our wrappers don't provide
- Mitigation: Wrappers expose underlying components via getter methods
Memory Management: Diffusers CPU offloading vs ComfyUI's model management
- Mitigation: Made cpu_offload optional, users can disable if conflicts arise
Scheduler Compatibility: ComfyUI may want to override the scheduler
- Mitigation: KSampler can access scheduler through pipeline
Text Encoding: CLIP wrapper's encode() method may not match ComfyUI's expected interface
- Mitigation: Simple wrapper, easy to adjust based on testing

Remember: This file is your memory between sessions. Future you will thank present you for detailed notes!

35 KiB Raw Permalink Blame History

context.md - ComfyUI-SDNQ Development Context

Project Status

Upstream Review — 2026-04-12

Original Implementation (2025-12-09)

What's Been Done (This Session)

Lessons Learned - Quality Assurance

LoRA Integration Details

Scheduler Compatibility Matrix

Parameter Organization

Key Research Findings (Full Scheduler List)

Graceful Parameter Handling

Next Steps

PREVIOUS STATUS - Wrapper Approach (Archived)

Quick Reference

Key Commands

Key Files

✅ FIXED ISSUES - GitHub Issue Resolution (2025-11-27 - Session 3)

Issue #12: torch.compile Compiler Error ✅ FIXED (Commits: 661db1e, 0d9bc46)

Issue #9: hf-xet>=1.3.0 not found ✅ FIXED

Issue #15: Requirements Installation Conflicts ✅ FIXED

🚨 CRITICAL: diffusers 0.36.0+ Breaking Changes (2025-11-27 - Session 3)

AutoPipeline Removed in diffusers 0.36.0.dev0 ✅ FIXED

✅ PROPER FIX IMPLEMENTED (2025-11-27 - Session 3)

Switched from State Dict Extraction to Wrapper Approach

MAJOR REFACTOR (2025-11-27 - Session 2)

Complete Rewrite to Use ComfyUI Native Loading ✅ IN PROGRESS

AutoPipeline Fix for Video/Multimodal Models ✅ COMPLETE (2025-11-27 - Session 2)

Documentation Updates ✅ COMPLETE

CRITICAL BUG FIXES (2025-11-27)

Completed Tasks

Current Status (2025-11-27 - Session 2)

Completed in This Session:

Ready for Testing:

Current Blockers

Next Steps

Lessons Learned

ComfyUI Type System

SDNQ Integration

Diffusers Pipeline

Code Snippets to Remember

Basic SDNQ Loading Pattern

Wrapper Usage

Project Structure Created

Future TODOs

Immediate (Before Phase 2)

Phase 2 Enhancements

Phase 3 Advanced Features

Testing Notes

Models to Test (Priority Order)

Environments Tested

Test Checklist

Session Log

2025-11-26 - Session 1: Initial Implementation

2025-11-26 - Session 2: Phase 2 Implementation

Architecture Notes

Why This Approach Works

Potential Issues to Watch For

35 KiB

Raw Permalink Blame History

Issue #12: torch.compile Compiler Error ✅ FIXED (Commits: `661db1e`, `0d9bc46`)