From b37d93aa18eed7443872900b2b03205186362b1d Mon Sep 17 00:00:00 2001 From: Brendan ONeil Date: Sat, 21 Jun 2025 02:16:02 -0400 Subject: [PATCH] Add files via upload --- README.md | 835 +++++++++++++----------- download_models.py | 163 +++-- minimax_mask_node_bmo.py | 1310 ++++++++++++++++++++------------------ pyproject.toml | 160 ++--- requirements.txt | 44 +- 5 files changed, 1327 insertions(+), 1185 deletions(-) diff --git a/README.md b/README.md index 4e276af..54fd307 100644 --- a/README.md +++ b/README.md @@ -1,389 +1,446 @@ -# ๐ŸŽฏ MiniMax-Remover for ComfyUI - -**High-quality video object removal with automatic model management and universal resolution support** - -

- Huggingface Model - Github - arXiv -

- ---- - -## ๐Ÿš€ **Major Improvements & Features** - -### โœจ **Latest Enhancements** -- โœ… **Auto-Download Models**: Zero-configuration setup - models download automatically on first use -- โœ… **Universal Resolution Support**: Works with ANY resolution input (480p to 4K+) -- โœ… **Smart VAE Compatibility**: Automatic dimension adjustment for perfect compatibility -- โœ… **OpenCV Conflict Resolution**: No more OpenCV reinstallation issues -- โœ… **DW Preprocessor Compatibility**: Works seamlessly with DWPose and other preprocessors -- โœ… **Tensor Dimension Fixes**: Eliminates all dimension mismatch errors -- โœ… **Professional Quality**: Proper VAE normalization for natural, high-quality results -- โš ๏ธPending GPU times will vary dramatically. 720p runs on an A6000 in about 2-3 minutes. Vertical formats and the larger the area to inpaint will increase time to process - -### ๐ŸŽฎ **Core Features** -- **Fast**: Only 6-12 inference steps, highly optimized -- **Robust**: Handles any mask/video resolution combination automatically -- **Plug-and-Play**: Install and use immediately - no manual setup required -- **Production Ready**: Comprehensive error handling and diagnostics - ---- - -## ๐Ÿ“ฅ **Installation** - -### **Method 1: ComfyUI Manager (Recommended)** -1. Open ComfyUI Manager -2. Click "Install Custom Nodes" -3. Search for "MiniMax-Remover" -4. Click Install and restart ComfyUI - -### **Method 2: Manual Installation** -```bash -cd ComfyUI/custom_nodes -git clone https://github.com/CasterPollux/MiniMax-Remover.git -cd MiniMax-Remover -# Install with CUDA support (recommended) -pip install -r requirements.txt -``` - -### **Method 3: Automatic Setup Script** -```bash -# Run the setup script for automatic ComfyUI integration -python setup_comfyui_integration_bmo.py -``` - ---- - -## ๐Ÿค– **Model Management - Auto-Download System** - -### **Zero-Configuration Experience (Default)** -When you first use the node: - -``` -๐Ÿ” Checking for MiniMax-Remover models... -๐Ÿ“ฅ Models not found locally. Starting automatic download... -๐ŸŒ Downloading MiniMax-Remover models to: ./models -๐Ÿ“Š Expected download size: ~25-30 GB -โณ This may take several minutes depending on your connection... -๐ŸŽ‰ Models downloaded successfully! -โœ… Processing your video... -``` - -### **Model Download Details** -- **Total Size**: ~25-30 GB (one-time download) -- **Components**: - - VAE (~3GB) - Video encoding/decoding - - Transformer (~25GB) - Main diffusion model - - Scheduler (~1KB) - Denoising control -- **Source**: [HuggingFace: zibojia/minimax-remover](https://huggingface.co/zibojia/minimax-remover) - -### **Storage Locations (Auto-Detected)** -The system automatically tries these locations: - -1. **Project Directory**: `MiniMax-Remover/models/` -2. **ComfyUI Models**: `ComfyUI/models/` -3. **User Cache**: `~/.cache/minimax-remover/` - -### **Manual Download Options** -If auto-download fails or you prefer manual control: - -```bash -# Option 1: Use built-in script -python download_models.py - -# Option 2: Direct HuggingFace download -huggingface-cli download zibojia/minimax-remover --local-dir ./models --local-dir-use-symlinks False - -# Option 3: Download to ComfyUI models directory -huggingface-cli download zibojia/minimax-remover --local-dir ComfyUI/models --local-dir-use-symlinks False -``` - -**๐Ÿ“‹ For detailed setup instructions, see [`AUTO_DOWNLOAD_GUIDE.md`](./AUTO_DOWNLOAD_GUIDE.md)** - ---- - -## ๐ŸŽฏ **Universal Resolution Support** - -### **ANY Resolution Now Supported!** -The node now handles **any resolution input** with **automatic compatibility**: - -- โœ… **Any input resolution** - 480p, 720p, 1080p, 4K, custom resolutions -- โœ… **Mixed mask/image sizes** - automatically handles different resolution inputs -- โœ… **VAE compatibility** - dimensions automatically adjusted for perfect compatibility -- โœ… **No configuration needed** - works automatically - -### **Example Resolution Fixes** -| Input Resolution | Problem | Auto-Fixed To | Result | -|------------------|---------|---------------|--------| -| `1000x1778` | 125x222.25 latent | `1008x1784` | 126x223 โœ… | -| `720x480` | No issue | `720x480` | 90x60 โœ… | -| `1920x1080` | 240x135 | `1920x1088` | 240x136 โœ… | - -**๐Ÿ“‹ For technical details, see [`TENSOR_DIMENSION_FIX_GUIDE.md`](./TENSOR_DIMENSION_FIX_GUIDE.md)** - ---- - -## ๐Ÿ› ๏ธ **Compatibility Fixes** - -### **OpenCV Conflict Resolution โœ…** - -**Problem Solved**: Users were experiencing OpenCV reinstallation prompts and conflicts. - -**Root Cause**: Multiple conflicting OpenCV version requirements across different files: -- `requirements.txt`: `opencv-python==4.5.5.64` (exact old version) -- `requirements_bmo.txt`: `opencv-python>=4.8.0` (newer minimum) -- `pyproject.toml`: `opencv-python>=4.7.0` (different minimum) - -**Solution Applied**: -- โœ… **Standardized OpenCV requirements** to `opencv-python>=4.5.0,<5.0.0` across all files -- โœ… **Safe installation procedures** that don't break existing OpenCV installations -- โœ… **Recovery scripts** for users with broken installations - -**Result**: No more OpenCV reinstallation prompts or conflicts! - -### **DW Preprocessor Compatibility โœ…** - -**Problem Solved**: Conflicts between MiniMax-Remover and DWPose/DW preprocessors. - -**Root Cause**: Multiple compatibility issues: -- PyTorch version conflicts (`torch==2.6` vs DWPose requirement `<2.4`) -- Segment Anything model loading conflicts -- ONNX runtime conflicts -- CUDA memory issues from simultaneous GPU usage - -**Solution Applied**: -- โœ… **Compatible PyTorch range**: `torch>=2.0.0,<2.5.0` for DWPose compatibility -- โœ… **TorchVision compatibility**: Proper version alignment -- โœ… **Memory management**: Smart GPU memory handling -- โœ… **Model isolation**: Prevents loading conflicts - -**Result**: MiniMax-Remover now works seamlessly with DWPose and other preprocessors! - -**๐Ÿ“‹ For detailed compatibility info, see [`DW_PREPROCESSOR_COMPATIBILITY.md`](./DW_PREPROCESSOR_COMPATIBILITY.md)** - ---- - -## ๐Ÿš€ **Usage** - -### **In ComfyUI** - -1. **Add the Node**: Look for "MiniMax-Remover (BMO)" in the MiniMax-Remover category - -2. **Connect Inputs**: - - `images`: Your video frames as IMAGE type - - `masks`: Your binary masks as MASK type - -3. **Configure (Optional)**: - - `num_inference_steps`: 12 (default, optimal quality/speed balance) - - `iterations`: 6 (mask expansion, default works for most cases) - - `seed`: Any number for reproducible results - - `auto_download`: True (default, enables automatic model management) - -4. **Run**: Perfect results with any resolution combination! - -### **Node Parameters** - -#### **Core Settings** -- **`auto_download`**: `True` (default) - Automatic model management -- **`num_inference_steps`**: `12` (default) - Quality/speed balance -- **`iterations`**: `6` (default) - Mask expansion iterations -- **`seed`**: `42` (default) - Random seed for reproducible results - -#### **Advanced Settings (Auto-Detected)** -- **`vae_path`**: `"auto"` (default) - Auto-detected VAE path -- **`transformer_path`**: `"auto"` (default) - Auto-detected Transformer path -- **`scheduler_path`**: `"auto"` (default) - Auto-detected Scheduler path - -### **What You'll See** - -#### **First Use (Auto-Download)** -``` -๐Ÿ”ง Using BMO MiniMax-Remover Pipeline (Flexible Resolution) -๐Ÿ“ Input Analysis: - Images: torch.Size([165, 1920, 1080, 3]) - Masks: torch.Size([165, 1024, 576, 1]) - Using input dimensions as target: 1920x1080 - -๐Ÿ”ง Auto-resizing for compatibility to 1920x1080 - VAE-compatible target: 1920x1088 - Resizing masks: 1024x576 -> 1920x1088 - -๐ŸŽ‰ Models downloaded successfully! -โœ… Processing complete - perfect results! -``` - -#### **Subsequent Uses (Instant)** -``` -๐Ÿ” Checking for MiniMax-Remover models... -โœ… Found existing models at: ./models -โœ… BMO MiniMax-Remover models loaded successfully! -๐Ÿš€ Processing your video... -``` - - ---- - -## ๐Ÿ”ง **Troubleshooting** - -### **Common Issues & Solutions** - -#### **Models Not Downloading** -```bash -# Check internet connection and try manual download -python download_models.py - -# Or use HuggingFace CLI -huggingface-cli download zibojia/minimax-remover --local-dir ./models -``` - -#### **PyTorch CUDA Issues** -```bash -# If PyTorch gets downgraded to CPU version during installation: -# 1. Reinstall PyTorch with CUDA support first -pip install torch>=2.0.0,<2.8.0 torchvision>=0.15.0,<0.20.0 --index-url https://download.pytorch.org/whl/cu121 - -# 2. Then install other dependencies -pip install -r requirements.txt --no-deps --force-reinstall diffusers transformers accelerate -``` - -#### **OpenCV Conflicts** -```bash -# Install with compatible versions -pip install -r requirements.txt - -# If issues persist, see INSTALLATION_GUIDE_OPENCV_FIX.md -``` - -#### **DW Preprocessor Conflicts** -```bash -# Use compatible PyTorch version -pip install "torch>=2.0.0,<2.5.0" "torchvision>=0.15.0,<0.20.0" - -# See DW_PREPROCESSOR_COMPATIBILITY.md for details -``` - -#### **Memory Issues** -- Use smaller input resolutions -- Enable model offloading in ComfyUI settings -- Close other GPU-intensive applications - -#### **Poor Quality Results** -- Ensure you're using the latest BMO version -- Try different seeds (use seed parameter) -- Adjust mask expansion (iterations parameter: 6-10) -- Check that masks are clean and binary - ---- - -## ๐Ÿ“ **Project Structure** - -``` -MiniMax-Remover/ -โ”œโ”€โ”€ ๐Ÿ“„ README.md (this file) -โ”œโ”€โ”€ ๐Ÿ”ง minimax_mask_node_bmo.py (main ComfyUI node) -โ”œโ”€โ”€ ๐Ÿ”ง pipeline_minimax_remover_bmo.py (processing pipeline) -โ”œโ”€โ”€ ๐Ÿ”ง transformer_minimax_remover.py (model architecture) -โ”œโ”€โ”€ ๐Ÿ“ฆ requirements.txt (dependencies) -โ”œโ”€โ”€ ๐Ÿ“ models/ (auto-downloaded models) -โ”‚ โ”œโ”€โ”€ vae/ -โ”‚ โ”œโ”€โ”€ transformer/ -โ”‚ โ””โ”€โ”€ scheduler/ -โ”œโ”€โ”€ ๐Ÿ“‹ Documentation/ -โ”‚ โ”œโ”€โ”€ AUTO_DOWNLOAD_GUIDE.md -โ”‚ โ”œโ”€โ”€ TENSOR_DIMENSION_FIX_GUIDE.md -โ”‚ โ”œโ”€โ”€ DW_PREPROCESSOR_COMPATIBILITY.md -โ”‚ โ”œโ”€โ”€ INSTALLATION_GUIDE_OPENCV_FIX.md -โ”‚ โ””โ”€โ”€ OPENCV_FIX_SUMMARY.md -โ””โ”€โ”€ ๐Ÿ› ๏ธ Scripts/ - โ”œโ”€โ”€ download_models.py - โ”œโ”€โ”€ setup_comfyui_integration_bmo.py - โ””โ”€โ”€ fix_comfyui_diffusers.py - ---- - -## ๐Ÿ“š **Documentation** - -### **Complete Guides Available** -- **[`AUTO_DOWNLOAD_GUIDE.md`](./AUTO_DOWNLOAD_GUIDE.md)** - Model download and management -- **[`TENSOR_DIMENSION_FIX_GUIDE.md`](./TENSOR_DIMENSION_FIX_GUIDE.md)** - Resolution compatibility details -- **[`DW_PREPROCESSOR_COMPATIBILITY.md`](./DW_PREPROCESSOR_COMPATIBILITY.md)** - DWPose integration guide -- **[`INSTALLATION_GUIDE_OPENCV_FIX.md`](./INSTALLATION_GUIDE_OPENCV_FIX.md)** - OpenCV issue resolution -- **[`OPENCV_FIX_SUMMARY.md`](./OPENCV_FIX_SUMMARY.md)** - Complete fix summary - -### **Support Resources** -- **GitHub Issues**: Report bugs and request features -- **Documentation**: Comprehensive guides for all features -- **Community**: Share workflows and tips - - - -### **Before vs After** - -#### **โŒ Old Experience** -``` -- Manual model download required (25GB) -- Resolution errors with non-standard sizes -- OpenCV conflicts breaking installations -- DW preprocessor incompatibility -- Complex setup and configuration -``` - -#### **โœ… New Experience** -``` -- Plug-and-play: install and use immediately -- Works with ANY resolution automatically -- No OpenCV conflicts or reinstallations -- Perfect DW preprocessor compatibility -- Professional-quality results every time -``` - - - ---- - -## ๐Ÿ”— **Links & Resources** - -### **Official Project** -- **[Original Paper](https://arxiv.org/abs/2505.24873)** - Technical details and methodology -- **[Official Repository](https://github.com/zibojia/MiniMax-Remover)** - Original implementation -- **[HuggingFace Models](https://huggingface.co/zibojia/minimax-remover)** - Pre-trained models -- **[Demo Page](https://minimax-remover.github.io)** - Live demonstrations - -### **ComfyUI Integration** -- **[ComfyUI](https://github.com/comfyanonymous/ComfyUI)** - Main ComfyUI project -- **[ComfyUI Manager](https://github.com/ltdrdata/ComfyUI-Manager)** - Node management tool - ---- - -## ๐Ÿ“ง **Contact & Support** - -- **Issues**: [GitHub Issues](https://github.com/YOUR_USERNAME/MiniMax-Remover/issues) -- **Email**: [19210240030@fudan.edu.cn](mailto:19210240030@fudan.edu.cn) -- **Documentation**: All guides included in repository - ---- - -## ๐Ÿ“œ **License** - -This project is licensed under the terms specified in the [LICENSE](./LICENSE) file. - ---- - -## ๐Ÿ™ **Credits** - -### **Original Authors** -Bojia Zi*, Weixuan Peng*, Xianbiao Qiโ€ , Jianan Wang, Shihao Zhao, Rong Xiao, Kam-Fai Wong -*Equal contribution. โ€ Corresponding author. - -### **ComfyUI Integration & Enhancements** -- Auto-download system implementation -- Universal resolution support -- OpenCV & DW preprocessor compatibility fixes -- Comprehensive documentation and guides - ---- - -**๐ŸŽจ Happy Video Inpainting! โœจ** - -*Experience professional-quality video object removal with zero configuration required.* +# ๐ŸŽฏ MiniMax-Remover for ComfyUI + +**High-quality video object removal with automatic model management and universal resolution support** + +

+ Huggingface Model + Github + arXiv +

+ +--- + +## ๐Ÿš€ **Major Improvements & Features** + +### โœจ **Latest Enhancements** +- โœ… **Auto-Download Models**: Zero-configuration setup - models download automatically on first use +- โœ… **Universal Resolution Support**: Works with ANY resolution input (480p to 4K+) +- โœ… **Smart VAE Compatibility**: Automatic dimension adjustment for perfect compatibility +- โœ… **OpenCV Conflict Resolution**: No more OpenCV reinstallation issues +- โœ… **DW Preprocessor Compatibility**: Works seamlessly with DWPose and other preprocessors +- โœ… **Tensor Dimension Fixes**: Eliminates all dimension mismatch errors +- โœ… **Professional Quality**: Proper VAE normalization for natural, high-quality results + +### ๐ŸŽฎ **Core Features** +- **Fast**: Only 6-12 inference steps, highly optimized +- **Robust**: Handles any mask/video resolution combination automatically +- **Plug-and-Play**: Install and use immediately - no manual setup required +- **Production Ready**: Comprehensive error handling and diagnostics + +--- + +## ๐Ÿ“ฅ **Installation** + +### **Method 1: ComfyUI Manager (Recommended)** +1. Open ComfyUI Manager +2. Click "Install Custom Nodes" +3. Search for "MiniMax-Remover" +4. Click Install and restart ComfyUI + +### **Method 2: Manual Installation** +```bash +cd ComfyUI/custom_nodes +git clone https://github.com/CasterPollux/MiniMax-Remover.git +cd MiniMax-Remover +# Install with CUDA support (recommended) +pip install -r requirements.txt +``` + +### **Method 3: Automatic Setup Script** +```bash +# Run the setup script for automatic ComfyUI integration +python setup_comfyui_integration_bmo.py +``` + +--- + +## ๐Ÿค– **Model Management - Auto-Download System** + +### **Zero-Configuration Experience (Default)** +When you first use the node: + +``` +๐Ÿ” Checking for MiniMax-Remover models... +๐Ÿ“ฅ Models not found locally. Starting automatic download... +๐ŸŒ Downloading MiniMax-Remover models to: ./models +๐Ÿ“Š Expected download size: ~25-30 GB +โณ This may take several minutes depending on your connection... +๐ŸŽ‰ Models downloaded successfully! +โœ… Processing your video... +``` + +### **Model Download Details** +- **Total Size**: ~25-30 GB (one-time download) +- **Components**: + - VAE (~3GB) - Video encoding/decoding + - Transformer (~25GB) - Main diffusion model + - Scheduler (~1KB) - Denoising control +- **Source**: [HuggingFace: zibojia/minimax-remover](https://huggingface.co/zibojia/minimax-remover) + +### **Storage Locations (Auto-Detected)** +The system automatically tries these locations in priority order: + +#### **Option 1: Custom Node Directory (Default)** +``` +ComfyUI/custom_nodes/MiniMax-Remover/models/ +โ”œโ”€โ”€ minimax_vae/ (VAE encoder/decoder) +โ”œโ”€โ”€ minimax_transformer/ (main diffusion model) +โ””โ”€โ”€ minimax_scheduler/ (denoising scheduler) +``` + +#### **Option 2: ComfyUI Models Directory** +``` +ComfyUI/models/ +โ”œโ”€โ”€ minimax_vae/ (MiniMax VAE models) +โ”œโ”€โ”€ minimax_transformer/ (MiniMax Transformer models) +โ””โ”€โ”€ minimax_scheduler/ (MiniMax Scheduler configs) +``` + +#### **Option 3: User Cache Directory (Fallback)** +``` +~/.cache/minimax-remover/models/ +โ”œโ”€โ”€ minimax_vae/ +โ”œโ”€โ”€ minimax_transformer/ +โ””โ”€โ”€ minimax_scheduler/ +``` +*Windows: `C:\Users\[Username]\.cache\minimax-remover\models\`* + +### **Manual Download Options** +If auto-download fails or you prefer manual control: + +```bash +# Option 1: Use built-in script +python download_models.py + +# Option 2: Direct HuggingFace download +huggingface-cli download zibojia/minimax-remover --local-dir ./models --local-dir-use-symlinks False +# Then rename folders for clarity: +mv ./models/vae ./models/minimax_vae +mv ./models/transformer ./models/minimax_transformer +mv ./models/scheduler ./models/minimax_scheduler + +# Option 3: Download to ComfyUI models directory +huggingface-cli download zibojia/minimax-remover --local-dir ComfyUI/models --local-dir-use-symlinks False +# Then rename folders for clarity: +mv ComfyUI/models/vae ComfyUI/models/minimax_vae +mv ComfyUI/models/transformer ComfyUI/models/minimax_transformer +mv ComfyUI/models/scheduler ComfyUI/models/minimax_scheduler +``` + +**๐Ÿ“‹ For detailed setup instructions, see [`AUTO_DOWNLOAD_GUIDE.md`](./AUTO_DOWNLOAD_GUIDE.md)** + +--- + +## ๐ŸŽฏ **Universal Resolution Support** + +### **ANY Resolution Now Supported!** +The node now handles **any resolution input** with **automatic compatibility**: + +- โœ… **Any input resolution** - 480p, 720p, 1080p, 4K, custom resolutions +- โœ… **Mixed mask/image sizes** - automatically handles different resolution inputs +- โœ… **VAE compatibility** - dimensions automatically adjusted for perfect compatibility +- โœ… **No configuration needed** - works automatically + +### **Example Resolution Fixes** +| Input Resolution | Problem | Auto-Fixed To | Result | +|------------------|---------|---------------|--------| +| `1000x1778` | 125x222.25 latent | `1008x1784` | 126x223 โœ… | +| `720x480` | No issue | `720x480` | 90x60 โœ… | +| `1920x1080` | 240x135 | `1920x1088` | 240x136 โœ… | + +**๐Ÿ“‹ For technical details, see [`TENSOR_DIMENSION_FIX_GUIDE.md`](./TENSOR_DIMENSION_FIX_GUIDE.md)** + +--- + +## ๐Ÿ› ๏ธ **Compatibility Fixes** + +### **OpenCV Conflict Resolution โœ…** + +**Problem Solved**: Users were experiencing OpenCV reinstallation prompts and conflicts. + +**Root Cause**: Multiple conflicting OpenCV version requirements across different files: +- `requirements.txt`: `opencv-python==4.5.5.64` (exact old version) +- `requirements_bmo.txt`: `opencv-python>=4.8.0` (newer minimum) +- `pyproject.toml`: `opencv-python>=4.7.0` (different minimum) + +**Solution Applied**: +- โœ… **Standardized OpenCV requirements** to `opencv-python>=4.5.0,<5.0.0` across all files +- โœ… **Safe installation procedures** that don't break existing OpenCV installations +- โœ… **Recovery scripts** for users with broken installations + +**Result**: No more OpenCV reinstallation prompts or conflicts! + +### **DW Preprocessor Compatibility โœ…** + +**Problem Solved**: Conflicts between MiniMax-Remover and DWPose/DW preprocessors. + +**Root Cause**: Multiple compatibility issues: +- PyTorch version conflicts (`torch==2.6` vs DWPose requirement `<2.4`) +- Segment Anything model loading conflicts +- ONNX runtime conflicts +- CUDA memory issues from simultaneous GPU usage + +**Solution Applied**: +- โœ… **Compatible PyTorch range**: `torch>=2.0.0,<2.5.0` for DWPose compatibility +- โœ… **TorchVision compatibility**: Proper version alignment +- โœ… **Memory management**: Smart GPU memory handling +- โœ… **Model isolation**: Prevents loading conflicts + +**Result**: MiniMax-Remover now works seamlessly with DWPose and other preprocessors! + +**๐Ÿ“‹ For detailed compatibility info, see [`DW_PREPROCESSOR_COMPATIBILITY.md`](./DW_PREPROCESSOR_COMPATIBILITY.md)** + +--- + +## ๐Ÿš€ **Usage** + +### **In ComfyUI** + +1. **Add the Node**: Look for "MiniMax-Remover (BMO)" in the MiniMax-Remover category + +2. **Connect Inputs**: + - `images`: Your video frames as IMAGE type + - `masks`: Your binary masks as MASK type + +3. **Configure (Optional)**: + - `num_inference_steps`: 12 (default, optimal quality/speed balance) + - `iterations`: 6 (mask expansion, default works for most cases) + - `seed`: Any number for reproducible results + - `auto_download`: True (default, enables automatic model management) + +4. **Run**: Perfect results with any resolution combination! + +### **Node Parameters** + +#### **Core Settings** +- **`auto_download`**: `True` (default) - Automatic model management +- **`num_inference_steps`**: `12` (default) - Quality/speed balance +- **`iterations`**: `6` (default) - Mask expansion iterations +- **`seed`**: `42` (default) - Random seed for reproducible results + +#### **Advanced Settings (Auto-Detected)** +- **`vae_path`**: `"auto"` (default) - Auto-detected VAE path +- **`transformer_path`**: `"auto"` (default) - Auto-detected Transformer path +- **`scheduler_path`**: `"auto"` (default) - Auto-detected Scheduler path + +### **What You'll See** + +#### **First Use (Auto-Download)** +``` +๐Ÿ”ง Using BMO MiniMax-Remover Pipeline (Flexible Resolution) +๐Ÿ“ Input Analysis: + Images: torch.Size([165, 1920, 1080, 3]) + Masks: torch.Size([165, 1024, 576, 1]) + Using input dimensions as target: 1920x1080 + +๐Ÿ”ง Auto-resizing for compatibility to 1920x1080 + VAE-compatible target: 1920x1088 + Resizing masks: 1024x576 -> 1920x1088 + +๐ŸŽ‰ Models downloaded successfully with descriptive names! +๐Ÿ“ MiniMax VAE: D:\ComfyUI\custom_nodes\MiniMax-Remover\models\minimax_vae +๐Ÿ“ MiniMax Transformer: D:\ComfyUI\custom_nodes\MiniMax-Remover\models\minimax_transformer +๐Ÿ“ MiniMax Scheduler: D:\ComfyUI\custom_nodes\MiniMax-Remover\models\minimax_scheduler +โœ… Processing complete - perfect results! +``` + +#### **Subsequent Uses (Instant)** +``` +๐Ÿ” Checking for MiniMax-Remover models... +โœ… Found existing models at: D:\ComfyUI\custom_nodes\MiniMax-Remover\models (descriptive names) + VAE: D:\ComfyUI\custom_nodes\MiniMax-Remover\models\minimax_vae + Transformer: D:\ComfyUI\custom_nodes\MiniMax-Remover\models\minimax_transformer + Scheduler: D:\ComfyUI\custom_nodes\MiniMax-Remover\models\minimax_scheduler +โœ… BMO MiniMax-Remover models loaded successfully! +๐Ÿš€ Processing your video... +``` + + +--- + +## ๐Ÿ”ง **Troubleshooting** + +### **Common Issues & Solutions** + +#### **Models Not Downloading** +```bash +# Check internet connection and try manual download +python download_models.py + +# Or use HuggingFace CLI +huggingface-cli download zibojia/minimax-remover --local-dir ./models +``` + +#### **PyTorch CUDA Issues** +```bash +# If PyTorch gets downgraded to CPU version during installation: +# 1. Reinstall PyTorch with CUDA support first +pip install torch>=2.0.0,<2.8.0 torchvision>=0.15.0,<0.20.0 --index-url https://download.pytorch.org/whl/cu121 + +# 2. Then install other dependencies +pip install -r requirements.txt --no-deps --force-reinstall diffusers transformers accelerate +``` + +#### **OpenCV Conflicts** +```bash +# Install with compatible versions +pip install -r requirements.txt + +# If issues persist, see INSTALLATION_GUIDE_OPENCV_FIX.md +``` + +#### **DW Preprocessor Conflicts** +```bash +# Use compatible PyTorch version +pip install "torch>=2.0.0,<2.5.0" "torchvision>=0.15.0,<0.20.0" + +# See DW_PREPROCESSOR_COMPATIBILITY.md for details +``` + +#### **Memory Issues** +- Use smaller input resolutions +- Enable model offloading in ComfyUI settings +- Close other GPU-intensive applications + +#### **Poor Quality Results** +- Ensure you're using the latest BMO version +- Try different seeds (use seed parameter) +- Adjust mask expansion (iterations parameter: 6-10) +- Check that masks are clean and binary + +--- + +## ๐Ÿ“ **Project Structure** + +The MiniMax-Remover models can be stored in two different locations. Check both locations to find your models: + +``` +MiniMax-Remover/ +โ”œโ”€โ”€ ๐Ÿ“„ README.md (this file) +โ”œโ”€โ”€ ๐Ÿ”ง minimax_mask_node_bmo.py (main ComfyUI node) +โ”œโ”€โ”€ ๐Ÿ”ง pipeline_minimax_remover_bmo.py (processing pipeline) +โ”œโ”€โ”€ ๐Ÿ”ง transformer_minimax_remover.py (model architecture) +โ”œโ”€โ”€ ๐Ÿ“ฆ requirements.txt (dependencies) +โ”œโ”€โ”€ ๐Ÿ“ models/ (auto-downloaded models with descriptive names) +โ”‚ โ”œโ”€โ”€ minimax_vae/ (VAE encoder/decoder) +โ”‚ โ”œโ”€โ”€ minimax_transformer/ (main diffusion model) +โ”‚ โ””โ”€โ”€ minimax_scheduler/ (denoising scheduler) +โ”œโ”€โ”€ ๐Ÿ“‹ Documentation/ +โ”‚ โ”œโ”€โ”€ AUTO_DOWNLOAD_GUIDE.md +โ”‚ โ”œโ”€โ”€ TENSOR_DIMENSION_FIX_GUIDE.md +โ”‚ โ”œโ”€โ”€ DW_PREPROCESSOR_COMPATIBILITY.md +โ”‚ โ”œโ”€โ”€ INSTALLATION_GUIDE_OPENCV_FIX.md +โ”‚ โ””โ”€โ”€ OPENCV_FIX_SUMMARY.md +โ””โ”€โ”€ ๐Ÿ› ๏ธ Scripts/ + โ”œโ”€โ”€ download_models.py + โ”œโ”€โ”€ setup_comfyui_integration_bmo.py + โ””โ”€โ”€ fix_comfyui_diffusers.py +``` + +### **Option 2: ComfyUI Models Directory Structure (Alternative)** +``` +ComfyUI/ +โ”œโ”€โ”€ models/ +โ”‚ โ”œโ”€โ”€ checkpoints/ (Stable Diffusion models) +โ”‚ โ”œโ”€โ”€ vae/ (Standard VAE models) +โ”‚ โ”œโ”€โ”€ loras/ (LoRA models) +โ”‚ โ”œโ”€โ”€ minimax_vae/ (๐ŸŽฏ MiniMax VAE - auto-downloaded here) +โ”‚ โ”œโ”€โ”€ minimax_transformer/ (๐ŸŽฏ MiniMax Transformer - auto-downloaded here) +โ”‚ โ””โ”€โ”€ minimax_scheduler/ (๐ŸŽฏ MiniMax Scheduler - auto-downloaded here) +โ””โ”€โ”€ custom_nodes/ + โ””โ”€โ”€ MiniMax-Remover/ + โ”œโ”€โ”€ ๐Ÿ“„ README.md + โ”œโ”€โ”€ ๐Ÿ”ง minimax_mask_node_bmo.py + โ”œโ”€โ”€ ๐Ÿ”ง pipeline_minimax_remover_bmo.py + โ””โ”€โ”€ ๐Ÿ“ฆ requirements.txt +``` + +**๐Ÿ’ก Tip**: When the node runs, it will display the **actual local paths** where your models are found, so you'll know exactly which location is being used. + +--- + +## ๐Ÿ“š **Documentation** + +### **Complete Guides Available** +- **[`AUTO_DOWNLOAD_GUIDE.md`](./AUTO_DOWNLOAD_GUIDE.md)** - Model download and management +- **[`TENSOR_DIMENSION_FIX_GUIDE.md`](./TENSOR_DIMENSION_FIX_GUIDE.md)** - Resolution compatibility details +- **[`DW_PREPROCESSOR_COMPATIBILITY.md`](./DW_PREPROCESSOR_COMPATIBILITY.md)** - DWPose integration guide +- **[`INSTALLATION_GUIDE_OPENCV_FIX.md`](./INSTALLATION_GUIDE_OPENCV_FIX.md)** - OpenCV issue resolution +- **[`OPENCV_FIX_SUMMARY.md`](./OPENCV_FIX_SUMMARY.md)** - Complete fix summary + +### **Support Resources** +- **GitHub Issues**: Report bugs and request features +- **Documentation**: Comprehensive guides for all features +- **Community**: Share workflows and tips + + + +### **Before vs After** + +#### **โŒ Old Experience** +``` +- Manual model download required (25GB) +- Resolution errors with non-standard sizes +- OpenCV conflicts breaking installations +- DW preprocessor incompatibility +- Complex setup and configuration +``` + +#### **โœ… New Experience** +``` +- Plug-and-play: install and use immediately +- Works with ANY resolution automatically +- No OpenCV conflicts or reinstallations +- Perfect DW preprocessor compatibility +- Professional-quality results every time +``` + + + +--- + +## ๐Ÿ”— **Links & Resources** + +### **Official Project** +- **[Original Paper](https://arxiv.org/abs/2505.24873)** - Technical details and methodology +- **[Official Repository](https://github.com/zibojia/MiniMax-Remover)** - Original implementation +- **[HuggingFace Models](https://huggingface.co/zibojia/minimax-remover)** - Pre-trained models +- **[Demo Page](https://minimax-remover.github.io)** - Live demonstrations + +### **ComfyUI Integration** +- **[ComfyUI](https://github.com/comfyanonymous/ComfyUI)** - Main ComfyUI project +- **[ComfyUI Manager](https://github.com/ltdrdata/ComfyUI-Manager)** - Node management tool + +--- + +## ๐Ÿ“ง **Contact & Support** + +- **Issues**: [GitHub Issues](https://github.com/YOUR_USERNAME/MiniMax-Remover/issues) +- **Email**: [19210240030@fudan.edu.cn](mailto:19210240030@fudan.edu.cn) +- **Documentation**: All guides included in repository + +--- + +## ๐Ÿ“œ **License** + +This project is licensed under the terms specified in the [LICENSE](./LICENSE) file. + +--- + +## ๐Ÿ™ **Credits** + +### **Original Authors** +Bojia Zi*, Weixuan Peng*, Xianbiao Qiโ€ , Jianan Wang, Shihao Zhao, Rong Xiao, Kam-Fai Wong +*Equal contribution. โ€ Corresponding author. + +### **ComfyUI Integration & Enhancements** +- Auto-download system implementation +- Universal resolution support +- OpenCV & DW preprocessor compatibility fixes +- Comprehensive documentation and guides + +--- + +**๐ŸŽจ Happy Video Inpainting! โœจ** + +*Experience professional-quality video object removal with zero configuration required.* \ No newline at end of file diff --git a/download_models.py b/download_models.py index 27a3a68..b48b7a1 100644 --- a/download_models.py +++ b/download_models.py @@ -1,67 +1,98 @@ -import os -import subprocess -import sys -from huggingface_hub import snapshot_download - -def download_sam_model(): - """Download SAM model if not already present""" - sam_path = "sam_vit_h_4b8939.pth" - if not os.path.exists(sam_path): - print("Downloading SAM model...") - url = "https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth" - try: - subprocess.run(["wget", url, "-O", sam_path], check=True) - print("SAM model downloaded successfully!") - except subprocess.CalledProcessError: - print("Error downloading SAM model. Please download manually from:") - print("https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth") - sys.exit(1) - else: - print("SAM model already exists.") - -def download_minimax_models(): - """Download Minimax Remover models if not already present""" - models_dir = "models" - if not os.path.exists(models_dir): - os.makedirs(models_dir) - - # Check if models are already downloaded - required_dirs = ["vae", "transformer", "scheduler"] - if all(os.path.exists(os.path.join(models_dir, d)) for d in required_dirs): - print("Minimax models already exist.") - return - - print("Downloading Minimax Remover models...") - try: - # Download using huggingface-cli - subprocess.run([ - "huggingface-cli", "download", - "zibojia/minimax-remover", - "--local-dir", models_dir, - "--local-dir-use-symlinks", "False" - ], check=True) - print("Minimax models downloaded successfully!") - except subprocess.CalledProcessError: - print("Error downloading Minimax models. Please download manually using:") - print("huggingface-cli download zibojia/minimax-remover --local-dir ./models --local-dir-use-symlinks False") - sys.exit(1) - -def main(): - print("Starting model downloads...") - download_sam_model() - download_minimax_models() - print("\nAll models downloaded successfully!") - print("\nProject structure should now look like this:") - print(""" - your_project/ - โ”œโ”€โ”€ sam_vit_h_4b8939.pth - โ”œโ”€โ”€ models/ - โ”‚ โ”œโ”€โ”€ vae/ - โ”‚ โ”œโ”€โ”€ transformer/ - โ”‚ โ””โ”€โ”€ scheduler/ - โ”œโ”€โ”€ minimax_sam_node.py - โ””โ”€โ”€ test_run.py - """) - -if __name__ == "__main__": +import os +import subprocess +import sys +from huggingface_hub import snapshot_download + +def download_sam_model(): + """Download SAM model if not already present""" + sam_path = "sam_vit_h_4b8939.pth" + if not os.path.exists(sam_path): + print("Downloading SAM model...") + url = "https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth" + try: + subprocess.run(["wget", url, "-O", sam_path], check=True) + print("SAM model downloaded successfully!") + except subprocess.CalledProcessError: + print("Error downloading SAM model. Please download manually from:") + print("https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth") + sys.exit(1) + else: + print("SAM model already exists.") + +def download_minimax_models(): + """Download Minimax Remover models if not already present""" + models_dir = "models" + if not os.path.exists(models_dir): + os.makedirs(models_dir) + + # Check if models are already downloaded (try descriptive names first) + descriptive_dirs = ["minimax_vae", "minimax_transformer", "minimax_scheduler"] + legacy_dirs = ["vae", "transformer", "scheduler"] + + if all(os.path.exists(os.path.join(models_dir, d)) for d in descriptive_dirs): + print("Minimax models already exist (descriptive names).") + return + elif all(os.path.exists(os.path.join(models_dir, d)) for d in legacy_dirs): + print("Minimax models already exist (legacy names).") + return + + print("Downloading Minimax Remover models...") + try: + # Download to temporary location first + temp_dir = os.path.join(models_dir, "_temp_download") + subprocess.run([ + "huggingface-cli", "download", + "zibojia/minimax-remover", + "--local-dir", temp_dir, + "--local-dir-use-symlinks", "False" + ], check=True) + + # Rename to descriptive folder names + import shutil + + rename_map = { + "vae": "minimax_vae", + "transformer": "minimax_transformer", + "scheduler": "minimax_scheduler" + } + + for old_name, new_name in rename_map.items(): + old_path = os.path.join(temp_dir, old_name) + new_path = os.path.join(models_dir, new_name) + + if os.path.exists(old_path): + if os.path.exists(new_path): + shutil.rmtree(new_path) + shutil.move(old_path, new_path) + print(f"๐Ÿ“ Moved {old_name} to {new_name}: {os.path.abspath(new_path)}") + + # Clean up temp directory + if os.path.exists(temp_dir): + shutil.rmtree(temp_dir) + + print("Minimax models downloaded successfully with descriptive names!") + except subprocess.CalledProcessError: + print("Error downloading Minimax models. Please download manually using:") + print("huggingface-cli download zibojia/minimax-remover --local-dir ./models --local-dir-use-symlinks False") + print("Then rename folders: vae->minimax_vae, transformer->minimax_transformer, scheduler->minimax_scheduler") + sys.exit(1) + +def main(): + print("Starting model downloads...") + download_sam_model() + download_minimax_models() + print("\nAll models downloaded successfully!") + print("\nProject structure should now look like this:") + print(""" + MiniMax-Remover/ + โ”œโ”€โ”€ sam_vit_h_4b8939.pth + โ”œโ”€โ”€ models/ + โ”‚ โ”œโ”€โ”€ minimax_vae/ (VAE encoder/decoder) + โ”‚ โ”œโ”€โ”€ minimax_transformer/ (main diffusion model) + โ”‚ โ””โ”€โ”€ minimax_scheduler/ (denoising scheduler) + โ”œโ”€โ”€ minimax_mask_node_bmo.py + โ””โ”€โ”€ pipeline_minimax_remover_bmo.py + """) + +if __name__ == "__main__": main() \ No newline at end of file diff --git a/minimax_mask_node_bmo.py b/minimax_mask_node_bmo.py index 2356990..ffa5816 100644 --- a/minimax_mask_node_bmo.py +++ b/minimax_mask_node_bmo.py @@ -1,629 +1,683 @@ -#!/usr/bin/env python3 -""" -BMO MiniMax-Remover ComfyUI Node -High-quality video object removal - based on official implementation -""" - -import os -import sys -import torch -import numpy as np -import cv2 -from typing import Optional, Union, List -import folder_paths -import comfy.model_management as model_management - -# Import the BMO pipeline -from pathlib import Path - -# Add current directory to path for imports -current_dir = os.path.dirname(os.path.abspath(__file__)) -if current_dir not in sys.path: - sys.path.append(current_dir) - -# Handle import path for ComfyUI -def get_comfyui_base_path(): - """Get the ComfyUI base path for proper imports""" - current_dir = Path(__file__).parent.absolute() - - # Look for ComfyUI base directory - comfyui_paths = [ - current_dir.parent.parent, # custom_nodes/minimax-remover-bmo/ - Path("D:/COMFY_UI/ComfyUI"), # Direct path - Path(os.environ.get("COMFYUI_BASE", "")) # Environment variable - ] - - for path in comfyui_paths: - if path.exists() and (path / "nodes.py").exists(): - return str(path) - - return str(current_dir.parent.parent) - -# Add ComfyUI to path -comfyui_base = get_comfyui_base_path() -if comfyui_base not in sys.path: - sys.path.insert(0, comfyui_base) - -try: - # ComfyUI imports - import comfy.model_management as model_management - print("โœ… ComfyUI model_management imported successfully") - COMFYUI_AVAILABLE = True -except ImportError: - print(f"โš ๏ธ Could not import ComfyUI model_management") - COMFYUI_AVAILABLE = False - # Fallback device management - class MockModelManagement: - @staticmethod - def get_torch_device(): - return torch.device("cuda" if torch.cuda.is_available() else "cpu") - model_management = MockModelManagement() - -# Import required modules -from diffusers.models import AutoencoderKLWan -from diffusers.schedulers import UniPCMultistepScheduler - -# Enhanced imports with path handling -current_dir = Path(__file__).parent -try: - # Try local import first - sys.path.insert(0, str(current_dir)) - from pipeline_minimax_remover_bmo import Minimax_Remover_Pipeline_BMO - from transformer_minimax_remover import Transformer3DModel - print("โœ… Local BMO modules imported successfully") -except ImportError as e: - print(f"โŒ Failed to import BMO modules: {e}") - # Attempt to find and import from different locations - possible_paths = [ - current_dir, - current_dir.parent, - Path(comfyui_base) / "custom_nodes" / "minimax-remover-bmo" - ] - - imported = False - for path in possible_paths: - try: - if str(path) not in sys.path: - sys.path.insert(0, str(path)) - from pipeline_minimax_remover_bmo import Minimax_Remover_Pipeline_BMO - from transformer_minimax_remover import Transformer3DModel - print(f"โœ… BMO modules imported from: {path}") - imported = True - break - except ImportError: - continue - - if not imported: - raise ImportError("Could not import BMO pipeline modules from any location") - -# Lazy imports for diffusers components -def lazy_import_diffusers(): - """Lazy import diffusers components to avoid import issues""" - global AutoencoderKLWan, UniPCMultistepScheduler, Minimax_Remover_Pipeline_BMO, Transformer3DModel - - if 'AutoencoderKLWan' not in globals(): - from diffusers.models import AutoencoderKLWan - from diffusers.schedulers import UniPCMultistepScheduler - from pipeline_minimax_remover_bmo import Minimax_Remover_Pipeline_BMO - from transformer_minimax_remover import Transformer3DModel - - return AutoencoderKLWan, UniPCMultistepScheduler, Minimax_Remover_Pipeline_BMO, Transformer3DModel - -class MinimaxRemoverBMONode: - """ - BMO MiniMax-Remover Node for ComfyUI - High-quality video object removal with separate model path inputs - """ - - def __init__(self): - self.pipe = None - self.device = model_management.get_torch_device() - self.comfyui_models_path = Path(comfyui_base) / "models" - - def auto_download_models(self, force_download=False): - """ - Automatically download models if they don't exist - Returns the paths to the downloaded models - """ - print("๐Ÿ” Checking for MiniMax-Remover models...") - - # Standard model locations to try - possible_locations = [ - # Project directory - Path("models"), - # ComfyUI models directory - self.comfyui_models_path, - # User's cache directory - Path.home() / ".cache" / "minimax-remover" - ] - - # Check if models already exist - for base_path in possible_locations: - vae_path = base_path / "vae" - transformer_path = base_path / "transformer" - scheduler_path = base_path / "scheduler" - - if (vae_path.exists() and transformer_path.exists() and scheduler_path.exists() and - (vae_path / "config.json").exists() and - (transformer_path / "config.json").exists() and - (scheduler_path / "scheduler_config.json").exists()): - - print(f"โœ… Found existing models at: {base_path}") - return str(vae_path), str(transformer_path), str(scheduler_path) - - if not force_download: - print("๐Ÿ“ฅ Models not found locally. Starting automatic download...") - - # Choose download location (prefer project directory, fallback to cache) - download_base = Path("models") - if not download_base.parent.exists() or not os.access(download_base.parent, os.W_OK): - download_base = Path.home() / ".cache" / "minimax-remover" / "models" - print(f"๐Ÿ“ Using cache directory for models: {download_base}") - - download_base.mkdir(parents=True, exist_ok=True) - - try: - print(f"๐ŸŒ Downloading MiniMax-Remover models to: {download_base}") - print(f"๐Ÿ“Š Expected download size: ~25-30 GB") - print(f"โณ This may take several minutes depending on your connection...") - - # Import huggingface_hub for downloading - try: - from huggingface_hub import snapshot_download - except ImportError: - print("โŒ huggingface_hub not found. Installing...") - import subprocess - subprocess.check_call([sys.executable, "-m", "pip", "install", "huggingface_hub"]) - from huggingface_hub import snapshot_download - - # Download with progress - snapshot_download( - repo_id="zibojia/minimax-remover", - local_dir=str(download_base), - local_dir_use_symlinks=False, - allow_patterns=["vae/*", "transformer/*", "scheduler/*"], - ignore_patterns=["*.git*", "README.md", "*.txt"] - ) - - # Verify download - vae_path = download_base / "vae" - transformer_path = download_base / "transformer" - scheduler_path = download_base / "scheduler" - - if (vae_path.exists() and transformer_path.exists() and scheduler_path.exists()): - print("๐ŸŽ‰ Models downloaded successfully!") - print(f"๐Ÿ“ VAE: {vae_path}") - print(f"๐Ÿ“ Transformer: {transformer_path}") - print(f"๐Ÿ“ Scheduler: {scheduler_path}") - return str(vae_path), str(transformer_path), str(scheduler_path) - else: - raise Exception("Download completed but models not found in expected locations") - - except Exception as e: - print(f"โŒ Auto-download failed: {e}") - print("\n๐Ÿ”ง Manual download options:") - print("1. Run: python download_models.py") - print("2. Run: huggingface-cli download zibojia/minimax-remover --local-dir ./models") - print("3. See MODEL_DOWNLOAD_GUIDE.md for detailed instructions") - - # Return default paths so user can manually configure - return "models/vae/", "models/transformer/", "models/scheduler/" - - @classmethod - def INPUT_TYPES(cls): - return { - "required": { - "images": ("IMAGE",), - "masks": ("MASK",), - "num_inference_steps": ("INT", { - "default": 12, - "min": 6, - "max": 50, - "step": 1, - "display": "number" - }), - "iterations": ("INT", { - "default": 6, - "min": 1, - "max": 20, - "step": 1, - "display": "number" - }), - "seed": ("INT", { - "default": 42, - "min": 0, - "max": 0xffffffffffffffff, - "display": "number" - }), - }, - "optional": { - "auto_download": ("BOOLEAN", { - "default": True, - "tooltip": "Automatically download models if not found" - }), - "vae_path": ("STRING", { - "default": "auto", - "multiline": False, - "tooltip": "Path to VAE model directory (auto = automatic detection)" - }), - "transformer_path": ("STRING", { - "default": "auto", - "multiline": False, - "tooltip": "Path to Transformer model directory (auto = automatic detection)" - }), - "scheduler_path": ("STRING", { - "default": "auto", - "multiline": False, - "tooltip": "Path to Scheduler config directory (auto = automatic detection)" - }), - } - } - - RETURN_TYPES = ("IMAGE",) - RETURN_NAMES = ("images",) - FUNCTION = "process_video" - CATEGORY = "MiniMax-Remover" - DESCRIPTION = "BMO MiniMax video object removal with separate model paths" - - def resolve_model_path(self, model_path: str, model_type: str) -> str: - """ - Resolve model path with intelligent fallbacks - - Args: - model_path: User-provided path (can be relative or absolute) - model_type: Type of model (vae, transformer, scheduler) - - Returns: - Absolute path to model directory - """ - # Convert to Path object - path = Path(model_path) - - # If absolute path and exists, use it - if path.is_absolute() and path.exists(): - print(f"โœ… Using absolute path for {model_type}: {path}") - return str(path) - - # Try relative to ComfyUI base - comfyui_relative = Path(comfyui_base) / model_path - if comfyui_relative.exists(): - print(f"โœ… Using ComfyUI relative path for {model_type}: {comfyui_relative}") - return str(comfyui_relative) - - # Try in ComfyUI models directory - models_path = self.comfyui_models_path / model_type - if models_path.exists(): - print(f"โœ… Using ComfyUI models path for {model_type}: {models_path}") - return str(models_path) - - # Try original path as-is (fallback) - print(f"โš ๏ธ Using fallback path for {model_type}: {path}") - return str(path) - - def load_models(self, vae_path: str, transformer_path: str, scheduler_path: str, auto_download: bool = True): - """Load MiniMax-Remover models from paths with auto-download support""" - if self.pipe is not None: - print("โ„น๏ธ Models already loaded, skipping...") - return # Already loaded - - print("๐Ÿ”„ Loading BMO MiniMax-Remover models...") - print(f"๐Ÿ—‚๏ธ ComfyUI base path: {comfyui_base}") - print(f"๐Ÿ—‚๏ธ ComfyUI models path: {self.comfyui_models_path}") - - # Handle auto-download and path resolution - if auto_download and (vae_path == "auto" or transformer_path == "auto" or scheduler_path == "auto"): - print("๐Ÿค– Auto-download mode enabled") - try: - auto_vae, auto_transformer, auto_scheduler = self.auto_download_models() - - # Use auto-detected paths for "auto" values - if vae_path == "auto": - vae_path = auto_vae - if transformer_path == "auto": - transformer_path = auto_transformer - if scheduler_path == "auto": - scheduler_path = auto_scheduler - - print(f"๐ŸŽฏ Auto-resolved paths:") - print(f" VAE: {vae_path}") - print(f" Transformer: {transformer_path}") - print(f" Scheduler: {scheduler_path}") - - except Exception as e: - print(f"โš ๏ธ Auto-download failed, falling back to manual paths: {e}") - # Keep original paths if auto-download fails - - try: - # Resolve model paths - resolved_vae_path = self.resolve_model_path(vae_path, "vae") - resolved_transformer_path = self.resolve_model_path(transformer_path, "transformer") - resolved_scheduler_path = self.resolve_model_path(scheduler_path, "scheduler") - - print(f"๐Ÿ“ Loading VAE from: {resolved_vae_path}") - print(f"๐Ÿ“ Loading Transformer from: {resolved_transformer_path}") - print(f"๐Ÿ“ Loading Scheduler from: {resolved_scheduler_path}") - - # Check if paths exist before attempting to load - missing_paths = [] - for name, path in [("VAE", resolved_vae_path), ("Transformer", resolved_transformer_path), ("Scheduler", resolved_scheduler_path)]: - if not Path(path).exists(): - missing_paths.append(f"{name}: {path}") - - if missing_paths: - print(f"โŒ Missing model paths:") - for missing in missing_paths: - print(f" {missing}") - - if auto_download: - print("๐Ÿ”„ Attempting to re-download missing models...") - auto_vae, auto_transformer, auto_scheduler = self.auto_download_models(force_download=True) - resolved_vae_path = auto_vae - resolved_transformer_path = auto_transformer - resolved_scheduler_path = auto_scheduler - else: - raise FileNotFoundError(f"Models not found. Enable auto_download or check paths.") - - # Load models from resolved paths - AutoencoderKLWan, UniPCMultistepScheduler, Minimax_Remover_Pipeline_BMO, Transformer3DModel = lazy_import_diffusers() - - vae = AutoencoderKLWan.from_pretrained( - resolved_vae_path, - torch_dtype=torch.float16 - ) - transformer = Transformer3DModel.from_pretrained( - resolved_transformer_path, - torch_dtype=torch.float16 - ) - scheduler = UniPCMultistepScheduler.from_pretrained( - resolved_scheduler_path - ) - - # Create the BMO pipeline - self.pipe = Minimax_Remover_Pipeline_BMO( - vae=vae, - transformer=transformer, - scheduler=scheduler - ).to(self.device) - - print("โœ… BMO MiniMax-Remover models loaded successfully!") - print(f" Using device: {self.device}") - print(f" VAE: {type(vae).__name__}") - print(f" Transformer: {type(transformer).__name__}") - print(f" Scheduler: {type(scheduler).__name__}") - - except Exception as e: - print(f"โŒ Failed to load models: {e}") - print("๐Ÿ” Debug info:") - print(f" VAE path exists: {Path(resolved_vae_path).exists()}") - print(f" Transformer path exists: {Path(resolved_transformer_path).exists()}") - print(f" Scheduler path exists: {Path(resolved_scheduler_path).exists()}") - - # List available files for debugging - for name, path in [("VAE", resolved_vae_path), ("Transformer", resolved_transformer_path), ("Scheduler", resolved_scheduler_path)]: - if Path(path).exists(): - files = list(Path(path).glob("*")) - print(f" {name} directory contents: {[f.name for f in files]}") - - raise e - - def process_video( - self, - images, - masks, - num_inference_steps=12, - iterations=6, - seed=42, - auto_download=True, - vae_path="auto", - transformer_path="auto", - scheduler_path="auto" - ): - """ - Process video with BMO MiniMax-Remover with auto-download support - - Args: - images: Input video frames [B, H, W, C] in [0, 1] - masks: Binary masks [B, H, W] in [0, 1] - num_inference_steps: Number of denoising steps (official default: 12) - iterations: Mask expansion iterations (official default: 6) - seed: Random seed for reproducible results - auto_download: Automatically download models if not found - vae_path: Path to VAE model directory (auto = automatic detection) - transformer_path: Path to Transformer model directory (auto = automatic detection) - scheduler_path: Path to Scheduler config directory (auto = automatic detection) - - Returns: - Processed video frames [B, H, W, C] in [0, 1] - """ - - # Load models with auto-download support - self.load_models(vae_path, transformer_path, scheduler_path, auto_download) - - print("๐Ÿš€ Running BMO MiniMax-Remover") - print("=" * 50) - - # Convert ComfyUI tensors to the format expected by MiniMax - batch_size, height, width, channels = images.shape - num_frames = batch_size - - print(f"๐Ÿ“ Input: {images.shape} frames, {masks.shape} masks") - print(f"๐ŸŽฏ Parameters: steps={num_inference_steps}, iterations={iterations}, seed={seed}") - print(f"๐Ÿค– Auto-download: {'enabled' if auto_download else 'disabled'}") - - # Show resolved paths - if vae_path == "auto" or transformer_path == "auto" or scheduler_path == "auto": - print(f"๐Ÿ—‚๏ธ Using auto-detected model paths") - else: - print(f"๐Ÿ—‚๏ธ Model paths: VAE={vae_path}, Transformer={transformer_path}, Scheduler={scheduler_path}") - - # Prepare images: ComfyUI [B, H, W, C] -> MiniMax [F, H, W, C] -> [-1, 1] - images_np = images.detach().cpu().numpy() # [B, H, W, C] in [0, 1] - images_minimax = images_np * 2.0 - 1.0 # Convert to [-1, 1] for MiniMax - - # Prepare masks: ComfyUI [B, H, W] -> MiniMax [F, H, W, C] - if len(masks.shape) == 3: # [B, H, W] - masks_np = masks.detach().cpu().numpy() - masks_minimax = np.expand_dims(masks_np, axis=-1) # [F, H, W, 1] - else: # [B, H, W, C] - masks_minimax = masks.detach().cpu().numpy() - - print(f"๐Ÿ”„ Converted to MiniMax format:") - print(f" Images: {images_minimax.shape} [{images_minimax.min():.3f}, {images_minimax.max():.3f}]") - print(f" Masks: {masks_minimax.shape} [{masks_minimax.min():.3f}, {masks_minimax.max():.3f}]") - - # Convert to tensors - images_tensor = torch.from_numpy(images_minimax).float() - masks_tensor = torch.from_numpy(masks_minimax).float() - - # Set up generator for reproducible results - generator = torch.Generator(device=self.device).manual_seed(seed) - - # Run the BMO pipeline - print(f"๐Ÿ”ฅ Processing with BMO pipeline...") - - try: - with torch.no_grad(): - result = self.pipe( - images=images_tensor, - masks=masks_tensor, - num_frames=num_frames, - height=height, - width=width, - num_inference_steps=num_inference_steps, - iterations=iterations, - generator=generator, - output_type="np" # Get numpy output - ) - - # Extract frames - output_frames = result.frames - print(f"โœ… Pipeline completed!") - print(f"๐Ÿ“Š Output: {output_frames.shape} [{output_frames.min():.3f}, {output_frames.max():.3f}]") - - # Convert back to ComfyUI format [B, H, W, C] - if len(output_frames.shape) == 5: # [1, F, H, W, C] - output_frames = output_frames[0] # Remove batch dimension - - # Ensure we have the right number of frames - if output_frames.shape[0] != num_frames: - print(f"๐Ÿ”ง Adjusting frame count: {output_frames.shape[0]} -> {num_frames}") - if output_frames.shape[0] < num_frames: - # Pad by repeating last frame - last_frame = output_frames[-1:].repeat(num_frames - output_frames.shape[0], axis=0) - output_frames = np.concatenate([output_frames, last_frame], axis=0) - else: - # Truncate to desired number of frames - output_frames = output_frames[:num_frames] - - # Ensure correct shape and range - output_frames = np.clip(output_frames, 0.0, 1.0) - - print(f"๐Ÿ“ค Final output: {output_frames.shape} [{output_frames.min():.3f}, {output_frames.max():.3f}]") - - # Convert back to tensor for ComfyUI - result_tensor = torch.from_numpy(output_frames).float() - - return (result_tensor,) - - except Exception as e: - print(f"โŒ Processing failed: {e}") - import traceback - traceback.print_exc() - # Return original images as fallback - return (images,) - - def diagnose_inputs(self, images, masks): - """ - Diagnostic function to check input compatibility - Helps users identify potential dimension issues before processing - """ - print("๐Ÿ” DIAGNOSTIC MODE: Analyzing inputs for compatibility") - print("=" * 60) - - # Analyze images - if images is None: - print("โŒ ERROR: Images tensor is None") - return False - - print(f"๐Ÿ“Š Images analysis:") - print(f" Shape: {images.shape}") - print(f" Type: {type(images)}") - print(f" Data type: {images.dtype if hasattr(images, 'dtype') else 'N/A'}") - print(f" Range: [{images.min():.3f}, {images.max():.3f}]" if hasattr(images, 'min') else "") - - if len(images.shape) != 4: - print(f"โŒ ERROR: Expected 4D tensor [B, H, W, C], got {len(images.shape)}D") - return False - - batch_size, height, width, channels = images.shape - print(f" Frames: {batch_size}") - print(f" Resolution: {height}x{width}") - print(f" Channels: {channels}") - - # Analyze masks - if masks is None: - print("โŒ ERROR: Masks tensor is None") - return False - - print(f"\n Masks analysis:") - print(f" Shape: {masks.shape}") - print(f" Type: {type(masks)}") - print(f" Data type: {masks.dtype if hasattr(masks, 'dtype') else 'N/A'}") - print(f" Range: [{masks.min():.3f}, {masks.max():.3f}]" if hasattr(masks, 'min') else "") - - # Check dimension compatibility - print(f"\n๐Ÿ”ง Compatibility analysis:") - - # VAE spatial compatibility - vae_h = ((height + 7) // 8) * 8 - vae_w = ((width + 7) // 8) * 8 - if height == vae_h and width == vae_w: - print(f" โœ… Resolution {height}x{width} is VAE-compatible") - else: - print(f" โš ๏ธ Resolution {height}x{width} will be adjusted to {vae_h}x{vae_w} for VAE compatibility") - - # Temporal compatibility - vae_scale_factor_temporal = 4 # Default value - temporal_latent_frames = (batch_size - 1) // vae_scale_factor_temporal + 1 - print(f" ๐Ÿ“Š Temporal: {batch_size} frames โ†’ {temporal_latent_frames} latent frames") - - # Check common issues - issues = [] - - if channels != 3: - issues.append(f"Expected 3 channels (RGB), got {channels}") - - if batch_size < 1: - issues.append(f"Invalid frame count: {batch_size}") - - if len(masks.shape) not in [3, 4]: - issues.append(f"Masks should be 3D [F,H,W] or 4D [F,H,W,1], got {len(masks.shape)}D") - - if len(masks.shape) == 3 and masks.shape != (batch_size, height, width): - issues.append(f"Mask shape {masks.shape} doesn't match image frames") - - if len(masks.shape) == 4 and masks.shape != (batch_size, height, width, 1): - issues.append(f"Mask shape {masks.shape} doesn't match expected [F,H,W,1]") - - # Report results - if issues: - print(f"\nโŒ ISSUES DETECTED:") - for i, issue in enumerate(issues, 1): - print(f" {i}. {issue}") - return False - else: - print(f"\nโœ… ALL CHECKS PASSED - Inputs are compatible!") - return True - - -# ComfyUI Node Mappings -NODE_CLASS_MAPPINGS = { - "MinimaxRemoverBMO": MinimaxRemoverBMONode, -} - -NODE_DISPLAY_NAME_MAPPINGS = { - "MinimaxRemoverBMO": "MiniMax-Remover (BMO)", -} - -# Export for ComfyUI +#!/usr/bin/env python3 +""" +BMO MiniMax-Remover ComfyUI Node +High-quality video object removal - based on official implementation +""" + +import os +import sys +import torch +import numpy as np +import cv2 +from typing import Optional, Union, List +import folder_paths +import comfy.model_management as model_management + +# Import the BMO pipeline +from pathlib import Path + +# Add current directory to path for imports +current_dir = os.path.dirname(os.path.abspath(__file__)) +if current_dir not in sys.path: + sys.path.append(current_dir) + +# Handle import path for ComfyUI +def get_comfyui_base_path(): + """Get the ComfyUI base path for proper imports""" + current_dir = Path(__file__).parent.absolute() + + # Look for ComfyUI base directory + comfyui_paths = [ + current_dir.parent.parent, # custom_nodes/minimax-remover-bmo/ + Path("D:/COMFY_UI/ComfyUI"), # Direct path + Path(os.environ.get("COMFYUI_BASE", "")) # Environment variable + ] + + for path in comfyui_paths: + if path.exists() and (path / "nodes.py").exists(): + return str(path) + + return str(current_dir.parent.parent) + +# Add ComfyUI to path +comfyui_base = get_comfyui_base_path() +if comfyui_base not in sys.path: + sys.path.insert(0, comfyui_base) + +try: + # ComfyUI imports + import comfy.model_management as model_management + print("โœ… ComfyUI model_management imported successfully") + COMFYUI_AVAILABLE = True +except ImportError: + print(f"โš ๏ธ Could not import ComfyUI model_management") + COMFYUI_AVAILABLE = False + # Fallback device management + class MockModelManagement: + @staticmethod + def get_torch_device(): + return torch.device("cuda" if torch.cuda.is_available() else "cpu") + model_management = MockModelManagement() + +# Import required modules +from diffusers.models import AutoencoderKLWan +from diffusers.schedulers import UniPCMultistepScheduler + +# Enhanced imports with path handling +current_dir = Path(__file__).parent +try: + # Try local import first + sys.path.insert(0, str(current_dir)) + from pipeline_minimax_remover_bmo import Minimax_Remover_Pipeline_BMO + from transformer_minimax_remover import Transformer3DModel + print("โœ… Local BMO modules imported successfully") +except ImportError as e: + print(f"โŒ Failed to import BMO modules: {e}") + # Attempt to find and import from different locations + possible_paths = [ + current_dir, + current_dir.parent, + Path(comfyui_base) / "custom_nodes" / "minimax-remover-bmo" + ] + + imported = False + for path in possible_paths: + try: + if str(path) not in sys.path: + sys.path.insert(0, str(path)) + from pipeline_minimax_remover_bmo import Minimax_Remover_Pipeline_BMO + from transformer_minimax_remover import Transformer3DModel + print(f"โœ… BMO modules imported from: {path}") + imported = True + break + except ImportError: + continue + + if not imported: + raise ImportError("Could not import BMO pipeline modules from any location") + +# Lazy imports for diffusers components +def lazy_import_diffusers(): + """Lazy import diffusers components to avoid import issues""" + global AutoencoderKLWan, UniPCMultistepScheduler, Minimax_Remover_Pipeline_BMO, Transformer3DModel + + if 'AutoencoderKLWan' not in globals(): + from diffusers.models import AutoencoderKLWan + from diffusers.schedulers import UniPCMultistepScheduler + from pipeline_minimax_remover_bmo import Minimax_Remover_Pipeline_BMO + from transformer_minimax_remover import Transformer3DModel + + return AutoencoderKLWan, UniPCMultistepScheduler, Minimax_Remover_Pipeline_BMO, Transformer3DModel + +class MinimaxRemoverBMONode: + """ + BMO MiniMax-Remover Node for ComfyUI + High-quality video object removal with separate model path inputs + """ + + def __init__(self): + self.pipe = None + self.device = model_management.get_torch_device() + self.comfyui_models_path = Path(comfyui_base) / "models" + + def auto_download_models(self, force_download=False): + """ + Automatically download models if they don't exist + Returns the paths to the downloaded models + """ + print("๐Ÿ” Checking for MiniMax-Remover models...") + + # Standard model locations to try + possible_locations = [ + # Project directory + Path("models"), + # ComfyUI models directory + self.comfyui_models_path, + # User's cache directory + Path.home() / ".cache" / "minimax-remover" + ] + + # Check if models already exist (try both old and new naming) + for base_path in possible_locations: + # Try new descriptive names first + vae_path = base_path / "minimax_vae" + transformer_path = base_path / "minimax_transformer" + scheduler_path = base_path / "minimax_scheduler" + + if (vae_path.exists() and transformer_path.exists() and scheduler_path.exists() and + (vae_path / "config.json").exists() and + (transformer_path / "config.json").exists() and + (scheduler_path / "scheduler_config.json").exists()): + + print(f"โœ… Found existing models at: {os.path.abspath(base_path)} (descriptive names)") + print(f" VAE: {os.path.abspath(vae_path)}") + print(f" Transformer: {os.path.abspath(transformer_path)}") + print(f" Scheduler: {os.path.abspath(scheduler_path)}") + return str(vae_path), str(transformer_path), str(scheduler_path) + + # Fallback to old generic names for backward compatibility + vae_path_old = base_path / "vae" + transformer_path_old = base_path / "transformer" + scheduler_path_old = base_path / "scheduler" + + if (vae_path_old.exists() and transformer_path_old.exists() and scheduler_path_old.exists() and + (vae_path_old / "config.json").exists() and + (transformer_path_old / "config.json").exists() and + (scheduler_path_old / "scheduler_config.json").exists()): + + print(f"โœ… Found existing models at: {os.path.abspath(base_path)} (legacy names)") + print(f" VAE: {os.path.abspath(vae_path_old)}") + print(f" Transformer: {os.path.abspath(transformer_path_old)}") + print(f" Scheduler: {os.path.abspath(scheduler_path_old)}") + return str(vae_path_old), str(transformer_path_old), str(scheduler_path_old) + + if not force_download: + print("๐Ÿ“ฅ Models not found locally. Starting automatic download...") + + # Choose download location (prefer project directory, fallback to cache) + download_base = Path("models") + if not download_base.parent.exists() or not os.access(download_base.parent, os.W_OK): + download_base = Path.home() / ".cache" / "minimax-remover" / "models" + print(f"๐Ÿ“ Using cache directory for models: {download_base}") + + download_base.mkdir(parents=True, exist_ok=True) + + try: + print(f"๐ŸŒ Downloading MiniMax-Remover models to: {download_base}") + print(f"๐Ÿ“Š Expected download size: ~25-30 GB") + print(f"โณ This may take several minutes depending on your connection...") + + # Import huggingface_hub for downloading + try: + from huggingface_hub import snapshot_download + except ImportError: + print("โŒ huggingface_hub not found. Installing...") + import subprocess + subprocess.check_call([sys.executable, "-m", "pip", "install", "huggingface_hub"]) + from huggingface_hub import snapshot_download + + # Download with progress to temporary location first + temp_download = download_base / "_temp_download" + snapshot_download( + repo_id="zibojia/minimax-remover", + local_dir=str(temp_download), + local_dir_use_symlinks=False, + allow_patterns=["vae/*", "transformer/*", "scheduler/*"], + ignore_patterns=["*.git*", "README.md", "*.txt"] + ) + + # Move to descriptive folder names + import shutil + + vae_path = download_base / "minimax_vae" + transformer_path = download_base / "minimax_transformer" + scheduler_path = download_base / "minimax_scheduler" + + # Move downloaded folders to descriptive names + if (temp_download / "vae").exists(): + if vae_path.exists(): + shutil.rmtree(vae_path) + shutil.move(str(temp_download / "vae"), str(vae_path)) + print(f"๐Ÿ“ Moved VAE to: {vae_path}") + + if (temp_download / "transformer").exists(): + if transformer_path.exists(): + shutil.rmtree(transformer_path) + shutil.move(str(temp_download / "transformer"), str(transformer_path)) + print(f"๐Ÿ“ Moved Transformer to: {transformer_path}") + + if (temp_download / "scheduler").exists(): + if scheduler_path.exists(): + shutil.rmtree(scheduler_path) + shutil.move(str(temp_download / "scheduler"), str(scheduler_path)) + print(f"๐Ÿ“ Moved Scheduler to: {scheduler_path}") + + # Clean up temp directory + if temp_download.exists(): + shutil.rmtree(temp_download) + + if (vae_path.exists() and transformer_path.exists() and scheduler_path.exists()): + print("๐ŸŽ‰ Models downloaded successfully with descriptive names!") + print(f"๐Ÿ“ MiniMax VAE: {os.path.abspath(vae_path)}") + print(f"๐Ÿ“ MiniMax Transformer: {os.path.abspath(transformer_path)}") + print(f"๐Ÿ“ MiniMax Scheduler: {os.path.abspath(scheduler_path)}") + return str(vae_path), str(transformer_path), str(scheduler_path) + else: + raise Exception("Download completed but models not found in expected locations") + + except Exception as e: + print(f"โŒ Auto-download failed: {e}") + print("\n๐Ÿ”ง Manual download options:") + print("1. Run: python download_models.py") + print("2. Run: huggingface-cli download zibojia/minimax-remover --local-dir ./models") + print("3. Then rename folders: vae->minimax_vae, transformer->minimax_transformer, scheduler->minimax_scheduler") + print("4. See MODEL_DOWNLOAD_GUIDE.md for detailed instructions") + + # Return default paths so user can manually configure (using descriptive names) + return "models/minimax_vae/", "models/minimax_transformer/", "models/minimax_scheduler/" + + @classmethod + def INPUT_TYPES(cls): + return { + "required": { + "images": ("IMAGE",), + "masks": ("MASK",), + "num_inference_steps": ("INT", { + "default": 12, + "min": 6, + "max": 50, + "step": 1, + "display": "number" + }), + "iterations": ("INT", { + "default": 6, + "min": 1, + "max": 20, + "step": 1, + "display": "number" + }), + "seed": ("INT", { + "default": 42, + "min": 0, + "max": 0xffffffffffffffff, + "display": "number" + }), + }, + "optional": { + "auto_download": ("BOOLEAN", { + "default": True, + "tooltip": "Automatically download models if not found" + }), + "vae_path": ("STRING", { + "default": "auto", + "multiline": False, + "tooltip": "Path to VAE model directory (auto = automatic detection)" + }), + "transformer_path": ("STRING", { + "default": "auto", + "multiline": False, + "tooltip": "Path to Transformer model directory (auto = automatic detection)" + }), + "scheduler_path": ("STRING", { + "default": "auto", + "multiline": False, + "tooltip": "Path to Scheduler config directory (auto = automatic detection)" + }), + } + } + + RETURN_TYPES = ("IMAGE",) + RETURN_NAMES = ("images",) + FUNCTION = "process_video" + CATEGORY = "MiniMax-Remover" + DESCRIPTION = "BMO MiniMax video object removal with separate model paths" + + def resolve_model_path(self, model_path: str, model_type: str) -> str: + """ + Resolve model path with intelligent fallbacks + + Args: + model_path: User-provided path (can be relative or absolute) + model_type: Type of model (vae, transformer, scheduler) + + Returns: + Absolute path to model directory + """ + # Convert to Path object + path = Path(model_path) + + # If absolute path and exists, use it + if path.is_absolute() and path.exists(): + print(f"โœ… Using absolute path for {model_type}: {os.path.abspath(path)}") + return str(path) + + # Try relative to ComfyUI base + comfyui_relative = Path(comfyui_base) / model_path + if comfyui_relative.exists(): + print(f"โœ… Using ComfyUI relative path for {model_type}: {os.path.abspath(comfyui_relative)}") + return str(comfyui_relative) + + # Try in ComfyUI models directory (descriptive names first) + models_path_descriptive = self.comfyui_models_path / f"minimax_{model_type}" + if models_path_descriptive.exists(): + print(f"โœ… Using ComfyUI models path for {model_type}: {os.path.abspath(models_path_descriptive)}") + return str(models_path_descriptive) + + # Fallback to generic names for backward compatibility + models_path = self.comfyui_models_path / model_type + if models_path.exists(): + print(f"โœ… Using ComfyUI models path for {model_type}: {os.path.abspath(models_path)} (legacy)") + return str(models_path) + + # Try original path as-is (fallback) + print(f"โš ๏ธ Using fallback path for {model_type}: {path}") + return str(path) + + def load_models(self, vae_path: str, transformer_path: str, scheduler_path: str, auto_download: bool = True): + """Load MiniMax-Remover models from paths with auto-download support""" + if self.pipe is not None: + print("โ„น๏ธ Models already loaded, skipping...") + return # Already loaded + + print("๐Ÿ”„ Loading BMO MiniMax-Remover models...") + print(f"๐Ÿ—‚๏ธ ComfyUI base path: {comfyui_base}") + print(f"๐Ÿ—‚๏ธ ComfyUI models path: {self.comfyui_models_path}") + print(f"๐Ÿ“ Current working directory: {os.getcwd()}") + + # Handle auto-download and path resolution + if auto_download and (vae_path == "auto" or transformer_path == "auto" or scheduler_path == "auto"): + print("๐Ÿค– Auto-download mode enabled") + try: + auto_vae, auto_transformer, auto_scheduler = self.auto_download_models() + + # Use auto-detected paths for "auto" values + if vae_path == "auto": + vae_path = auto_vae + if transformer_path == "auto": + transformer_path = auto_transformer + if scheduler_path == "auto": + scheduler_path = auto_scheduler + + print(f"๐ŸŽฏ Auto-resolved paths:") + print(f" VAE: {os.path.abspath(vae_path)}") + print(f" Transformer: {os.path.abspath(transformer_path)}") + print(f" Scheduler: {os.path.abspath(scheduler_path)}") + + except Exception as e: + print(f"โš ๏ธ Auto-download failed, falling back to manual paths: {e}") + # Keep original paths if auto-download fails + + try: + # Resolve model paths + resolved_vae_path = self.resolve_model_path(vae_path, "vae") + resolved_transformer_path = self.resolve_model_path(transformer_path, "transformer") + resolved_scheduler_path = self.resolve_model_path(scheduler_path, "scheduler") + + print(f"๐Ÿ“ Loading VAE from: {resolved_vae_path}") + print(f"๐Ÿ“ Loading Transformer from: {resolved_transformer_path}") + print(f"๐Ÿ“ Loading Scheduler from: {resolved_scheduler_path}") + + # Check if paths exist before attempting to load + missing_paths = [] + for name, path in [("VAE", resolved_vae_path), ("Transformer", resolved_transformer_path), ("Scheduler", resolved_scheduler_path)]: + if not Path(path).exists(): + missing_paths.append(f"{name}: {path}") + + if missing_paths: + print(f"โŒ Missing model paths:") + for missing in missing_paths: + print(f" {missing}") + + if auto_download: + print("๐Ÿ”„ Attempting to re-download missing models...") + auto_vae, auto_transformer, auto_scheduler = self.auto_download_models(force_download=True) + resolved_vae_path = auto_vae + resolved_transformer_path = auto_transformer + resolved_scheduler_path = auto_scheduler + else: + raise FileNotFoundError(f"Models not found. Enable auto_download or check paths.") + + # Load models from resolved paths + AutoencoderKLWan, UniPCMultistepScheduler, Minimax_Remover_Pipeline_BMO, Transformer3DModel = lazy_import_diffusers() + + vae = AutoencoderKLWan.from_pretrained( + resolved_vae_path, + torch_dtype=torch.float16 + ) + transformer = Transformer3DModel.from_pretrained( + resolved_transformer_path, + torch_dtype=torch.float16 + ) + scheduler = UniPCMultistepScheduler.from_pretrained( + resolved_scheduler_path + ) + + # Create the BMO pipeline + self.pipe = Minimax_Remover_Pipeline_BMO( + vae=vae, + transformer=transformer, + scheduler=scheduler + ).to(self.device) + + print("โœ… BMO MiniMax-Remover models loaded successfully!") + print(f" Using device: {self.device}") + print(f" VAE: {type(vae).__name__}") + print(f" Transformer: {type(transformer).__name__}") + print(f" Scheduler: {type(scheduler).__name__}") + + except Exception as e: + print(f"โŒ Failed to load models: {e}") + print("๐Ÿ” Debug info:") + print(f" VAE path exists: {Path(resolved_vae_path).exists()}") + print(f" Transformer path exists: {Path(resolved_transformer_path).exists()}") + print(f" Scheduler path exists: {Path(resolved_scheduler_path).exists()}") + + # List available files for debugging + for name, path in [("VAE", resolved_vae_path), ("Transformer", resolved_transformer_path), ("Scheduler", resolved_scheduler_path)]: + if Path(path).exists(): + files = list(Path(path).glob("*")) + print(f" {name} directory contents: {[f.name for f in files]}") + + raise e + + def process_video( + self, + images, + masks, + num_inference_steps=12, + iterations=6, + seed=42, + auto_download=True, + vae_path="auto", + transformer_path="auto", + scheduler_path="auto" + ): + """ + Process video with BMO MiniMax-Remover with auto-download support + + Args: + images: Input video frames [B, H, W, C] in [0, 1] + masks: Binary masks [B, H, W] in [0, 1] + num_inference_steps: Number of denoising steps (official default: 12) + iterations: Mask expansion iterations (official default: 6) + seed: Random seed for reproducible results + auto_download: Automatically download models if not found + vae_path: Path to VAE model directory (auto = automatic detection) + transformer_path: Path to Transformer model directory (auto = automatic detection) + scheduler_path: Path to Scheduler config directory (auto = automatic detection) + + Returns: + Processed video frames [B, H, W, C] in [0, 1] + """ + + # Load models with auto-download support + self.load_models(vae_path, transformer_path, scheduler_path, auto_download) + + print("๐Ÿš€ Running BMO MiniMax-Remover") + print("=" * 50) + + # Convert ComfyUI tensors to the format expected by MiniMax + batch_size, height, width, channels = images.shape + num_frames = batch_size + + print(f"๐Ÿ“ Input: {images.shape} frames, {masks.shape} masks") + print(f"๐ŸŽฏ Parameters: steps={num_inference_steps}, iterations={iterations}, seed={seed}") + print(f"๐Ÿค– Auto-download: {'enabled' if auto_download else 'disabled'}") + + # Show resolved paths + if vae_path == "auto" or transformer_path == "auto" or scheduler_path == "auto": + print(f"๐Ÿ—‚๏ธ Using auto-detected model paths") + else: + print(f"๐Ÿ—‚๏ธ Model paths: VAE={vae_path}, Transformer={transformer_path}, Scheduler={scheduler_path}") + + # Prepare images: ComfyUI [B, H, W, C] -> MiniMax [F, H, W, C] -> [-1, 1] + images_np = images.detach().cpu().numpy() # [B, H, W, C] in [0, 1] + images_minimax = images_np * 2.0 - 1.0 # Convert to [-1, 1] for MiniMax + + # Prepare masks: ComfyUI [B, H, W] -> MiniMax [F, H, W, C] + if len(masks.shape) == 3: # [B, H, W] + masks_np = masks.detach().cpu().numpy() + masks_minimax = np.expand_dims(masks_np, axis=-1) # [F, H, W, 1] + else: # [B, H, W, C] + masks_minimax = masks.detach().cpu().numpy() + + print(f"๐Ÿ”„ Converted to MiniMax format:") + print(f" Images: {images_minimax.shape} [{images_minimax.min():.3f}, {images_minimax.max():.3f}]") + print(f" Masks: {masks_minimax.shape} [{masks_minimax.min():.3f}, {masks_minimax.max():.3f}]") + + # Convert to tensors + images_tensor = torch.from_numpy(images_minimax).float() + masks_tensor = torch.from_numpy(masks_minimax).float() + + # Set up generator for reproducible results + generator = torch.Generator(device=self.device).manual_seed(seed) + + # Run the BMO pipeline + print(f"๐Ÿ”ฅ Processing with BMO pipeline...") + + try: + with torch.no_grad(): + result = self.pipe( + images=images_tensor, + masks=masks_tensor, + num_frames=num_frames, + height=height, + width=width, + num_inference_steps=num_inference_steps, + iterations=iterations, + generator=generator, + output_type="np" # Get numpy output + ) + + # Extract frames + output_frames = result.frames + print(f"โœ… Pipeline completed!") + print(f"๐Ÿ“Š Output: {output_frames.shape} [{output_frames.min():.3f}, {output_frames.max():.3f}]") + + # Convert back to ComfyUI format [B, H, W, C] + if len(output_frames.shape) == 5: # [1, F, H, W, C] + output_frames = output_frames[0] # Remove batch dimension + + # Ensure we have the right number of frames + if output_frames.shape[0] != num_frames: + print(f"๐Ÿ”ง Adjusting frame count: {output_frames.shape[0]} -> {num_frames}") + if output_frames.shape[0] < num_frames: + # Pad by repeating last frame + last_frame = output_frames[-1:].repeat(num_frames - output_frames.shape[0], axis=0) + output_frames = np.concatenate([output_frames, last_frame], axis=0) + else: + # Truncate to desired number of frames + output_frames = output_frames[:num_frames] + + # Ensure correct shape and range + output_frames = np.clip(output_frames, 0.0, 1.0) + + print(f"๐Ÿ“ค Final output: {output_frames.shape} [{output_frames.min():.3f}, {output_frames.max():.3f}]") + + # Convert back to tensor for ComfyUI + result_tensor = torch.from_numpy(output_frames).float() + + return (result_tensor,) + + except Exception as e: + print(f"โŒ Processing failed: {e}") + import traceback + traceback.print_exc() + # Return original images as fallback + return (images,) + + def diagnose_inputs(self, images, masks): + """ + Diagnostic function to check input compatibility + Helps users identify potential dimension issues before processing + """ + print("๐Ÿ” DIAGNOSTIC MODE: Analyzing inputs for compatibility") + print("=" * 60) + + # Analyze images + if images is None: + print("โŒ ERROR: Images tensor is None") + return False + + print(f"๐Ÿ“Š Images analysis:") + print(f" Shape: {images.shape}") + print(f" Type: {type(images)}") + print(f" Data type: {images.dtype if hasattr(images, 'dtype') else 'N/A'}") + print(f" Range: [{images.min():.3f}, {images.max():.3f}]" if hasattr(images, 'min') else "") + + if len(images.shape) != 4: + print(f"โŒ ERROR: Expected 4D tensor [B, H, W, C], got {len(images.shape)}D") + return False + + batch_size, height, width, channels = images.shape + print(f" Frames: {batch_size}") + print(f" Resolution: {height}x{width}") + print(f" Channels: {channels}") + + # Analyze masks + if masks is None: + print("โŒ ERROR: Masks tensor is None") + return False + + print(f"\n Masks analysis:") + print(f" Shape: {masks.shape}") + print(f" Type: {type(masks)}") + print(f" Data type: {masks.dtype if hasattr(masks, 'dtype') else 'N/A'}") + print(f" Range: [{masks.min():.3f}, {masks.max():.3f}]" if hasattr(masks, 'min') else "") + + # Check dimension compatibility + print(f"\n๐Ÿ”ง Compatibility analysis:") + + # VAE spatial compatibility + vae_h = ((height + 7) // 8) * 8 + vae_w = ((width + 7) // 8) * 8 + if height == vae_h and width == vae_w: + print(f" โœ… Resolution {height}x{width} is VAE-compatible") + else: + print(f" โš ๏ธ Resolution {height}x{width} will be adjusted to {vae_h}x{vae_w} for VAE compatibility") + + # Temporal compatibility + vae_scale_factor_temporal = 4 # Default value + temporal_latent_frames = (batch_size - 1) // vae_scale_factor_temporal + 1 + print(f" ๐Ÿ“Š Temporal: {batch_size} frames โ†’ {temporal_latent_frames} latent frames") + + # Check common issues + issues = [] + + if channels != 3: + issues.append(f"Expected 3 channels (RGB), got {channels}") + + if batch_size < 1: + issues.append(f"Invalid frame count: {batch_size}") + + if len(masks.shape) not in [3, 4]: + issues.append(f"Masks should be 3D [F,H,W] or 4D [F,H,W,1], got {len(masks.shape)}D") + + if len(masks.shape) == 3 and masks.shape != (batch_size, height, width): + issues.append(f"Mask shape {masks.shape} doesn't match image frames") + + if len(masks.shape) == 4 and masks.shape != (batch_size, height, width, 1): + issues.append(f"Mask shape {masks.shape} doesn't match expected [F,H,W,1]") + + # Report results + if issues: + print(f"\nโŒ ISSUES DETECTED:") + for i, issue in enumerate(issues, 1): + print(f" {i}. {issue}") + return False + else: + print(f"\nโœ… ALL CHECKS PASSED - Inputs are compatible!") + return True + + +# ComfyUI Node Mappings +NODE_CLASS_MAPPINGS = { + "MinimaxRemoverBMO": MinimaxRemoverBMONode, +} + +NODE_DISPLAY_NAME_MAPPINGS = { + "MinimaxRemoverBMO": "MiniMax-Remover (BMO)", +} + +# Export for ComfyUI __all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"] \ No newline at end of file diff --git a/pyproject.toml b/pyproject.toml index d7c6056..59f03ca 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,81 +1,81 @@ -# BMO MiniMax-Remover for ComfyUI -# -# IMPORTANT: This package requires PyTorch with CUDA support! -# -# For CUDA installation (recommended): -# pip install -r requirements.txt -# -# For manual CUDA installation: -# pip install torch>=2.0.0,<2.8.0 torchvision>=0.15.0,<0.20.0 --index-url https://download.pytorch.org/whl/cu121 -# pip install -e . - -[build-system] -requires = ["setuptools>=61.0", "wheel"] -build-backend = "setuptools.build_meta" - -[project] -name = "comfyui-minimax-remover" -description = "High-quality video object removal for ComfyUI using MiniMax optimization. Requires PyTorch with CUDA support - install from requirements.txt for CUDA compatibility." -version = "1.0.0" -readme = "README.md" -license = {file = "LICENSE"} -authors = [ - {name = "Your Name", email = "your.email@example.com"} -] -keywords = ["comfyui", "video", "inpainting", "object-removal", "minimax"] -classifiers = [ - "Development Status :: 4 - Beta", - "Intended Audience :: Developers", - "Topic :: Multimedia :: Video", - "Topic :: Scientific/Engineering :: Artificial Intelligence", - "License :: OSI Approved :: MIT License", - "Programming Language :: Python :: 3", - "Programming Language :: Python :: 3.8", - "Programming Language :: Python :: 3.9", - "Programming Language :: Python :: 3.10", - "Programming Language :: Python :: 3.11", -] -requires-python = ">=3.8" -dependencies = [ - "diffusers>=0.21.0", - "transformers>=4.25.0", - "numpy>=1.24.0", - "opencv-python>=4.5.0,<5.0.0", - "pillow>=9.0.0", - "tqdm>=4.64.0", - "decord>=0.6.0", - "segment-anything>=1.0", - "scipy>=1.9.0", - "einops>=0.6.0", - "accelerate>=0.20.0" -] - -[project.optional-dependencies] -cuda = [ - "torch>=2.0.0,<2.8.0", - "torchvision>=0.15.0,<0.20.0" -] -cpu = [ - "torch>=2.0.0,<2.8.0", - "torchvision>=0.15.0,<0.20.0" -] - -[project.urls] -Homepage = "https://github.com/YOUR_USERNAME/ComfyUI-MiniMax-Remover" -Repository = "https://github.com/YOUR_USERNAME/ComfyUI-MiniMax-Remover" -Documentation = "https://github.com/YOUR_USERNAME/ComfyUI-MiniMax-Remover#readme" -"Bug Reports" = "https://github.com/YOUR_USERNAME/ComfyUI-MiniMax-Remover/issues" - -[tool.setuptools.packages.find] -where = ["."] -include = ["*"] -exclude = ["test*", "tests*", "__pycache__*", "*.pyc", "minimax-env*"] - -[comfyui] -# ComfyUI specific metadata -PublisherId = "your-publisher-id" -DisplayName = "MiniMax-Remover" -Description = "High-quality video object removal using MiniMax optimization. Fast 6-step inference with professional results." -Version = "1.0.0" -Icon = "" +# BMO MiniMax-Remover for ComfyUI +# +# IMPORTANT: This package requires PyTorch with CUDA support! +# +# For CUDA installation (recommended): +# pip install -r requirements.txt +# +# For manual CUDA installation: +# pip install torch>=2.0.0,<2.8.0 torchvision>=0.15.0,<0.20.0 --index-url https://download.pytorch.org/whl/cu121 +# pip install -e . + +[build-system] +requires = ["setuptools>=61.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "comfyui-minimax-remover" +description = "High-quality video object removal for ComfyUI using MiniMax optimization. Requires PyTorch with CUDA support - install from requirements.txt for CUDA compatibility." +version = "1.0.0" +readme = "README.md" +license = {file = "LICENSE"} +authors = [ + {name = "Your Name", email = "your.email@example.com"} +] +keywords = ["comfyui", "video", "inpainting", "object-removal", "minimax"] +classifiers = [ + "Development Status :: 4 - Beta", + "Intended Audience :: Developers", + "Topic :: Multimedia :: Video", + "Topic :: Scientific/Engineering :: Artificial Intelligence", + "License :: OSI Approved :: MIT License", + "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3.8", + "Programming Language :: Python :: 3.9", + "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", +] +requires-python = ">=3.8" +dependencies = [ + "diffusers>=0.21.0", + "transformers>=4.25.0", + "numpy>=1.24.0", + "opencv-python>=4.5.0,<5.0.0", + "pillow>=9.0.0", + "tqdm>=4.64.0", + "decord>=0.6.0", + "segment-anything>=1.0", + "scipy>=1.9.0", + "einops>=0.6.0", + "accelerate>=0.20.0" +] + +[project.optional-dependencies] +cuda = [ + "torch>=2.0.0,<2.8.0", + "torchvision>=0.15.0,<0.20.0" +] +cpu = [ + "torch>=2.0.0,<2.8.0", + "torchvision>=0.15.0,<0.20.0" +] + +[project.urls] +Homepage = "https://github.com/YOUR_USERNAME/ComfyUI-MiniMax-Remover" +Repository = "https://github.com/YOUR_USERNAME/ComfyUI-MiniMax-Remover" +Documentation = "https://github.com/YOUR_USERNAME/ComfyUI-MiniMax-Remover#readme" +"Bug Reports" = "https://github.com/YOUR_USERNAME/ComfyUI-MiniMax-Remover/issues" + +[tool.setuptools.packages.find] +where = ["."] +include = ["*"] +exclude = ["test*", "tests*", "__pycache__*", "*.pyc", "minimax-env*"] + +[comfyui] +# ComfyUI specific metadata +PublisherId = "your-publisher-id" +DisplayName = "MiniMax-Remover" +Description = "High-quality video object removal using MiniMax optimization. Fast 6-step inference with professional results." +Version = "1.0.0" +Icon = "" Tags = ["video", "inpainting", "object-removal", "minimax", "diffusion"] \ No newline at end of file diff --git a/requirements.txt b/requirements.txt index 4ecc807..506a793 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,23 +1,23 @@ -# BMO MiniMax-Remover Requirements -# For detailed installation instructions, please see the README.md - -# Core ML/AI Libraries -# Pinned versions for stability. -diffusers==0.33.1 -transformers==4.41.2 -accelerate==0.30.1 - -# Tensor Operations -einops==0.7.0 -scipy==1.11.4 - -# Image/Video Processing -# Note: These are common but pinning them ensures compatibility. -opencv-python==4.9.0.80 -Pillow==10.3.0 - -# Optional dependencies that often require manual installation. -# Uncomment the lines below if you wish to try installing them via pip, -# but refer to their official projects for platform-specific instructions. -# xformers +# BMO MiniMax-Remover Requirements +# For detailed installation instructions, please see the README.md + +# Core ML/AI Libraries +# Pinned versions for stability. +diffusers==0.33.1 +transformers==4.41.2 +accelerate==0.30.1 + +# Tensor Operations +einops==0.7.0 +scipy==1.11.4 + +# Image/Video Processing +# Note: These are common but pinning them ensures compatibility. +opencv-python==4.9.0.80 +Pillow==10.3.0 + +# Optional dependencies that often require manual installation. +# Uncomment the lines below if you wish to try installing them via pip, +# but refer to their official projects for platform-specific instructions. +# xformers # bitsandbytes \ No newline at end of file