Fix dtype compatibility for FP8-converted and native Float models
Resolves "mat1 and mat2 must have the same dtype" errors for Flux (BFloat16 vs Half)
and "expected scalar type Float but found Half" for video models (WAN).
Changes:
- Implement universal dtype matching: inputs now dynamically convert to match
model parameter dtype (Float32, Float16, BFloat16) rather than assuming FP16
- Remove conditional _converted_from_fp8 flag in favor of runtime dtype detection
- Add dtype synchronization in both single-device fallback and multi-device worker paths
- Preserve non-floating point tensors (timesteps, masks) from unnecessary conversion
- Ensure FP8 weights converted to FP16 on pre-Hopper GPUs properly align with inputs
This fixes compatibility for:
- Flux FP8 models on V100/3090 (Ampere) where weights become FP16 but inputs are BF16
- WAN video models and other native Float32 architectures that require full precision
- Mixed precision environments where model replicas may have different dtypes per device
This fixes compatibility with models like LTX-Video, Wan, and others that may use different FP8 encoding schemes (particularly e5m2) on mixed GPU setups.
Changes:
- Add is_float8_dtype() to detect all PyTorch FP8 variants
- Update clone_module_simple() to handle FP8 in layer reconstruction and generic fallback
- Update safe_model_clone() to post-process all FP8 variants during incremental loading
- Update move_to_device() to convert FP8 inputs at runtime during parallel inference
- Ensure consistent FP8→FP16 conversion across weights, buffers, and activation tensors
Preserves FP8 performance on Hopper/Ada (SM90+) while ensuring compatibility with Ampere and older architectures that only support FP16/FP32 computation.
Fixes: Support for LTX-Video, Wan, and other FP8 models using e5m2 or alternative FP8 formats on mixed-GPU setups
Fixed infinite recursion: Added check if hasattr(replica, '_original_forward') and replica is target_model in the worker to use the original forward method when the replica is the original model, preventing parallel_forward from calling itself recursively.
Device reference safety: Converted devices_ref and weights_ref to tuples to prevent accidental modification after assignment.
Explicit device comparison: Used replica is target_model to detect when we're using the original model vs a clone.
Consistent fallback handling: Both the single-device fallback and the OOM fallback now check for _original_forward to use the correct method.
Better logging: Added explicit print at the end showing which devices are actually being used.
This ensures that when CUDA:0 is the original device and CUDA:1 is a clone, both are used correctly without the original device causing a recursive loop.
Critical memory optimization for Parallel Anything node to prevent CUDA out-of-memory
errors on first execution and improve stability across multi-GPU setups.
Problem:
- First-run OOM errors occurred due to CUDA memory fragmentation and holding
duplicate model copies (CPU + GPU) simultaneously during cloning
- No handling for devices that failed during cloning due to insufficient VRAM
- Original model restoration caused temporary 2x VRAM usage spikes
Solution:
- Skip cloning when target device matches source device (use reference)
- Incremental state_dict loading to minimize peak memory usage
- Aggressive ComfyUI model cache unloading before cloning operations
- Progressive cleanup between device clones with synchronized CUDA operations
- Graceful degradation: skip OOM devices and redistribute workload
Technical Changes:
- Added aggressive_cleanup() helper for forced GC and CUDA cache clearing
- Modified safe_model_clone() with incremental parameter loading and
early-exit if model already on target device
- Updated setup_parallel() to use reference counting for original device
- Implemented successful_devices tracking for OOM fallback handling
- Added periodic torch.cuda.empty_cache() during large model transfers
- Enhanced error handling with device-level try/except blocks
- Fixed state dict parsing for nested modules in incremental loading
Memory optimizations:
- Unload ComfyUI models (unload_all_models) before cloning
- Delete state_dict entries immediately after transfer
- Force CPU transition only when necessary (not when reusing original)
- Synchronize CUDA devices before/after memory operations
Breaking Changes: None
Backwards Compatible: Yes
Added VRAM purge toggles to INPUT_TYPES:
purge_cache: Clears CUDA cache during cleanup (default: True)
purge_models: Aggressively unloads all models from VRAM using comfy.model_management.unload_all_models() (default: False)
Updated setup_parallel method signature to accept purge_cache and purge_models parameters
Store purge preferences on the model: target_model._parallel_purge_cache and target_model._parallel_purge_models
Enhanced cleanup_parallel_model function to:
Read the purge flags from the model attributes
Conditionally run comfy.model_management.unload_all_models() if purge_models is True
Conditionally clear CUDA cache and soft empty cache based on purge_cache setting
Multi-GPU cache clearing (iterates through all CUDA devices)
The purge settings persist with the model and are applied when the parallel processing resources are cleaned up (either when the node is re-run with new settings or when the model is garbage collected).
Added VRAM purge toggles to INPUT_TYPES:
purge_cache: Clears CUDA cache during cleanup (default: True)
purge_models: Aggressively unloads all models from VRAM using comfy.model_management.unload_all_models() (default: False)
Updated setup_parallel method signature to accept purge_cache and purge_models parameters
Store purge preferences on the model: target_model._parallel_purge_cache and target_model._parallel_purge_models
Enhanced cleanup_parallel_model function to:
Read the purge flags from the model attributes
Conditionally run comfy.model_management.unload_all_models() if purge_models is True
Conditionally clear CUDA cache and soft empty cache based on purge_cache setting
Multi-GPU cache clearing (iterates through all CUDA devices)
The purge settings persist with the model and are applied when the parallel processing resources are cleaned up (either when the node is re-run with new settings or when the model is garbage collected).
Fixed Memory Leaks: Added explicit cleanup of partial replicas if cloning fails mid-process
Better Error Handling: Exception handling in worker() now properly returns exceptions rather than raising, allowing cleanup of other threads
Thread Safety: Removed unused imports and ensured proper synchronization points for CUDA/XPU devices
Robust Splitting: Fixed split_batch to handle mixed lists of tensors and non-tensors correctly
Cleanup Safety: cleanup_parallel_model now handles cases where the model was partially initialized
VRAM Management: Added soft_empty_cache calls between cloning operations to prevent OOM when cloning to multiple devices
Management: Added auto balance to improve performance
Attribute Handling: Improved clone_module_simple to better handle missing attributes in layer reconstruction
Config Extraction: Added serialization check to extract_model_config to prevent storing non-serializable objects
Critical Bug Fixes:
Memory Leak: weakref.finalize now uses weakref.ref(target_model) instead of direct reference
Race Condition: Worker threads now return exceptions instead of raising them, allowing proper aggregation and cleanup
Batch Validation: get_batch_size() now validates consistent batch dimensions across tensor lists
Double Device Transfer: Removed .to(device) from split_kwargs(), now only happens in worker()
Attribute Handling: Cleaner separation of in_features vs in_channels in layer cloning
Improvements Added:
CUDA Streams: Each GPU now uses its own CUDA stream for true parallel execution
Auto VRAM Balance: New auto_vram_balance parameter adjusts splits based on available VRAM (70% user preference / 30% availability)
Gradient Checkpointing: Automatically disabled on replicas to save VRAM
Accelerate Hooks: Clears _hf_hook and hooks attributes for compatibility with accelerate offloading
Enhanced Cache Clearing: Added rope_cache and freqs_cis_cache to FLUX cleanup
Thread Safety: Proper executor context management and result validation
Performance Optimizations:
Non-blocking transfers where safe
Synchronization only on CUDA/XPU devices
Batch size auto-adjustment prevents OOM on uneven GPU setups
Fixed "'list' object has no attribute 'shape'" error when using
LTX-2 Video and other diffusion models that return tuples/lists
instead of single tensors.
Key changes:
- Added tensor container helpers: get_batch_size(), split_batch(),
move_to_device(), concatenate_results()
- parallel_forward now handles list/tuple inputs/outputs (common in
video models with temporal conditioning)
- Enhanced split_kwargs() to split list-of-tensors kwargs
(frame-wise embeddings, temporal IDs)
- Results are now properly concatenated whether model returns
Tensor, Tuple[Tensor, ...], or List[Tensor]
Tested working with: FLUX, SD3.5, LTX-2 Video, HunyuanVideo
VRAM: Maintains existing N× model replication strategy
Breaking: None (backward compatible with single-tensor outputs)
FearL0rd