Gavroche
1466b7f75c
- Restored v1.0.13 with all 9 working nodes (ROCMOptimizedCheckpointLoader, ROCMOptimizedKSampler, etc.) - Implemented comprehensive debug flag system (ROCM_NINODES_DEBUG=1) - Created complete test suite with performance and correctness tests - Added data capture infrastructure for real workflow testing - Built standalone test runner with mock data support - Created ComfyUI integration tests for Flux and WAN workflows - Added comprehensive documentation (ARCHITECTURE.md, RULES.md, TESTING_GUIDE.md) - Updated README with testing section and performance targets - All tests passing with proper error handling and data validation Performance Targets: - Flux: 78% improvement (≤110s total) - WAN: 5.6% improvement (≤93s sampling) - VAE Decode: <10s for any resolution - Checkpoint Load: <30s for Flux models Ready for production use with comprehensive testing infrastructure!
150 lines
5.2 KiB
Markdown
150 lines
5.2 KiB
Markdown
# ROCM Ninodes Architecture Documentation
|
|
|
|
## Hardware Configuration
|
|
|
|
### Primary System
|
|
- **Platform**: GMTek Evo-X2 Strix Halo
|
|
- **GPU**: AMD gfx1151 (RDNA 3.5 architecture)
|
|
- **Memory**: 128GB Unified RAM (shared between CPU and GPU)
|
|
- **OS**: Manjaro Linux (kernel 6.17.1-1-MANJARO)
|
|
- **Shell**: Zsh (/usr/bin/zsh)
|
|
|
|
## Software Stack
|
|
|
|
### Core Dependencies
|
|
- **ROCm**: 7.10.0a (latest development build)
|
|
- **PyTorch**: 2.10.0a0+rocm7.10.0a20251011 (nightly build)
|
|
- **Python**: 3.11+ (ComfyUI requirement)
|
|
- **ComfyUI**: Latest main branch with custom optimizations
|
|
|
|
### Environment Configuration
|
|
|
|
#### ComfyUI Launch Flags
|
|
```bash
|
|
--use-pytorch-cross-attention # Enable PyTorch cross-attention for better performance
|
|
--highvram # Optimize for high VRAM systems
|
|
--cache-none # Disable caching to prevent memory issues
|
|
```
|
|
|
|
#### Environment Variables
|
|
```bash
|
|
# ROCm-specific optimizations
|
|
TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 # Enable experimental AOTRITON features
|
|
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True # Enable expandable memory segments
|
|
|
|
# Optional debugging
|
|
ROCM_NINODES_DEBUG=0 # Set to 1 to enable data capture for testing
|
|
```
|
|
|
|
## GPU Architecture Specifics
|
|
|
|
### gfx1151 Optimizations
|
|
- **Precision**: fp32 preferred over fp16 for stability
|
|
- **Tile Sizes**: 768-1024 optimal for this architecture
|
|
- **TF32**: Disabled (not supported/beneficial on gfx1151)
|
|
- **Memory Modifier**: Conservative 1.5x for attention operations
|
|
- **Batch Processing**: Conservative approach to prevent OOM
|
|
|
|
### Memory Management Strategy
|
|
- **Attention Memory**: 1.5x modifier for conservative allocation
|
|
- **VAE Processing**: Tiled decoding for large images/videos
|
|
- **Video Support**: 5D tensor processing with 4D output conversion
|
|
- **Cache Management**: Regular `torch.cuda.empty_cache()` calls
|
|
|
|
## Library Support Status
|
|
|
|
### HIPBlas
|
|
- **Status**: Fully supported
|
|
- **Configuration**: Automatic detection and optimization
|
|
- **Performance**: Significant speedup for matrix operations
|
|
|
|
### FlashInfer
|
|
- **Status**: Experimental support for gfx1151
|
|
- **Usage**: Enabled in ROCm builds but may have limitations
|
|
- **Fallback**: Standard attention mechanisms when FlashInfer fails
|
|
|
|
### GPU Direct Storage
|
|
- **Status**: Limited support on current setup
|
|
- **Workaround**: Standard file I/O with memory mapping where possible
|
|
- **Future**: May improve with driver updates
|
|
|
|
## Performance Characteristics
|
|
|
|
### Flux Workflow (1024x1024)
|
|
- **Target Improvement**: 78% over baseline
|
|
- **Checkpoint Load**: <30s target
|
|
- **Sampling**: Optimized for gfx1151 architecture
|
|
- **VAE Decode**: <10s target
|
|
|
|
### WAN Workflow (320x320, 17 frames)
|
|
- **Target Improvement**: 5.6% over baseline
|
|
- **Video Processing**: 5D tensor support with 4D output
|
|
- **Memory Usage**: Conservative allocation to prevent OOM
|
|
- **Total Time**: <100s target
|
|
|
|
## Node Architecture
|
|
|
|
### Core Nodes
|
|
1. **ROCMOptimizedCheckpointLoader**: Flux-optimized checkpoint loading
|
|
2. **ROCMOptimizedKSampler**: Basic sampling with ROCm tuning
|
|
3. **ROCMOptimizedKSamplerAdvanced**: Advanced sampling with step control
|
|
4. **ROCMOptimizedVAEDecode**: VAE decode with video support
|
|
5. **ROCMOptimizedVAEDecodeTiled**: Tiled VAE decode for large images
|
|
6. **ROCMVAEPerformanceMonitor**: VAE performance metrics
|
|
7. **ROCMSamplerPerformanceMonitor**: Sampler performance metrics
|
|
8. **ROCMFluxBenchmark**: Flux workflow benchmarking
|
|
9. **ROCMMemoryOptimizer**: Memory management helper
|
|
|
|
### Data Flow
|
|
```
|
|
Input → Checkpoint Loader → KSampler → VAE Decode → Output
|
|
↓ ↓ ↓ ↓
|
|
Monitor Performance Performance Monitor
|
|
```
|
|
|
|
## Testing Infrastructure
|
|
|
|
### Debug Mode
|
|
- **Environment Variable**: `ROCM_NINODES_DEBUG=1`
|
|
- **Data Capture**: Saves input/output tensors to pickle files
|
|
- **Performance Impact**: Zero when disabled
|
|
- **Location**: `test_data/captured/`
|
|
|
|
### Test Categories
|
|
1. **Unit Tests**: Individual node functionality
|
|
2. **Performance Tests**: Timing and memory usage validation
|
|
3. **Integration Tests**: Full workflow execution
|
|
4. **Regression Tests**: Prevent performance degradation
|
|
|
|
## Constraints and Limitations
|
|
|
|
### Hardware Constraints
|
|
- **Memory**: 128GB shared RAM limits batch sizes
|
|
- **GPU**: gfx1151 specific optimizations required
|
|
- **Storage**: No GPU Direct Storage support
|
|
|
|
### Software Constraints
|
|
- **ROCm Version**: Requires 7.10.0a+ for full features
|
|
- **PyTorch**: Nightly builds required for latest optimizations
|
|
- **ComfyUI**: Custom modifications needed for optimal performance
|
|
|
|
### Performance Constraints
|
|
- **Precision**: fp32 preferred over fp16 for stability
|
|
- **Memory**: Conservative allocation to prevent OOM
|
|
- **Batch Size**: Limited by available memory
|
|
- **Video Processing**: 5D tensor handling complexity
|
|
|
|
## Future Considerations
|
|
|
|
### Planned Improvements
|
|
- **FlashInfer**: Full support when available
|
|
- **Memory Optimization**: Better memory management strategies
|
|
- **Batch Processing**: Larger batch sizes with memory improvements
|
|
- **Video Support**: Enhanced video processing capabilities
|
|
|
|
### Research Areas
|
|
- **Attention Mechanisms**: Optimize for gfx1151
|
|
- **Memory Layout**: Better tensor organization
|
|
- **Pipeline Optimization**: Reduce data movement overhead
|
|
- **Quantization**: Explore fp16 where stable
|