iGavroche-rocm-ninodes/TESTING.md

ROCM Ninodes Testing Framework

This document describes the comprehensive testing framework for ROCM Ninodes, including unit tests, integration tests, benchmarks, and data collection.

Overview

The testing framework is designed to:

• Validate node functionality across different configurations
• Measure performance improvements on AMD GPUs
• Collect real-world data from Flux and WAN workflows
• Optimize nodes based on test results
• Ensure reliability and prevent regressions

Directory Structure

tests/
├── unit/                    # Unit tests for individual nodes
│   ├── test_vae_decode.py
│   ├── test_ksampler.py
│   ├── test_checkpoint_loader.py
│   ├── test_vae_error_scenarios.py
│   └── test_vae_optimization.py
├── integration/             # Integration tests for workflows
│   └── test_workflows.py
├── benchmarks/              # Performance benchmark tests
│   └── test_performance.py
├── fixtures/                # Test fixtures and data
├── conftest.py             # Pytest configuration and fixtures
├── run_tests.py            # Test runner script
└── __init__.py

test_data/                   # Collected test data
├── inputs/                 # Node input data
├── outputs/                # Node output data
├── benchmarks/             # Performance metrics
├── optimization/           # Optimization data
└── debug/                  # Debug data

instrumentation.py           # Data collection instrumentation
collect_test_data.py         # Data collection script
pytest.ini                  # Pytest configuration

Test Types

Unit Tests

Unit tests validate individual node functionality:

• test_vae_decode.py: Tests for ROCMOptimizedVAEDecode and ROCMOptimizedVAEDecodeTiled
• test_ksampler.py: Tests for ROCMOptimizedKSampler and ROCMOptimizedKSamplerAdvanced
• test_checkpoint_loader.py: Tests for ROCMOptimizedCheckpointLoader
• test_vae_error_scenarios.py: Error handling and edge cases
• test_vae_optimization.py: Optimization-specific tests

Integration Tests

Integration tests validate complete workflows:

• test_workflows.py: Tests for Flux and WAN workflows
• Validates end-to-end functionality
• Tests node interactions
• Validates workflow compatibility

Benchmark Tests

Benchmark tests measure performance:

• test_performance.py: Performance benchmarks
• Measures execution time and memory usage
• Compares different configurations
• Tracks performance improvements

Running Tests

Prerequisites

Install required dependencies:

pip install pytest pytest-mock torch numpy

Run All Tests

python tests/run_tests.py

Run Specific Test Types

# Unit tests only
python tests/run_tests.py unit

# Integration tests only
python tests/run_tests.py integration

# Benchmark tests only
python tests/run_tests.py benchmark

Run Individual Test Files

# Run specific test file
pytest tests/unit/test_vae_decode.py -v

# Run with specific markers
pytest -m "benchmark" -v
pytest -m "integration" -v
pytest -m "requires_gpu" -v

Data Collection

Collect Test Data

Run the data collection script to gather test data from workflows:

python collect_test_data.py

This will:

• Instrument all nodes for data collection
• Run Flux and WAN workflows
• Collect inputs, outputs, and performance metrics
• Generate optimization data

Data Collection Features

• Automatic Instrumentation: All nodes are automatically instrumented
• Workflow Testing: Tests both Flux and WAN workflows
• Performance Profiling: Collects execution time and memory usage
• Configuration Testing: Tests different precision modes and settings
• Error Handling: Captures data even when errors occur

Test Configuration

Pytest Configuration

The pytest.ini file contains:

• Test discovery patterns
• Output formatting
• Markers for different test types
• Timeout settings
• Warning filters

Test Markers

• @pytest.mark.unit: Unit tests
• @pytest.mark.integration: Integration tests
• @pytest.mark.benchmark: Performance tests
• @pytest.mark.slow: Slow tests
• @pytest.mark.requires_gpu: Tests requiring GPU
• @pytest.mark.requires_amd: Tests requiring AMD GPU

Instrumentation

Node Instrumentation

The instrumentation.py module provides:

• Input Capture: Saves all node inputs
• Output Capture: Saves all node outputs
• Performance Metrics: Tracks execution time and memory usage
• Data Serialization: Converts tensors to serializable format
• Test Fixtures: Generates test fixtures from collected data

Data Collection Process

1. Instrument Nodes: Automatically instrument all ROCM nodes
2. Run Workflows: Execute Flux and WAN workflows
3. Capture Data: Save inputs, outputs, and performance metrics
4. Generate Fixtures: Create test fixtures from collected data
5. Optimize: Use data for node optimization

Performance Testing

Benchmark Categories

1. Resolution Tests: Different image/video resolutions
2. Precision Tests: Different precision modes (fp32, fp16, bf16)
3. Configuration Tests: Different tile sizes and settings
4. Memory Tests: Memory usage and optimization
5. Workflow Tests: Complete workflow performance

Performance Metrics

• Execution Time: Time to complete operations
• Memory Usage: Peak and average memory consumption
• Throughput: Operations per second
• Efficiency: Memory efficiency ratios
• Improvement: Performance improvements over baseline

Optimization

Data-Driven Optimization

The testing framework enables data-driven optimization:

1. Collect Data: Run workflows and collect performance data
2. Analyze Results: Identify bottlenecks and optimization opportunities
3. Implement Changes: Optimize nodes based on findings
4. Validate Improvements: Re-run tests to verify improvements
5. Track Progress: Monitor performance over time

Optimization Targets

• Memory Usage: Reduce VRAM consumption
• Execution Time: Improve processing speed
• Stability: Reduce errors and crashes
• Compatibility: Ensure cross-platform compatibility
• Scalability: Support larger images and videos

Test Reports

Generated Reports

The test runner generates comprehensive reports:

• Test Summary: Pass/fail statistics
• Performance Metrics: Execution times and memory usage
• Device Information: GPU and system details
• Recommendations: Optimization suggestions
• Error Analysis: Failed test details

Report Location

Reports are saved to test_data/test_report.json and include:

• Test results for all test types
• Performance benchmarks
• Device compatibility information
• Optimization recommendations

Continuous Integration

Automated Testing

The testing framework supports CI/CD:

# Example GitHub Actions workflow
name: ROCM Ninodes Tests
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: 3.8
      - name: Install dependencies
        run: pip install -r requirements.txt
      - name: Run tests
        run: python tests/run_tests.py

Test Coverage

• Unit Tests: 100% node coverage
• Integration Tests: Complete workflow coverage
• Benchmark Tests: Performance validation
• Error Tests: Edge case coverage

Troubleshooting

Common Issues

1. Import Errors: Ensure all dependencies are installed
2. CUDA Errors: Check GPU availability and drivers
3. Memory Errors: Reduce batch sizes or use CPU fallback
4. Timeout Errors: Increase timeout values for slow tests

Debug Mode

Run tests in debug mode for detailed output:

pytest -v -s --tb=long

Test Data Issues

If test data is missing:

1. Run python collect_test_data.py to collect data
2. Check that workflow files exist
3. Verify node instrumentation is working

Contributing

Adding New Tests

1. Create test file in appropriate directory
2. Follow naming convention: test_*.py
3. Use appropriate markers
4. Include docstrings and comments
5. Test both success and failure cases

Test Guidelines

• Isolation: Tests should be independent
• Deterministic: Tests should produce consistent results
• Fast: Unit tests should be fast
• Clear: Test names should be descriptive
• Comprehensive: Cover edge cases and error conditions

Performance Monitoring

Benchmark Tracking

The framework tracks performance over time:

• Historical performance data
• Regression detection
• Improvement measurement
• Device-specific optimization

Optimization Metrics

Key metrics for optimization:

• Speed: Execution time improvements
• Memory: VRAM usage reduction
• Stability: Error rate reduction
• Compatibility: Cross-platform support

Future Enhancements

Planned Features

1. Automated Optimization: AI-driven node optimization
2. Real-time Monitoring: Live performance tracking
3. A/B Testing: Compare different implementations
4. Regression Testing: Automatic performance regression detection
5. Cloud Testing: Test on different hardware configurations

Extensibility

The framework is designed to be extensible:

• Easy to add new test types
• Pluggable instrumentation
• Configurable test parameters
• Custom performance metrics

Support

For issues with the testing framework:

1. Check the troubleshooting section
2. Review test logs and reports
3. Create an issue with detailed information
4. Include test data and error messages

The testing framework is essential for maintaining code quality and performance in ROCM Ninodes. Regular testing ensures reliability and helps identify optimization opportunities.