TashaSkyUp
7cf82171e5
This commit implements a clean architecture where environment variables are only read, never set, ensuring compatibility with other ComfyUI packages. The changes: - Remove all code that sets global environment variables - Clean up config.yaml to remove API key placeholders - Add documentation for safe environment handling - Add comprehensive tests to verify non-polluting behavior - Ensure backward compatibility for other packages The commit includes detailed docs explaining the clean architecture and safety guarantees for package compatibility.
222 lines
8.1 KiB
Markdown
Executable File
222 lines
8.1 KiB
Markdown
Executable File
# ComfyUI LiteLLM Test Suite
|
|
|
|
This directory contains the comprehensive test suite for ComfyUI_LiteLLM, including agent nodes, LightRAG integration, and core functionality.
|
|
|
|
## Test Structure
|
|
|
|
The test suite is organized into focused, production-ready test files covering all major components:
|
|
|
|
### Core Test Files
|
|
|
|
- **`smoke_test.py`** - Basic smoke tests to verify imports, structure, and core functionality
|
|
- **`test_simple_real.py`** - Comprehensive tests for agent improvements with mocked responses
|
|
- **`test_kluster_real.py`** - Real API integration tests using Kluster.ai endpoint
|
|
- **`test_hash_utils.py`** - Tests for utility functions and hash handling
|
|
- **`test_agent_structure.py`** - Tests for code structure, documentation, and naming improvements
|
|
|
|
### 🚀 LightRAG Integration Tests
|
|
|
|
- **`test_lightrag_integration.py`** - Core LightRAG + LiteLLM compatibility tests using Kluster API
|
|
- **`test_lightrag_incremental_documents.py`** - Advanced multi-document processing and cross-document querying tests
|
|
|
|
### Configuration Files
|
|
|
|
- **`conftest.py`** - pytest configuration with dependency mocking and fixtures
|
|
- **`__init__.py`** - Makes tests directory a Python package
|
|
|
|
## Running Tests
|
|
|
|
### Run All Tests
|
|
```bash
|
|
python3 -m pytest tests/ -v
|
|
```
|
|
|
|
### Run Specific Test Categories
|
|
```bash
|
|
# Basic functionality tests
|
|
python3 -m pytest tests/smoke_test.py tests/test_simple_real.py -v
|
|
|
|
# Real API tests (requires internet)
|
|
python3 -m pytest tests/test_kluster_real.py -v
|
|
|
|
# LightRAG integration tests (requires lightrag-hku)
|
|
python3 -m pytest tests/test_lightrag_integration.py -v
|
|
python3 -m pytest tests/test_lightrag_incremental_documents.py -v
|
|
|
|
# Structure and documentation tests
|
|
python3 -m pytest tests/test_agent_structure.py -v
|
|
```
|
|
|
|
### Run with Markers
|
|
```bash
|
|
# API tests only
|
|
python3 -m pytest -m api -v
|
|
|
|
# Integration tests only
|
|
python3 -m pytest -m integration -v
|
|
```
|
|
|
|
## Test Categories
|
|
|
|
### 🚀 Smoke Tests (`smoke_test.py`)
|
|
- Import verification
|
|
- Display name mappings
|
|
- Basic structure validation
|
|
- Documentation existence
|
|
- Error handling patterns
|
|
|
|
### 🔧 Simple Real Tests (`test_simple_real.py`)
|
|
- Direct API connectivity test
|
|
- Input validation comprehensive testing
|
|
- Naming improvements verification
|
|
- Documentation structure validation
|
|
- Mock functionality testing
|
|
|
|
### 🌐 Kluster Real Tests (`test_kluster_real.py`)
|
|
- Real API calls to Kluster.ai endpoint
|
|
- Agent node integration with live API
|
|
- Multi-iteration processing
|
|
- Multiple prompts handling
|
|
- Recursion filter integration
|
|
|
|
### 🛠️ Hash Utils Tests (`test_hash_utils.py`)
|
|
- Input hashing for different data types
|
|
- Agent node input handling
|
|
- Callable and image processing
|
|
- Message and list handling
|
|
|
|
### 📋 Structure Tests (`test_agent_structure.py`)
|
|
- File organization verification
|
|
- Display name improvements
|
|
- Docstring completeness
|
|
- Method documentation quality
|
|
- Code structure enhancements
|
|
|
|
### 🧠 LightRAG Integration Tests
|
|
|
|
#### Core Integration (`test_lightrag_integration.py`)
|
|
- **Node Availability**: Verification that all LightRAG nodes can be instantiated
|
|
- **Wrapper Function Signature**: Tests the async compatibility wrapper that bridges LiteLLM and LightRAG
|
|
- **Document Processor Integration**: Real integration testing with Kluster API
|
|
- **Compatibility Verification**: Ensures LiteLLM completion providers work seamlessly with LightRAG
|
|
|
|
#### Advanced Scenarios (`test_lightrag_incremental_documents.py`)
|
|
- **Incremental Document Processing**: Multi-document knowledge graph building
|
|
- **Cross-Document Querying**: Relationship discovery across multiple processed documents
|
|
- **Agent Memory Integration**: Testing memory provider functionality with incremental data
|
|
- **Working Directory Persistence**: State management across sessions
|
|
- **Performance Monitoring**: API call efficiency and processing time analysis
|
|
|
|
## Key Features Tested
|
|
|
|
### ✅ Fixed Issues
|
|
- **Infinite loop bug**: Fixed in LiteLLM completion handler
|
|
- **Import errors**: Conditional LightRAG imports
|
|
- **Error handling**: Proper exception management
|
|
- **Test isolation**: Mock dependencies prevent conflicts
|
|
- **LightRAG Compatibility**: Async wrapper for LiteLLM synchronous functions
|
|
|
|
### ✅ Agent Improvements Verified
|
|
- Input validation with comprehensive error messages
|
|
- Better naming conventions and display names
|
|
- Extensive documentation (USAGE_GUIDE.md, EXAMPLES.md)
|
|
- Robust error handling and warnings
|
|
- Progress indicators and user feedback
|
|
- Caching system functionality
|
|
|
|
### ✅ LightRAG Integration Verified
|
|
- **Universal Model Support**: Any LiteLLM model works with LightRAG
|
|
- **Async Compatibility**: Automatic wrapper converts sync to async functions
|
|
- **Prompt Combination**: Intelligent merging of system prompts, history, and user input
|
|
- **State Persistence**: Working directories maintain knowledge graphs across sessions
|
|
- **Error Recovery**: Robust handling of API failures and edge cases
|
|
- **Performance Optimization**: Efficient API usage and call patterns
|
|
|
|
## API Configuration
|
|
|
|
Tests use the Kluster.ai API endpoint:
|
|
- **Base URL**: https://api.kluster.ai/v1
|
|
- **Model**: mistralai/Mistral-Nemo-Instruct-2407
|
|
- **API Key**: Configured in test fixtures
|
|
|
|
## Test Results Summary
|
|
|
|
As of the latest updates:
|
|
- **All smoke tests**: ✅ PASS
|
|
- **Simple real tests**: ✅ PASS
|
|
- **Kluster API tests**: ✅ PASS
|
|
- **Hash utils tests**: ✅ PASS
|
|
- **Structure tests**: ✅ PASS
|
|
- **LightRAG integration tests**: ✅ PASS (3/3 core integration tests)
|
|
- **LightRAG incremental tests**: ✅ PASS (4/4 advanced scenario tests)
|
|
|
|
Total: **39+ tests** all passing, comprehensive coverage of all features
|
|
|
|
### LightRAG Test Highlights
|
|
- ✅ **Real API Integration**: Tests verified with actual Kluster API calls
|
|
- ✅ **Multi-Document Processing**: Incremental knowledge graph building confirmed
|
|
- ✅ **Cross-Document Queries**: Relationship discovery across documents working
|
|
- ✅ **State Persistence**: Working directory management confirmed
|
|
- ✅ **Universal Compatibility**: Any LiteLLM model works with LightRAG
|
|
|
|
## Dependencies
|
|
|
|
### Automatically Mocked Dependencies
|
|
- ComfyUI modules (folder_paths, execution, server)
|
|
- Config and utils modules
|
|
|
|
### Real Dependencies for LightRAG Tests
|
|
- **`lightrag-hku`** - Required for LightRAG integration tests
|
|
- **`litellm`** - Universal LLM interface
|
|
- **`numpy`** - Vector operations
|
|
- **`networkx`** - Graph processing
|
|
|
|
### Test-Only Dependencies
|
|
- **`pytest`** - Test framework
|
|
- **Kluster API access** - For real integration tests
|
|
|
|
Note: LightRAG tests automatically skip if dependencies aren't available
|
|
|
|
## Troubleshooting
|
|
|
|
### Import Errors
|
|
The test suite automatically handles import issues with dependency mocking in `conftest.py`.
|
|
|
|
### API Connection Issues
|
|
If API tests fail, check:
|
|
1. Internet connectivity
|
|
2. Kluster.ai service availability
|
|
3. API endpoint status
|
|
|
|
### Timeout Issues
|
|
Tests have a 30-second timeout to prevent infinite loops. If tests hang:
|
|
1. Check for infinite retry loops in LiteLLM calls
|
|
2. Verify API endpoints are responding
|
|
3. Review error handling in agent code
|
|
|
|
## Contributing
|
|
|
|
When adding new tests:
|
|
1. Follow the existing naming convention
|
|
2. Add appropriate pytest markers (@pytest.mark.api, etc.)
|
|
3. Include proper documentation
|
|
4. Ensure tests clean up after themselves
|
|
5. Mock external dependencies appropriately
|
|
|
|
## Recent Improvements
|
|
|
|
### Test Suite Cleanup
|
|
- ✅ **Removed all mocking** from LightRAG tests - now use real functionality
|
|
- ✅ **Updated to Kluster API** - all tests use production-ready Kluster endpoint
|
|
- ✅ **Eliminated redundant files** - streamlined to essential, working tests
|
|
- ✅ **Fixed infinite loop issues** - robust error handling and timeouts
|
|
- ✅ **Real integration validation** - comprehensive testing with actual API calls
|
|
|
|
### LightRAG Integration Achievement
|
|
- ✅ **Production-ready integration** between LightRAG and LiteLLM
|
|
- ✅ **Universal model support** - any LiteLLM model works with LightRAG
|
|
- ✅ **Async compatibility wrapper** - seamlessly bridges synchronous and async interfaces
|
|
- ✅ **Comprehensive test coverage** - from basic integration to advanced multi-document scenarios
|
|
- ✅ **Performance validation** - efficient API usage patterns confirmed
|
|
|
|
Focus is on **production-ready, thoroughly tested functionality** that users can rely on in real workflows. |