ltdrdata-ComfyUI-Impact-Pack/docs/WILDCARD_TESTING_GUIDE.md

Wildcard System Testing Guide

Complete testing guide for the ComfyUI Impact Pack wildcard system.

---

📋 Table of Contents

1. Test Overview
2. Test Categories
3. Quick Start
4. Test Execution
5. Test Results
6. Troubleshooting

---

Test Overview

Test Structure

tests/
├── Unit Tests (Python)
│   ├── test_wildcard_lazy_loading.py      # LazyWildcardLoader tests
│   ├── test_progressive_loading.py        # Progressive loading tests
│   ├── test_wildcard_final.py             # Final validation
│   └── test_lazy_load_verification.py     # Lazy load verification
│
├── Integration Tests (Shell + API)
│   ├── test_lazy_load_api.sh              # Full vs on-demand comparison
│   ├── test_progressive_ondemand.sh       # Progressive loading verification
│   ├── test_sequential_loading.sh         # Transitive wildcard tests
│   ├── test_wildcard_consistency.sh       # Consistency validation
│   ├── test_wildcard_features.sh          # Feature tests
│   └── test_versatile_prompts.sh          # Versatile prompt tests
│
├── Utility Scripts
│   └── find_transitive_wildcards.sh       # Find transitive chains
│
└── Documentation
    ├── README_LAZY_LOAD_TEST.md           # Lazy loading docs
    ├── SEQUENTIAL_LOADING_TESTS.md        # Sequential loading docs
    ├── README_PROGRESSIVE_ONDEMAND.md     # Progressive loading docs
    └── VERSATILE_PROMPTS.md               # Versatile prompts docs

Test Requirements

All Tests Require:

• ComfyUI installed and configured
• Impact Pack installed
• Python 3.8+
• Bash shell

Integration Tests Require:

• ComfyUI server (port 8188 or custom)
• ~15-90 seconds server startup time
• API access (curl)

---

Test Categories

1. Progressive Loading Tests ⭐ NEW

Purpose: Verify that wildcards are loaded progressively as they are accessed.

Tests:

• test_progressive_ondemand.sh - API integration test
• test_progressive_loading.py - Unit tests

What's Tested:

• ✅ Early termination size calculation
• ✅ Metadata-only scanning
• ✅ Progressive wildcard loading
• ✅ /wildcards/list/loaded API endpoint
• ✅ Memory growth tracking

Expected Results:

Initial:         /list/loaded → 0
After __flower__: /list/loaded → 1
After __dragon__: /list/loaded → 2-3 (transitive)
After __colors__: /list/loaded → 3-4

Run:

# Integration test (requires server)
bash tests/test_progressive_ondemand.sh

# Unit test (standalone, may fail without ComfyUI env)
python3 tests/test_progressive_loading.py

Documentation: tests/README_PROGRESSIVE_ONDEMAND.md

---

2. Lazy Loading Tests

Purpose: Verify on-demand loading produces identical results to full cache mode.

Tests:

• test_lazy_load_api.sh - Full automation
• test_wildcard_lazy_loading.py - LazyWildcardLoader class
• test_lazy_load_verification.py - Verification tests

What's Tested:

• ✅ LazyWildcardLoader functionality
• ✅ Full cache vs on-demand consistency
• ✅ Automatic mode detection
• ✅ Cache size limits

Expected Results:

• All tests: Full cache results == On-demand results
• LazyWildcardLoader: Loads data only on first access
• Mode detection: Activates on-demand when size > limit

Run:

# Full automation (requires server, ~3 minutes)
bash tests/test_lazy_load_api.sh

# Unit tests (requires ComfyUI env)
python3 tests/test_wildcard_lazy_loading.py

Documentation: tests/README_LAZY_LOAD_TEST.md

---

3. Sequential/Transitive Loading Tests

Purpose: Verify transitive wildcards expand correctly across multiple stages.

Tests:

• test_sequential_loading.sh - Sequential expansion tests
• find_transitive_wildcards.sh - Transitive chain discovery

What's Tested:

• ✅ Depth 1-3 transitive expansion
• ✅ Mixed transitive scenarios
• ✅ Complex sequential scenarios
• ✅ Edge cases
• ✅ On-demand mode consistency

Expected Results:

Depth 1: __samples/flower__ → rose
Depth 2: __dragon__ → __dragon/warrior__
Depth 3: __adnd__ → __dragon__ → __dragon_spirit__ → content

Run:

# Full test suite (requires server, ~5 minutes)
bash tests/test_sequential_loading.sh

# Find transitive chains
bash tests/find_transitive_wildcards.sh

Documentation: tests/SEQUENTIAL_LOADING_TESTS.md

---

4. Wildcard Feature Tests

Purpose: Test all wildcard features and syntax.

Tests:

• test_wildcard_features.sh - Core features
• test_versatile_prompts.sh - Versatile prompt syntax
• test_wildcard_consistency.sh - Consistency validation
• test_wildcard_final.py - Final validation

What's Tested:

• ✅ Dynamic prompts: {option1|option2|option3}
• ✅ Weighted selection: {3::option1|1::option2}
• ✅ Multi-select: {2$$, $$option1|option2|option3}
• ✅ Quantifiers: 3#__wildcard__
• ✅ Wildcards: __wildcard_name__
• ✅ Nested syntax
• ✅ YAML wildcards
• ✅ Transitive wildcards

Expected Results:

• All syntax variations work correctly
• Deterministic results with same seed
• Proper probability distribution

Run:

# Feature tests
bash tests/test_wildcard_features.sh

# Versatile prompts (comprehensive)
bash tests/test_versatile_prompts.sh

# Consistency validation
bash tests/test_wildcard_consistency.sh

# Final validation
python3 tests/test_wildcard_final.py

Documentation: tests/VERSATILE_PROMPTS.md

---

Quick Start

Run All Tests (Automated)

cd /path/to/ComfyUI/custom_nodes/comfyui-impact-pack/tests

# Run test suite
for test in test_*.sh; do
    echo "=========================================="
    echo "Running: $test"
    echo "=========================================="
    bash "$test"
    echo ""
done

Run Specific Test Category

Progressive Loading:

bash tests/test_progressive_ondemand.sh

Lazy Loading:

bash tests/test_lazy_load_api.sh

Sequential Loading:

bash tests/test_sequential_loading.sh

Features:

bash tests/test_versatile_prompts.sh

---

Test Execution

Prerequisites

1. Install ComfyUI:

cd /path/to/ComfyUI
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

2. Install Impact Pack:

cd custom_nodes
git clone <impact-pack-repo>
cd comfyui-impact-pack
pip install -r requirements.txt

3. Configure Wildcards:

# Edit impact-pack.ini
[default]
wildcard_cache_limit_mb = 50  # Default
custom_wildcards = /path/to/custom/wildcards

Test Environment

Ports Used:

• 8188 - Default ComfyUI
• 8190 - Full cache mode tests
• 8191 - On-demand mode tests
• 8193 - Sequential tests
• 8195 - Progressive tests

Log Files:

• /tmp/comfyui_full_cache.log
• /tmp/comfyui_on_demand.log
• /tmp/sequential_test.log
• /tmp/progressive_test.log

Manual Test Execution

1. Start Server:

cd /path/to/ComfyUI
bash run.sh --listen 127.0.0.1 --port 8188

2. Wait for Startup:

# Check server is ready
curl -s http://127.0.0.1:8188/ > /dev/null && echo "Ready"

3. Run API Tests:

# Get wildcard list
curl http://127.0.0.1:8188/impact/wildcards/list

# Get loaded wildcards (progressive)
curl http://127.0.0.1:8188/impact/wildcards/list/loaded

# Process wildcard
curl -X POST http://127.0.0.1:8188/impact/wildcards \
  -H "Content-Type: application/json" \
  -d '{"text": "__samples/flower__", "seed": 42}'

---

Test Results

Success Criteria

Progressive Loading:

• ✅ /list/loaded starts at 0
• ✅ /list/loaded increases after each unique wildcard access
• ✅ /list/loaded unchanged on cache hits
• ✅ Transitive wildcards load multiple entries

Lazy Loading:

• ✅ Full cache == On-demand results (all tests)
• ✅ Mode detection correct based on size
• ✅ LazyWildcardLoader works correctly

Sequential Loading:

• ✅ All depths (1-3) expand correctly
• ✅ Complex scenarios work
• ✅ On-demand mode matches full cache

Features:

• ✅ All syntax variations work
• ✅ Deterministic with same seed
• ✅ Proper probability distribution

Expected Test Times

Test	Duration	Server Required
test_progressive_ondemand.sh	~2 min	Yes (port 8195)
test_lazy_load_api.sh	~3 min	Yes (ports 8190-8191)
test_sequential_loading.sh	~5 min	Yes (port 8193)
test_versatile_prompts.sh	~2 min	Yes (port 8188)
test_wildcard_consistency.sh	~1 min	Yes (port 8188)
Python unit tests	< 1 sec	No (standalone)

Sample Output

Progressive Loading Success:

========================================
Progressive Loading Verification
========================================

Step 1: Initial state (before any wildcard access)
  On-demand mode: True
  Total available wildcards: 1000
  Loaded wildcards: 0

Step 2: Access first wildcard (__samples/flower__)
  Result: rose
  Loaded wildcards: 1
✓ PASS: Wildcard count increased

Step 3: Access second wildcard (__dragon__)
  Result: ancient dragon
  Loaded wildcards: 3
✓ PASS: Wildcard count increased progressively

🎉 ALL TESTS PASSED
Progressive on-demand loading verified successfully!

---

Troubleshooting

Common Issues

1. Server Fails to Start

Symptoms:

✗ Server failed to start

Solutions:

# Check if port is in use
lsof -i :8188

# Kill existing process
pkill -f "python.*main.py"

# Increase startup wait time in test
sleep 15  # → sleep 30

2. Tests Timeout

Symptoms:

curl: (7) Failed to connect

Solutions:

# Check server logs
cat /tmp/progressive_test.log | grep -i "error\|wildcard"

# Manually start server and verify
cd /path/to/ComfyUI
bash run.sh --listen 127.0.0.1 --port 8188

3. Module Not Found (Python Tests)

Symptoms:

ModuleNotFoundError: No module named 'modules'

Solutions:

# Python tests require ComfyUI environment
cd /path/to/ComfyUI
source venv/bin/activate  # If using venv

# Or run from ComfyUI directory
cd /path/to/ComfyUI
python3 custom_nodes/comfyui-impact-pack/tests/test_progressive_loading.py

4. On-Demand Mode Not Activating

Symptoms:

Using full cache mode.

Check:

# Verify cache limit
grep wildcard_cache_limit_mb impact-pack.ini

# Check actual wildcard size
du -sh wildcards/

Solution:

# Lower cache limit to force on-demand
cat > impact-pack.ini << EOF
[default]
wildcard_cache_limit_mb = 0.5
EOF

5. Results Don't Match Between Modes

Symptoms:

✗ Results DIFFER

Debug:

# Save results for comparison
curl -X POST http://127.0.0.1:8190/impact/wildcards \
  -d '{"text": "__flower__", "seed": 42}' > full_cache.json

curl -X POST http://127.0.0.1:8191/impact/wildcards \
  -d '{"text": "__flower__", "seed": 42}' > on_demand.json

# Compare
diff full_cache.json on_demand.json

Performance Issues

Slow Startup

Expected: 15-90 seconds depending on wildcard size

If slower:

• Check disk I/O (SSD recommended)
• Verify wildcard size (use on-demand for >50MB)
• Check system resources

High Memory Usage

Expected:

• Full cache: Proportional to wildcard data size
• On-demand: < 100MB + loaded wildcards

If higher:

• Check /list/loaded to see what's loaded
• Verify on-demand mode is active
• Look for memory leaks in logs

---

Test Coverage

Feature Coverage Matrix

Feature	Unit Test	Integration Test	Status
LazyWildcardLoader	✅ test_wildcard_lazy_loading.py	✅ test_lazy_load_api.sh	Complete
Progressive Loading	✅ test_progressive_loading.py	✅ test_progressive_ondemand.sh	Complete
Early Termination	✅ test_progressive_loading.py	✅ All integration tests	Complete
Metadata Scan	✅ test_progressive_loading.py	✅ test_progressive_ondemand.sh	Complete
Transitive Wildcards	❌ N/A	✅ test_sequential_loading.sh	Complete
Dynamic Prompts	❌ N/A	✅ test_versatile_prompts.sh	Complete
YAML Wildcards	❌ N/A	✅ test_versatile_prompts.sh	Complete
API Endpoints	❌ N/A	✅ All integration tests	Complete

Test Statistics

• Total Tests: 11 files (4 Python, 7 Shell)
• Test Cases: 100+ individual test scenarios
• Coverage: ~95% of wildcard features
• Execution Time: ~15 minutes (all tests)

---

Continuous Integration

Automated Test Run

#!/bin/bash
# ci_test.sh - Run all wildcard tests

set -e

echo "Starting Wildcard Test Suite..."

# 1. Quick validation tests
echo "1. Quick Validation..."
python3 tests/test_wildcard_lazy_loading.py || echo "Warning: Needs ComfyUI env"

# 2. Progressive loading (new)
echo "2. Progressive Loading..."
bash tests/test_progressive_ondemand.sh

# 3. Lazy loading consistency
echo "3. Lazy Loading Consistency..."
bash tests/test_lazy_load_api.sh

# 4. Sequential/transitive
echo "4. Sequential Loading..."
bash tests/test_sequential_loading.sh

# 5. Feature tests
echo "5. Feature Tests..."
bash tests/test_versatile_prompts.sh

echo "✅ All tests completed!"

Test Report Generation

Tests generate logs in /tmp/:

• progressive_test.log
• comfyui_full_cache.log
• comfyui_on_demand.log
• sequential_test.log

Extract Results:

# Count passed tests
grep -c "✓ PASS" /tmp/progressive_test.log

# Find failures
grep "✗ FAIL\|ERROR" /tmp/*.log

# Summary
cat /tmp/progressive_test.log | grep -E "PASSED|FAILED"

---

Contributing

Adding New Tests

1. Create test file: tests/test_new_feature.sh or .py
2. Follow naming convention: test_<feature>_<type>.{sh|py}
3. Add documentation: Update this guide
4. Test locally: Verify test works
5. Update CI: Add to automated test suite

Test Template

Shell Test:

#!/bin/bash
# Test: Feature Name
# Purpose: What this test verifies

set -e

PORT=8XXX
CONFIG_FILE="impact-pack.ini"

# Setup
echo "Setting up test..."
cat > "$CONFIG_FILE" << EOF
[default]
wildcard_cache_limit_mb = 50
EOF

# Start server
bash run.sh --port $PORT &
sleep 15

# Test
echo "Running test..."
RESULT=$(curl -s http://127.0.0.1:$PORT/impact/wildcards/list)

# Validate
if [ "$RESULT" = "expected" ]; then
    echo "✅ PASS"
    exit 0
else
    echo "❌ FAIL"
    exit 1
fi

---

References

• Progressive Loading: tests/README_PROGRESSIVE_ONDEMAND.md
• Lazy Loading: tests/README_LAZY_LOAD_TEST.md
• Sequential Loading: tests/SEQUENTIAL_LOADING_TESTS.md
• Versatile Prompts: tests/VERSATILE_PROMPTS.md
• Main Documentation: docs/WILDCARD_SYSTEM_OVERVIEW.md