diodiogod-TTS-Audio-Suite/docs/Dev reports/pytest_implementation_report.md

Pytest Testing Framework Implementation Report

Date: 2024-12-18
Author: AI Assistant
Status: ✅ Working (60/64 tests passing)

---

Executive Summary

Implementing pytest for a ComfyUI custom node is non-trivial due to the way Python imports work with packages that have complex __init__.py files. This document summarizes the approaches tried, failures encountered, and the final working solution.

---

The Core Problem

When pytest discovers and imports test files, Python's import system also imports the parent package's __init__.py. For TTS-Audio-Suite, this __init__.py:

1. Applies compatibility patches (numba, PyTorch, transformers)
2. Imports nodes.py which imports folder_paths (ComfyUI-only module)
3. Loads all node modules dynamically
4. Registers API endpoints with ComfyUI server

Result: Running pytest tests/unit/ triggers the entire ComfyUI node loading chain, which fails because ComfyUI modules (folder_paths, comfy, server) don't exist in the test context.

---

Approaches Tried (and Why They Failed)

❌ Approach 1: Environment Variable Guard in __init__.py

Idea: Check os.environ['COMFYUI_TESTING'] at the top of __init__.py and skip loading.

# __init__.py
if os.environ.get('COMFYUI_TESTING') == '1':
    NODE_CLASS_MAPPINGS = {}
    # ... skip rest of file

Why it failed: Would require wrapping 300+ lines of code in an if/else block, causing massive indentation changes. Python doesn't have a "skip rest of file" statement.

---

❌ Approach 2: collect_ignore in conftest.py

Idea: Tell pytest to ignore __init__.py during collection.

# conftest.py
collect_ignore = ["__init__.py", "nodes.py"]

Why it failed: collect_ignore prevents pytest from collecting tests from those files, but doesn't prevent Python from importing them when test modules import utilities.

---

❌ Approach 3: Moving conftest.py to tests/ directory

Idea: If conftest.py is in tests/, pytest won't treat the parent as a package.

Why it failed: Pytest still finds pytest.ini in the parent and treats the directory structure as packages. The import chain is triggered when test files do from utils.timing.parser import ....

---

❌ Approach 4: Using --import-mode=importlib

Idea: Pytest's alternative import mode might avoid __init__.py loading.

pytest --import-mode=importlib tests/

Why it failed: Same underlying issue - Python still imports parent packages.

---

❌ Approach 5: Using importlib.util in test files

Idea: Bypass package imports entirely by loading modules directly.

# test_srt_parser.py
import importlib.util
parser_path = custom_node_root / "utils" / "timing" / "parser.py"
spec = importlib.util.spec_from_file_location("parser_module", parser_path)
parser_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(parser_module)

Why it failed: Conftest.py is still loaded first by pytest, and conftest.py's mere existence in the package triggers __init__.py import.

---

✅ Approach 6: Mock ComfyUI Modules at conftest.py Module Level (WORKING)

The Solution: Mock all ComfyUI modules before pytest imports anything else.

# tests/conftest.py - MUST be at very top
import os
import sys
from unittest.mock import MagicMock

os.environ['COMFYUI_TESTING'] = '1'
os.environ['PYTEST_CURRENT_TEST'] = 'true'

# Mock ComfyUI modules BEFORE any other imports
_MOCK_MODULES = [
    'comfy', 'comfy.model_management', 'comfy.utils',
    'nodes', 'folder_paths', 'server', 'execution', 'comfy_extras',
]

for _module_name in _MOCK_MODULES:
    if _module_name not in sys.modules:
        sys.modules[_module_name] = MagicMock()

# NOW safe to import pytest, etc.
import pytest

Why it works: When __init__.py executes import folder_paths, Python finds a MagicMock already in sys.modules and uses that instead of trying to find the real module.

---

Additional Bug Found

During testing, discovered a real bug in nodes.py:

# BEFORE (buggy)
try:
    from utils.compatibility import setup_numba_compatibility
    setup_numba_compatibility(...)
except ImportError:
    import sys  # ← Only imported in except block!
    import os   # ← Only imported in except block!
    
# Later in file...
current_dir = os.path.dirname(__file__)  # ← NameError if try succeeded!

Fix: Move import sys and import os to the top of the file, outside the try/except block.

---

Guidance for Future Developers

1. Test File Structure

TTS-Audio-Suite/
├── __init__.py           # Complex, avoid importing during tests
├── nodes.py              # Node registration  
├── pytest.ini            # Test configuration
├── tests/
│   ├── conftest.py       # CRITICAL: Mock ComfyUI here
│   ├── unit/
│   │   ├── test_srt_parser.py
│   │   └── test_audio_processing.py
│   └── integration/
│       └── test_workflows.py

2. conftest.py Template

"""Mock ComfyUI at module level BEFORE any imports"""
import os
import sys
from unittest.mock import MagicMock

# Environment flags
os.environ['COMFYUI_TESTING'] = '1'

# Mock ComfyUI modules
for mod in ['folder_paths', 'comfy', 'server', 'execution']:
    if mod not in sys.modules:
        sys.modules[mod] = MagicMock()

# Now safe for normal imports
import pytest
# ... rest of fixtures

3. Unit Test Pattern

# test_my_utility.py
import importlib.util
from pathlib import Path

custom_node_root = Path(__file__).parent.parent.parent
module_path = custom_node_root / "utils" / "my_utility.py"

spec = importlib.util.spec_from_file_location("my_utility", module_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)

# Now use module.MyClass, module.my_function, etc.

4. Running Tests

# Unit tests (fast, no server)
python -m pytest tests/unit/ -m unit -v

# Integration tests (requires ComfyUI running)
python -m pytest tests/integration/ -m integration -v

5. Key Gotchas

Issue	Solution
ModuleNotFoundError: folder_paths	Add to mock list in conftest.py
NameError: os not defined	Check all imports are at module top level
Tests pass locally but fail in CI	Ensure conftest.py loads first
__init__.py still runs	Mock must happen at conftest.py module level, not in a function

---

Conclusion

The working solution requires:

1. Mocking at conftest.py module level (not in pytest_configure hook - too late)
2. Ensuring os/sys imports are at top level in production code
3. Using importlib for utility imports in test files (optional but cleaner)

This approach allows unit tests to run without ComfyUI while integration tests can still use the real server.