diodiogod-TTS-Audio-Suite/tests/TESTING.md

TTS Audio Suite - Testing Guide

This guide covers running tests for the TTS Audio Suite custom node.

Quick Start

For a split installation, first create the ignored tests/.env.local file described under Path Configuration & Local Overrides. The wrapper cannot infer paths when ComfyUI and this repository live in separate directory trees.

cd /path/to/ComfyUI/custom_nodes/TTS-Audio-Suite

# Run all tests
python tests/run_tests.py -v

# Run only fast unit tests (no server needed)
python tests/run_tests.py unit/ -m unit -v

# Run integration tests (requires ComfyUI running)
python tests/run_tests.py integration/ -m integration -v

Always use tests/run_tests.py; it selects the configured Python, changes into the isolated test directory, and ensures the local pytest configuration wins.

---

Test Categories

Unit Tests (64 tests)

Fast tests that validate utility functions without ComfyUI.

Module	Tests	Description
test_srt_parser.py	23	SRT timestamp parsing, validation, subtitle handling
test_pause_processor.py	19	Pause tag detection, silence generation
test_audio_processing.py	22	Time conversions, silence creation, audio normalization

Integration Tests (23 tests)

Tests that run against a live ComfyUI server.

Category	Tests	Description
Node Registration	10	Verify all TTS nodes are registered
Engine Config	4	Validate engine configuration workflows
Workflow Fixtures	4	Load and validate workflow JSON files
Server Health	3	API endpoints, node counts
E2E Generation	2	Full TTS generation with audio output

---

Test Markers

# Run by marker
python tests/run_tests.py -m unit           # Fast, no server
python tests/run_tests.py -m integration    # Requires ComfyUI
python tests/run_tests.py -m slow           # Full generation tests
python tests/run_tests.py -m cosyvoice      # CosyVoice3 specific

---

Running Integration Tests

Integration tests require ComfyUI to be running:

# Terminal 1: Start ComfyUI
cd /path/to/ComfyUI
/path/to/comfyui/python main.py

# Terminal 2: Run tests
cd /path/to/ComfyUI/custom_nodes/TTS-Audio-Suite
python tests/run_tests.py integration/ -m integration -v

---

FL-MCP-Assisted Live Validation

ComfyUI_FL-MCP can drive and inspect a live ComfyUI session from an MCP-capable coding client. It complements pytest by exposing workflow state, browser screenshots, queue execution, history, errors, and output files. It is optional and is not a dependency of TTS Audio Suite.

Use it for new-engine acceptance testing after the implementation and automated tests are complete. Follow FL_MCP_VALIDATION.md for prerequisites, the standard smoke sequence, required evidence, safety boundaries, and the manual fallback.

New custom-node Python code is loaded only when ComfyUI starts. During implementation, follow the runbook's mandatory edit–restart–browser reconnect–validate loop. Keeping an old ComfyUI process open does not test newly written engine code.

REST availability alone is insufficient for canvas inspection. Before live validation, bind the MCP client to the browser's persistent FL-MCP session ID and verify one bridge session reports both frontend and MCP connections.

FL-MCP does not determine whether generated speech sounds good. A human must assess pronunciation, voice similarity, artifacts, pacing, and other subjective audio qualities.

---

Test Fixtures

Test workflow files are in tests/fixtures/workflows/:

File	Purpose
test_chatterbox_e2e.json	Full ChatterBox generation pipeline
test_cosyvoice_e2e.json	Full CosyVoice3 generation pipeline
test_chatterbox_engine.json	Engine config validation
test_cosyvoice_engine.json	Engine config validation
test_f5tts_engine.json	F5-TTS config validation
test_indextts_engine.json	IndexTTS config validation

---

Path Configuration & Local Overrides

By default, the tests attempt to locate ComfyUI and its python virtual environment using default relative paths (../../../).

If you use a custom setup (such as stability-matrix or a different venv location), you can override these paths locally without editing any tracked git files:

1. Create a file named tests/.env.local (this file is ignored by git).
2. Define the path overrides:

# Root directory of your ComfyUI installation
TTS_SUITE_TEST_COMFY_ROOT=/path/to/ComfyUI

# Path to the python executable inside the virtual environment
TTS_SUITE_TEST_VENV_PYTHON=/path/to/ComfyUI/venv/bin/python

These overrides will be automatically picked up by tests/env_config.py when you run tests.

---

For Developers

Adding Unit Tests

1. Create test file in tests/unit/test_*.py
2. Mark tests with @pytest.mark.unit
3. Import modules using importlib to avoid triggering __init__.py

import pytest
from pathlib import Path
import sys

# Add custom node root before imports
sys.path.insert(0, str(Path(__file__).parent.parent.parent))

from utils.timing.parser import SRTParser

@pytest.mark.unit
class TestMyFeature:
    def test_something(self):
        assert True

Adding Integration Tests

1. Create test in tests/integration/test_*.py
2. Mark with @pytest.mark.integration
3. Use provided fixtures: api_client, workflow_fixtures_path

@pytest.mark.integration
def test_my_workflow(self, api_client, workflow_fixtures_path):
    workflow = load_workflow(workflow_fixtures_path / "my_test.json")
    result = api_client.queue_prompt(workflow)
    assert "prompt_id" in result

Test Isolation

The test suite mocks ComfyUI modules (folder_paths, comfy, etc.) in tests/conftest.py to allow unit tests to run without a server. See docs/Dev reports/pytest_implementation_report.md for technical details.

---

Troubleshooting

Issue	Solution
ModuleNotFoundError: folder_paths	Ensure tests/conftest.py is present (it mocks ComfyUI)
Integration tests hang	Check ComfyUI is running at http://127.0.0.1:8188
E2E tests fail with model errors	Models auto-download on first run, may take time
ImportError in unit tests	Don't import from package root, use direct module paths

---

Architectural Roadblocks & Solutions

This section documents critical issues encountered when setting up pytest for ComfyUI custom nodes:

1. Pytest Package Discovery vs. ComfyUI

ComfyUI packages use relative imports in __init__.py (e.g., from .nodes import ...). When pytest runs from the project root, it tries to import the package, which fails with ImportError: attempted relative import with no known parent package.

Solution: The run_tests.py script changes directory to tests/ before running pytest (os.chdir(os.path.join(script_dir, 'tests'))). This prevents pytest from treating the root as a package.

2. Parent pytest.ini Interference

ComfyUI's root pytest.ini has pythonpath = . which adds the package root to sys.path, triggering package discovery issues.

Solution: Use a local tests/pytest.ini with pythonpath = (empty) to override parent settings.

3. Module Dependency Isolation

Node modules typically depend on ComfyUI (folder_paths, comfy, etc.). Only modules in utils/ subdirectories that are free of ComfyUI dependencies can be safely unit tested standalone.

4. Key Takeaways

• Use run_tests.py: Always run tests via the wrapper script, not directly with pytest.
• Don't modify __init__.py: Adding COMFYUI_TESTING checks to skip imports breaks node registration in the actual server.
• Isolate testable code: Keep utility functions in utils/ subdirectories separate from node definitions.