The five review findings of 2026-10-23 for phase P5:
1. The P5.5 fallback branch (a checkpoint without flownet.* keys) died with
a bare RuntimeError at the strict load: the arch always owns the flownet
(keep_arch.py:882), so the state dict of the net always carries the
flownet.* keys - and the GMFlow weights the arch had just loaded were
discarded. The flownet tensors of the built net (the trained ones - the
arch was built with flownet_path) are now merged into the checkpoint
state dict before load_state_dict(..., strict=True). The load stays
strict (an unexpected key still raises) and the trained flow weights
survive. The stub arches of the tests behave like nn.Module now - a
no-op recorder had hidden the defect.
2. P5.4's acceptance is no longer unmet: the GPU acceptance run was
executed (real weights, real ComfyUI modules, RTX 4070 Ti SUPER) and is
green (5 passed, exit code 0; the run log is in the audit doc section 5).
KEEP_MODEL_DIR now really decides where the weights come from - it is the
models root of the run (folder_paths.models_dir), the harness fixture
no longer isolates the gpu_acceptance tier, and the run needs
KEEP_NO_COMFY_MOCKS=1 so the real ComfyUI modules are in charge.
3. The "the flow weights are not random" assertion of the acceptance test
could not distinguish random weights from trained ones. It now compares
the flownet tensors with the tensors of their source (the KEEP checkpoint
or the GMFlow file) bit for bit.
4. The superseded download helper load_file_from_url_comfy is gone
(src/keep_paths.py only resolves paths; the download lives in
src/keep_download.py since P5.2). The stale docstrings, the stale
comment in the vendored facelib and the two tests that called the dead
helper are fixed.
5. The risk list mentioned a pending deletion of assets/keep-preview.png
that does not exist - the asset is committed and present (93279 bytes).
The rule in the implementation workflow now states the truth.
Checks: the gate (pytest -q -m "unit or mock_integration") is green (234
passed, 1 skipped, 1 xfailed); the opt-in acceptance run is green (5 passed);
the acceptance tier still skips cleanly without its environment variables.
P5.0-P5.5 are implemented and committed. This commit adds the phase audit (docs/audit/2026-10-23-p5-weights.md, deviations D1-D8), the working-state review (docs/2026-10-23-p5-weights-progress.md) and marks the P5 row in the plan's progress table as done.
Two review findings of the P4 phase, both fixed at their root:
1. (medium) Structure guard (g) did not scan the node layer. Its file set
came from the import closure of tools/scan_compat.py (the three src/
entry points) plus src/** - the node layer (nodes/**) and the root
__init__.py were invisible, although guard (h) of the same phase scans
them. A torch.cuda.empty_cache() added to the shipped node code passed
the guard silently.
tests/test_structure.py: new PACKAGE_OWN_SOURCE_ROOTS (the pack's own
code without the vendored tree), _package_files() takes the trees to
walk (the default is unchanged, so guards (d) and (h) are untouched),
and guard (g) now scans the own code (13 files) plus the vendored
modules/deps closure (139 files - the vendored part stays
closure-scoped as the plan and D8 require). Two new self-tests cover
the node layer and the entry point.
Proof (old guard vs new guard on the same injected copy):
OLD guard (HEAD) -> []
NEW guard (fixed) -> ['__init__.py: 1 occurrence(s) of
torch.cuda.empty_cache() (allowlisted: 0)', 'nodes/
keep_process_image_sequence.py: 1 occurrence(s) of
torch.cuda.empty_cache() (allowlisted: 0)']
2. (low) The P4.5 "CPU-mode variant" residency test was vacuous: the
conftest mock already answers torch.device("cpu") for
get_torch_device(), so the monkeypatch was a no-op and the variant
exercised the default path.
tests/test_keep_model_loader.py: the test now runs a GPU leg and a CPU
leg - both must produce the same residency calls (load_models_gpu,
unload_all_models, soft_empty_cache, once each) and the patched device
must really reach the pack.
Proof (mutation probes against the production code):
mutation A (the pack hardcodes the cpu device and ignores ComfyUI):
the old test (from HEAD) passes (10 passed), the new test fails
(assert device(type='cpu') == device(type='cuda')).
mutation B (the pack skips load_models_gpu on a cpu device): the new
test fails (the two legs no longer match).
Docs: the plan's P4 progress row, the guard table of the P4 audit and the
P4 working state are current; the round is recorded in
docs/2026-10-23-p4-review-findings.md.
Verification (run in this round):
python_embeded/python.exe -m pytest -q -m "unit or mock_integration"
-> 189 passed, 1 skipped, 5 deselected, 2 xfailed
(baseline before the round: 187 passed)
COMFYUI_ROOT=... python_embeded/python.exe -m pytest -q -m headless
-> 5 passed, 192 deselected
ruff check . -> All checks passed!
Five review findings of the P3 phase, each fixed at its root:
1. (medium) The P3.7 metadata test imported `tomllib` at module level: the
**3.10 CI job** (the declared `requires-python >=3.10` minimum) died
during collection, so the registry-metadata assertions never ran on 3.10.
The parser import is now guarded (`tomllib` -> `tomli`, the package the
3.10 job installs); without any parser the module *skips* instead of
dying. `pyproject.toml` gained the test extra
`tomli>=2.0; python_version < '3.11'`.
Proof: a 3.10 simulation (the stdlib `tomllib` refused for the test
module, `tomli` on sys.path) -> 6 passed; the control without `tomli`
-> 6 skipped (no collection error).
2. (medium) The P3.5 "primary-root-first multi-root" resolution was not
wired into the production load path: `resolve_model_file` (the only
`get_full_path_or_raise` caller) was used by tests only, so a weight in
a secondary model root was downloaded into the primary root again.
`src/keep_paths.py` now has `find_model_file()` (the multi-root lookup
through `folder_paths.get_full_path_or_raise`) and `resolve_model_root()`
(the root the weights really live in); `load_file_from_url_comfy()` - the
production path of `src/keep_model_loader.py:107` - uses the lookup, and
the facelib root follows the weights (the facelib helper builds its
paths from a single root). New tests prove that the production path
uses a weight of an extra model root and that the primary root wins.
The `models_dir` grep guard is no longer a blanket exemption: only the
registration in `src/keep_paths.py` may use the attribute.
3. (low) `[tool.comfy].Icon` was an empty string and the test only checked
that the key exists. The preview asset **is** committed
(`git cat-file -s HEAD:assets/keep-preview.png` -> 93279 bytes); only the
working copy was missing (an uncommitted local deletion, restored from
HEAD - no asset change is in this commit). The key now points at
`assets/keep-preview.png` and the test asserts that the value is not
empty and names a file that is really in the repository.
4. (low) `tests/test_package_entrypoint.py` hard-coded the V1 state and
would have gone red (not xfail) when P4.4 flips the root `__init__.py`
to `comfy_entrypoint`. The assertions now follow the entry point that is
really there: exactly **one** mapping style (V1 XOR V3). The mocked
smoke test of `tests/test_package_import.py` got the same treatment; the
headless variant is P4.4's job (its test list names it).
5. (low) The plan's §3 rule ("`comfy.*` imports inside `src/` are lazy
(function-level) so the pack imports without a ComfyUI server") was
violated and unguarded. The `comfy` imports of `src/keep_model_loader.py`
and `src/keep_processor.py` are function-level now; the vendored facelib
helper (which imports `comfy` itself) is imported lazily through a module
`__getattr__` (PEP 562), so `loader.FaceRestoreHelper` still resolves to
the vendored class. Structure **guard (f)** (`tests/test_structure.py`)
forbids a module-level `comfy`/`comfy_api` import in `src/**` (with the
self-test the guards file demands), and the new
`tests/test_lazy_imports.py` proves at runtime that the engines import
while every `comfy` import is blocked.
Gate after the commit:
python.exe -m pytest -q tests -m "unit or mock_integration"
-> 153 passed, 1 skipped, 5 deselected, 3 xfailed
COMFYUI_ROOT=C:/_Dev/ComfyUI_dev/ComfyUI-source/ComfyUI python.exe
-m pytest -q tests -m "unit or mock_integration or headless"
-> 158 passed, 1 skipped, 3 xfailed
Details: docs/2026-10-23-p3-review-findings.md
Section 9 of the plan: the P3.1-P3.7 row is now marked done with the per-step
summary. The phase review (docs/2026-10-23-comfyui-keep-p3-phase.md) carries the final
state, the gate outcomes and the remaining risks. The nine deviations of the phase are
recorded in docs/audit/2026-10-23-p3-restructure.md.
Five review findings of the P2 phase, each fixed at its root:
1. (medium) The P2.6 guard could not see an undeclared third-party import:
the closure's imports were filtered through a hard-coded 10-name map, so
every module outside it was dropped silently (a module-level `import av`
in a copy of the repo still gave "1 passed"). The map is now a *rename*
map (cv2 -> opencv-python, PIL -> Pillow, yaml -> pyyaml); every other
module is mapped to its own name, so an unknown module shows up as an
undeclared dependency. The guard is split by import time: every
third-party module imported at import time must be a declared
distribution, every lazy (function-level) one must be allowlisted with a
reason (dlib, gdown, mc, wandb - all unreachable from the node).
2. (low) The P2.1 (c) progression test was vacuous: it returned before its
assertions and probed a second, top-level copy of the vendored tree
(`sys.path.insert(modules/deps)`). It now imports the packaged
architecture, asserts that FeedForward / AdaLayerNorm come from the
vendored _diffusers_compat and that no top-level `basicsr` shadows an
installed one.
3. (low) Two swallow-and-stub fallbacks remained in the vendored tree
(facelib/detection, facelib/parsing - their stubs called an unimported
`urlparse` and would have raised NameError). Both now import the vendored
symbol directly; a guard forbids a fallback stub in the vendored tree.
4. (low) `basicsr/utils/file_client.py` still appended to `sys.path` (the
optional `sys_path` of MemcachedBackend), which the plan's §3 rule
forbids anywhere in the package. The mutation and the parameter are gone
and guard (d) now covers the whole package (the vendored tree included).
5. (low) The unreachable body of `KEEPFaceProcessor.process_video` still
called the deleted VideoReader / VideoWriter. The 141 dead lines are
removed; an AST guard keeps the references out.
Found while fixing the findings: the CI job did not install `requests`,
which the vendored `basicsr/utils/download_util.py` imports at module level
(P2.6 added it to requirements.txt but not to the workflow) - a clean CI
runner died with "No module named 'requests'". The install list is now
completed and a test derives it from requirements.txt.
Gate: pytest -m "unit or mock_integration" -> 109 passed, 1 skipped,
4 xfailed; plan's local gate (+ headless) -> 109 passed, 3 skipped,
4 xfailed; tools/scan_compat.py -> finding_count 0 (exit 0); ruff check . ->
All checks passed. A mutation check re-introduces six defects into a copy of
the repository: the matching guard fails in 6/6 cases.
Seven review findings of the P1 phase, each fixed at its root:
1. (high) The CI install list omitted PyYAML, but the vendored tree imports
yaml eagerly (modules/deps/basicsr/utils/options.py:5), so the CI job
failed on its first run. pyyaml is now installed and asserted by
tests/test_ci_workflow.py.
2. (medium) tests/test_ci_workflow.py skipped the whole module without
PyYAML. The module never skips now: a minimal indentation parser is the
fallback, and it is cross-checked against the PyYAML parse.
3. (medium) Guard (e)'s self-test asserted that the tree still contains
breakage (it would break exactly at P2.1). It now proves the guard
instead; P2.1's work list is the scanner output and
test_guard_e_flip_owner_covers_every_current_finding keeps plan and tree
in sync.
4. (medium) The mock-fidelity tripwire skipped the comfy_api.latest
symbols. Every mocked symbol is now checked against the real source
(re-export / definition / instance attribute), including io.ComfyNode,
ComfyExtension, ComfyAPI and ComfyAPISync, plus the reverse check that
the mock invents nothing.
5. (low) The phase review listed committed work as outstanding and
misstated the ruff ignore list; both are corrected.
6. (low) The reported whole-suite count was impossible; the phase review
now carries the measured numbers.
7. (low) The root conftest installed the mocks unconditionally. The
KEEP_NO_COMFY_MOCKS=1 opt-out lets the headless tier import the real
comfy/comfy_api modules in-process.
tools/ci_env_sim.py runs the gate in an environment without the packages
the CI install list omits.
- plan section 9: P1.1-P1.5 marked done (2026-10-23)
- docs/audit/2026-10-23-p1-test-infrastructure.md: the deviations of the
phase (the pytest collection artifact had to stay, the CI install list was
completed, the ruff scope, the shadowed tests package)
- docs/2026-10-23-comfyui-keep-p1-phase.md: the phase review with the gate
results
.github/workflows/tests.yml:
- lint job: pip install 'ruff==0.8.*' (pinned major) + ruff check .
- test job: matrix python 3.10/3.12/3.13, CPU torch + torchvision from the
pytorch CPU index, numpy<2.3 and tomli on 3.10 only, opencv-python-headless,
einops, scipy, Pillow, tqdm, pytest; runs
python -m pytest -q -m 'unit or mock_integration or headless' with
COMFYUI_ROOT empty so the env-dependent tests skip
- publish.yml untouched
pyproject.toml: [tool.ruff] scope (E9+F, F401/F541 ignored) and the vendored
tree modules/deps excluded - with the default rules the repo has 127 findings.
tests/test_ci_workflow.py: parses the workflow and asserts the jobs, the
matrix, the install steps, COMFYUI_ROOT='' and that the CI command equals the
local gate command.
tests/_headless.py resolves the ComfyUI tree from COMFYUI_ROOT and the weight
directory from KEEP_MODEL_DIR (no hardcoded paths, actionable errors), loads
extra_model_paths.yaml through utils.extra_config like the server does,
registers a package under an importable alias with __path__ set and can stub
only the ui preview helpers.
tests/test_headless_bootstrap.py proves the helper on a dummy package: alias
resolution, submodule import, the missing-environment-variable errors and the
absence of machine-specific paths.
tests/test_structure.py implements the five plan guards as pure functions over
a repository root, each xfail(strict=False) with the step that flips it
strict in the reason:
(a) nodes/ classes exported by nodes/__init__.py -> P3.1
(b) no io.Schema(/define_schema in src/ -> P3.2
(c) __init__.py exposes exactly comfy_entrypoint -> P4.4
(d) no sys.path mutation outside modules/deps/ -> P2.5
(e) no pdb/profiler/pip in the vendored closure -> P2.1
The self-tests copy the repository into tmp_path and prove that each guard
fails on a deliberately violating file.
- conftest.py: comfy_api.latest (io/ui as siblings, ComfyExtension,
ComfyAPI/ComfyAPISync with an instance-only progress recorder),
comfy.model_management, comfy.model_patcher, comfy.utils and folder_paths
are installed before collection; the package alias ComfyUI_KEEP keeps
__path__ + submodule_search_locations and provides the logger the
modules/* code imports
- fixtures: deterministic sample_image and an autouse per-test
models_dir/folder-registry reset
- tests/test_conftest_mocks.py: dummy V3 node end-to-end through the mock
io, folder_paths isolation, mock-fidelity tripwire against the real
ComfyUI source (skips cleanly without COMFYUI_ROOT)
- pyproject: [tool.pytest.ini_options] with testpaths=tests, the four plan
markers (unit/mock_integration/headless/gpu_acceptance), addopts
'-v --tb=short --import-mode=importlib', consider_namespace_packages and
pythonpath=['.']
- tests/test_sanity.py::test_python_and_torch pins the supported interpreter
and the torch import
- 'python -m pytest --collect-only -q' exits 0 (40 tests collected)
All 7 P0 review findings were verified by execution and fixed.
F1 (medium) - the compat scanner modelled only the basicsr/archs eager scan.
The vendored basicsr also scans data/, losses/ and models/ eagerly
(basicsr/__init__.py:3-11 imports all of them), so the closure missed 32
files that the repaired package really imports (incl. models/keep_model.py and
models/keep_gan_model.py, both with 'import pdb' at line 4). The scanner now
models all four scans plus 'from <package> import <submodule>': closure
100 -> 132 files, findings 8 -> 10, and 0 runtime-loaded vendored files are
missing from the closure (verified on a temp copy of modules/deps).
The audit doc's opposite claim is corrected and P2.4 in the plan now has to
rewrite basicsr/__init__.py and prove 'import basicsr' still succeeds after
pruning (otherwise ModuleNotFoundError is swallowed into DummyArchRegistry).
F2 (medium) - the Asian manifest entry is now decided (P0.4 section 4.1):
url 404, sha256 null, sha256_prefix '4765ebe0', unverified true, available
false, required FALSE (a required entry would break the default KEEP model
through P5.3's fail-fast rule). Plan P5.0/P5.1/P5.3 and the weights scanner
were aligned.
F3 (low) - the plan's section 8 decision 1, the B3 baseline row, Goal 5, the
risk table and P5.5 carried the superseded GMFlow premise; they now match the
P0.4 decision (the shipped checkpoint carries the flownet weights; the GMFlow
file is defensive and resolved only when the checkpoint has no flownet.* keys).
F4 (low) - the P0.1 fixture asserted rc == 0 before checking the probe's fatal
report, so the documented skip was unreachable. It now handles the probe's
exit code 2 + fatal report and skips with a reason.
F5 (low) - the baseline-import doc claimed all eight stub symbols fire in the
ffmpeg-blocked run; it is 7 of 8 (loader_face_restore_helper_is_dummy does not
fire - the real facelib imports fine).
F6 (low) - the scanner's exit-code contract (0 = no findings, 1 = findings) is
documented in the tool docstring and in the audit doc.
F7 (low) - the root conftest registered '__init__' and the invalid directory
basename; it now registers the plan's alias 'ComfyUI_KEEP' and provides
execute_package_init() so the V3 entrypoint is executed rather than bypassed.
P1.2 is specified to replace that bootstrap.
Gates: pytest tests/ -q -> 37 passed, 1 xfailed; scan_compat.py --quiet ->
closure 132, 10 findings, exit 1; baseline_probe.py -> import ok / model build
fails (unchanged baseline); scan_weights.py -> 0 of 8 present.
WildAi