WildAi
4c44a8e275
Five review findings of the P2 phase, each fixed at its root: 1. (medium) The P2.6 guard could not see an undeclared third-party import: the closure's imports were filtered through a hard-coded 10-name map, so every module outside it was dropped silently (a module-level `import av` in a copy of the repo still gave "1 passed"). The map is now a *rename* map (cv2 -> opencv-python, PIL -> Pillow, yaml -> pyyaml); every other module is mapped to its own name, so an unknown module shows up as an undeclared dependency. The guard is split by import time: every third-party module imported at import time must be a declared distribution, every lazy (function-level) one must be allowlisted with a reason (dlib, gdown, mc, wandb - all unreachable from the node). 2. (low) The P2.1 (c) progression test was vacuous: it returned before its assertions and probed a second, top-level copy of the vendored tree (`sys.path.insert(modules/deps)`). It now imports the packaged architecture, asserts that FeedForward / AdaLayerNorm come from the vendored _diffusers_compat and that no top-level `basicsr` shadows an installed one. 3. (low) Two swallow-and-stub fallbacks remained in the vendored tree (facelib/detection, facelib/parsing - their stubs called an unimported `urlparse` and would have raised NameError). Both now import the vendored symbol directly; a guard forbids a fallback stub in the vendored tree. 4. (low) `basicsr/utils/file_client.py` still appended to `sys.path` (the optional `sys_path` of MemcachedBackend), which the plan's §3 rule forbids anywhere in the package. The mutation and the parameter are gone and guard (d) now covers the whole package (the vendored tree included). 5. (low) The unreachable body of `KEEPFaceProcessor.process_video` still called the deleted VideoReader / VideoWriter. The 141 dead lines are removed; an AST guard keeps the references out. Found while fixing the findings: the CI job did not install `requests`, which the vendored `basicsr/utils/download_util.py` imports at module level (P2.6 added it to requirements.txt but not to the workflow) - a clean CI runner died with "No module named 'requests'". The install list is now completed and a test derives it from requirements.txt. Gate: pytest -m "unit or mock_integration" -> 109 passed, 1 skipped, 4 xfailed; plan's local gate (+ headless) -> 109 passed, 3 skipped, 4 xfailed; tools/scan_compat.py -> finding_count 0 (exit 0); ruff check . -> All checks passed. A mutation check re-introduces six defects into a copy of the repository: the matching guard fails in 6/6 cases.
103 lines
3.2 KiB
Python
103 lines
3.2 KiB
Python
# Modified from https://github.com/open-mmlab/mmcv/blob/master/mmcv/fileio/file_client.py # noqa: E501
|
|
#
|
|
# The optional ``sys_path`` parameter of ``MemcachedBackend`` (it appended to
|
|
# ``sys.path``) is removed: the package must never mutate ``sys.path``
|
|
# (plan §3, import rules). Nothing in the closure passes it.
|
|
from abc import ABCMeta, abstractmethod
|
|
|
|
|
|
class BaseStorageBackend(metaclass=ABCMeta):
|
|
"""Abstract class of storage backends.
|
|
|
|
All backends need to implement two apis: ``get()`` and ``get_text()``.
|
|
``get()`` reads the file as a byte stream and ``get_text()`` reads the file
|
|
as texts.
|
|
"""
|
|
|
|
@abstractmethod
|
|
def get(self, filepath):
|
|
pass
|
|
|
|
@abstractmethod
|
|
def get_text(self, filepath):
|
|
pass
|
|
|
|
|
|
class MemcachedBackend(BaseStorageBackend):
|
|
"""Memcached storage backend.
|
|
|
|
Attributes:
|
|
server_list_cfg (str): Config file for memcached server list.
|
|
client_cfg (str): Config file for memcached client.
|
|
"""
|
|
|
|
def __init__(self, server_list_cfg, client_cfg):
|
|
try:
|
|
import mc
|
|
except ImportError:
|
|
raise ImportError('Please install memcached to enable MemcachedBackend.')
|
|
|
|
self.server_list_cfg = server_list_cfg
|
|
self.client_cfg = client_cfg
|
|
self._client = mc.MemcachedClient.GetInstance(self.server_list_cfg, self.client_cfg)
|
|
# mc.pyvector servers as a point which points to a memory cache
|
|
self._mc_buffer = mc.pyvector()
|
|
|
|
def get(self, filepath):
|
|
filepath = str(filepath)
|
|
import mc
|
|
self._client.Get(filepath, self._mc_buffer)
|
|
value_buf = mc.ConvertBuffer(self._mc_buffer)
|
|
return value_buf
|
|
|
|
def get_text(self, filepath):
|
|
raise NotImplementedError
|
|
|
|
|
|
class HardDiskBackend(BaseStorageBackend):
|
|
"""Raw hard disks storage backend."""
|
|
|
|
def get(self, filepath):
|
|
filepath = str(filepath)
|
|
with open(filepath, 'rb') as f:
|
|
value_buf = f.read()
|
|
return value_buf
|
|
|
|
def get_text(self, filepath):
|
|
filepath = str(filepath)
|
|
with open(filepath, 'r') as f:
|
|
value_buf = f.read()
|
|
return value_buf
|
|
|
|
|
|
class FileClient(object):
|
|
"""A general file client to access files in different backend.
|
|
|
|
The client loads a file or text in a specified backend from its path
|
|
and return it as a binary file. it can also register other backend
|
|
accessor with a given name and backend class.
|
|
|
|
Attributes:
|
|
backend (str): The storage backend type. Options are "disk" and
|
|
"memcached".
|
|
client (:obj:`BaseStorageBackend`): The backend object.
|
|
"""
|
|
|
|
_backends = {
|
|
'disk': HardDiskBackend,
|
|
'memcached': MemcachedBackend,
|
|
}
|
|
|
|
def __init__(self, backend='disk', **kwargs):
|
|
if backend not in self._backends:
|
|
raise ValueError(f'Backend {backend} is not supported. Currently supported ones'
|
|
f' are {list(self._backends.keys())}')
|
|
self.backend = backend
|
|
self.client = self._backends[backend](**kwargs)
|
|
|
|
def get(self, filepath, client_key='default'):
|
|
return self.client.get(filepath)
|
|
|
|
def get_text(self, filepath):
|
|
return self.client.get_text(filepath)
|