* fix(logs): declare the compressed length so a log download finishes The gzip middleware compressed a text body but kept the handler's declared Content-Length (the uncompressed size), so a browser waited for bytes that never came: downloading any log over 1 KB hung, in the workspace and on the admin page. The middleware now drops a declared length before compressing. Part of #329 * feat(system): show Diagnostics & logs natively and retire the hand-off panel System › Diagnostics & logs is native (D109, D110): the concept's Diagnostics surface with the live database, source, Viewer.js and renderer lines, Run checks (the shared Health report), the diagnostic report, Inspect and the confirmed test link; and the Logs surface with the minimum-level viewer, Lines 1-500, statistics, the log download, the runtime level and console setting saved by Save preferences, the log files with their downloads, Review log cleanup (the rotated_logs maintenance plan, no backup box, Listed files) and a confirmed Clear all logs. D94's hand-off panel, SECTION_HANDOFFS and the undelivered runtime adapter are gone. The browser server logs into its own temporary root. Part of #329 * test(system): compare Diagnostics & logs with the concept System › Diagnostics & logs and its log cleanup review are compared with the concept (app.js 694–695 and maintenancePlan("rotated logs")) in both themes at 1600x1100 and 390x844, with D109 and D110 (and D100 for the review) as the only mapped differences, a 320-wide select check and a mutation check. The fixture gains a concept seed mode (the concept's four log lines, a 1.4 MB active log with three rotated copies, console logging off), and the plan dialog's measureReview and mapPlanConcept move unchanged into concept-inventory.mjs for Maintenance and this review. The comparison found that the Database line read a report key that does not exist (database_info.size_bytes) and so always showed "0 bytes"; it now reads database_size_bytes, the key get_database_info() returns. Part of #329 * test(system): prove every legacy System capability has an owner TESTING.md gains "Legacy System capability coverage": every System capability of the legacy admin and gallery pages with its anchor on main, owner, status (covered, kept in legacy or approved removal), native control and the exact title of the test that proves it. tests/vendor/legacy-coverage.test.mjs (with tools/vendor/legacy-coverage.mjs) fails on a row with no owner, an unknown status, a malformed native control, a control on a row that is not covered, or a test title its file does not declare; a title only in a comment does not count. workspace-system.spec's "every covered legacy System capability has its native control" opens each covered row's section and finds the control by role and accessible name. Versus the design's table: the anchors are re-checked against main, "Run health checks" is on Maintenance (D123), not Overview, Diagnostics run is proven by "Run checks opens the shared Health report", and the export rows repeat their test title instead of "(same as above)". Part of #329 * fix(system): confirm Clear all logs on a fresh file list and record coverage Review fixes for the Diagnostics & logs section: - Clear all logs reads the log files again before it confirms, so the dialog names the copies it deletes now (a list it cannot read gives the general sentence, never an old count); Refresh reads the files too. - Runtime values that cannot be read stay disabled and say so. - Create test link holds its button until the write answered, so a second press cannot write a second record. - The Lines field names its range. - The browser server builds its logger in its own root, so no log file is ever opened in another folder; the fixture closes every directory scan and replaces a link instead of writing through it. - system.mjs drops the unused section fallback; frontend coverage floors recorded with coverage:update (system-diagnostics.mjs 97.5% lines). Part of #329 * fix(system): read the database size the report sends The Database line read database_info.size_bytes, a key the report does not have (get_database_info answers database_size_bytes), so it always read "Readable · 0 bytes · N images". The browser test now requires a non-zero size. Found by PR 3's concept comparison. Part of #329 * test(system): require a running test and a control for every covered capability Review fixes for the legacy coverage proof: - A title declared only with test.skip, .fixme or .fail no longer proves a row (.only and .slow still do). - Every covered row names its native control, except the rows in CONTROL_EXEMPTIONS, each with its reason (today only the gallery per-image auto-tag row, owned by the Images selection bar). The browser test asserts that the covered rows without a control are exactly the exemptions. - JavaScript comments are stripped across the whole source, wherever a block comment starts, skipping strings, template literals and regular expressions. A Python def test_ counts only at function or method indentation and outside comments and docstrings. - A control name several rows share ("Review plan", "Review options", "Refresh") is looked up inside the row the table names (· in "<row>"). - The diagnostics comparison records each select's chosen option, so a wrong default level is caught. Part of #329
268 lines
11 KiB
Python
268 lines
11 KiB
Python
"""System › Diagnostics & logs test hooks for the browser tests (#249 System S12).
|
||
|
||
The logger writes into ``<server root>/logs`` (``repoint_logs``, called by
|
||
``create_app`` before any record is written), so no test reads or deletes a
|
||
real log folder. Every route refuses a log folder outside the server root.
|
||
|
||
- ``POST /__test__/logs/reset`` restores the default logging configuration,
|
||
empties the active log, removes every other entry of the log folder (a
|
||
link is removed, never followed) and clears the in-memory buffer. The
|
||
global ``/__test__/reset`` calls it too;
|
||
- ``POST /__test__/logs/seed`` ``{records, filler, rotated, active, link,
|
||
other}`` writes buffer records with fixed times (``records``: the concept's
|
||
four lines, then one DEBUG, ERROR and CRITICAL record, older; ``filler``
|
||
more INFO records older still), ``active`` bytes into the active log,
|
||
rotated copies ``prompt_manager.log.<n>`` of the listed sizes (fixed
|
||
times), on POSIX a symlink ``prompt_manager.log.<n+1>`` to a file outside
|
||
the folder (``link``) and a user's ``prompt_manager.log.bak`` (``other``).
|
||
``{mode: "concept"}`` reproduces the concept's Logs sample (app.js 695)
|
||
for the comparison: the buffer holds only its four lines, the active log
|
||
is 1.4 MB with three rotated copies ("prompt_manager.log · 1.4 MB · rotated
|
||
copies: 3") and console logging is off, as the concept's toggle;
|
||
- ``POST /__test__/logs/rotate`` rolls the copies over as the handler does
|
||
(``.2`` → ``.3``, ``.1`` → ``.2``, a new ``.1``), so a reviewed plan no
|
||
longer matches;
|
||
- ``GET /__test__/logs/state`` answers the folder's entries (name, size, link)
|
||
and the buffer's length.
|
||
"""
|
||
|
||
import datetime
|
||
import logging
|
||
import os
|
||
import shutil
|
||
from pathlib import Path
|
||
|
||
from aiohttp import web
|
||
|
||
ACTIVE = "prompt_manager.log"
|
||
SEED_DAY = datetime.datetime(2026, 10, 26)
|
||
# The concept's four lines (app.js 695), oldest first: the viewer shows them
|
||
# newest first.
|
||
CONCEPT_RECORDS = (
|
||
("09:38:52", logging.INFO, "Source monitor started: output"),
|
||
("09:39:04", logging.INFO, "Database backup verified"),
|
||
("09:42:18", logging.WARNING, "3 images have ambiguous prompt metadata"),
|
||
("09:42:18", logging.INFO, "Scan completed: 12 enriched, 0 duplicate links"),
|
||
)
|
||
# Older than the concept's lines, so they come after them: the levels the
|
||
# concept does not show.
|
||
LEVEL_RECORDS = (
|
||
("09:30:01", logging.DEBUG, "Thumbnail cache lookup"),
|
||
("09:30:02", logging.ERROR, "A source folder could not be read"),
|
||
("09:30:03", logging.CRITICAL, "The database could not be opened"),
|
||
)
|
||
ROTATED_TIME = datetime.datetime(2026, 10, 25, 14, 2)
|
||
# The concept's "prompt_manager.log · 1.4 MB · rotated copies: 3": the active
|
||
# log stays under 1.45 MB while the server writes its own lines.
|
||
CONCEPT_ACTIVE = int(1.4 * 1024 * 1024)
|
||
CONCEPT_ROTATED = (3072, 2048, 1024)
|
||
MAX_SEED_BYTES = 64 * 1024 * 1024
|
||
DEFAULT_CONFIG = {
|
||
"level": "INFO",
|
||
"max_file_size": 10 * 1024 * 1024,
|
||
"backup_count": 5,
|
||
"console_logging": True,
|
||
"file_logging": True,
|
||
"buffer_size": 1000,
|
||
}
|
||
|
||
|
||
def _manager():
|
||
from utils.logging_config import get_logger_manager
|
||
|
||
return get_logger_manager()
|
||
|
||
|
||
def repoint_logs(root: Path) -> Path:
|
||
"""Point the logger at ``<root>/logs`` before anything is logged.
|
||
|
||
The logger is built while ``logs_dir`` answers that folder, so its first
|
||
handler never opens a log file anywhere else (a node folder with a legacy
|
||
database would otherwise name a real log folder).
|
||
"""
|
||
from utils import library_location
|
||
|
||
folder = Path(root) / "logs"
|
||
folder.mkdir(exist_ok=True)
|
||
original = library_location.logs_dir
|
||
library_location.logs_dir = lambda **_kwargs: str(folder)
|
||
try:
|
||
manager = _manager()
|
||
finally:
|
||
library_location.logs_dir = original
|
||
with manager._files_lock:
|
||
manager.log_dir = folder
|
||
manager._setup_loggers()
|
||
return folder
|
||
|
||
|
||
def _require_inside(root: Path, folder: Path) -> None:
|
||
resolved = Path(os.path.realpath(folder))
|
||
if Path(os.path.realpath(root)) not in resolved.parents:
|
||
raise web.HTTPForbidden(text="The log folder is outside the test server root")
|
||
|
||
|
||
def _record(time_text, level, message, index):
|
||
record = logging.LogRecord(
|
||
"prompt_manager.test", level, __file__, index, message, None, None
|
||
)
|
||
hours, minutes, seconds = (int(part) for part in time_text.split(":"))
|
||
moment = SEED_DAY.replace(hour=hours, minute=minutes, second=seconds)
|
||
# Equal times keep their order: each later record is a microsecond newer.
|
||
record.created = moment.timestamp() + index / 1e6
|
||
return record
|
||
|
||
|
||
def _touch(path: Path, size: int, moment: datetime.datetime) -> None:
|
||
# A link at this name is replaced, never written through.
|
||
if path.is_symlink():
|
||
path.unlink()
|
||
with open(path, "wb") as handle:
|
||
handle.write(b"x" * size)
|
||
stamp = moment.timestamp()
|
||
os.utime(path, (stamp, stamp))
|
||
|
||
|
||
def _bounded(value, name, top=MAX_SEED_BYTES):
|
||
if isinstance(value, bool) or not isinstance(value, int) or not 0 <= value <= top:
|
||
raise web.HTTPBadRequest(text=f"{name} must be an integer 0..{top}")
|
||
return value
|
||
|
||
|
||
def register_diagnostics_commands(app, routes, api, root: Path):
|
||
"""Register the hooks; returns ``reset()`` for the global reset."""
|
||
|
||
def folder() -> Path:
|
||
path = Path(_manager().log_dir)
|
||
_require_inside(root, path)
|
||
return path
|
||
|
||
def reset() -> None:
|
||
manager = _manager()
|
||
log_dir = folder()
|
||
if manager.get_config() != DEFAULT_CONFIG:
|
||
manager.update_config(dict(DEFAULT_CONFIG))
|
||
with manager._files_lock:
|
||
held = manager._acquire_file_handlers()
|
||
try:
|
||
with os.scandir(log_dir) as entries:
|
||
found = list(entries)
|
||
for entry in found:
|
||
path = Path(entry.path)
|
||
if entry.name == ACTIVE and entry.is_file(follow_symlinks=False):
|
||
manager._truncate_main_log(path)
|
||
elif entry.is_dir(follow_symlinks=False):
|
||
shutil.rmtree(path)
|
||
else:
|
||
os.unlink(path)
|
||
finally:
|
||
manager._release_handlers(held)
|
||
with manager._buffer_lock:
|
||
manager._log_buffer.clear()
|
||
|
||
def seed_concept() -> None:
|
||
manager = _manager()
|
||
log_dir = folder()
|
||
manager.update_config({"console_logging": False})
|
||
with manager._buffer_lock:
|
||
manager._log_buffer.clear()
|
||
for index, (time_text, level, message) in enumerate(CONCEPT_RECORDS):
|
||
record = _record(time_text, level, message, index)
|
||
manager.add_to_buffer(record, record.getMessage())
|
||
with open(log_dir / ACTIVE, "ab") as handle:
|
||
handle.write(b"x" * CONCEPT_ACTIVE)
|
||
for number, size in enumerate(CONCEPT_ROTATED, start=1):
|
||
moment = ROTATED_TIME - datetime.timedelta(hours=number)
|
||
_touch(log_dir / f"{ACTIVE}.{number}", size, moment)
|
||
|
||
def seed(data) -> None:
|
||
if data.get("mode") == "concept":
|
||
seed_concept()
|
||
return
|
||
manager = _manager()
|
||
log_dir = folder()
|
||
with_records = data.get("records", True) is not False
|
||
filler = _bounded(data.get("filler", 0), "filler", 900)
|
||
older = [
|
||
(f"08:{index // 60:02}:{index % 60:02}", logging.INFO, f"Filler {index}")
|
||
for index in range(filler)
|
||
]
|
||
named = list(LEVEL_RECORDS) + list(CONCEPT_RECORDS) if with_records else []
|
||
ordered = older + named
|
||
for index, (time_text, level, message) in enumerate(ordered):
|
||
record = _record(time_text, level, message, index)
|
||
manager.add_to_buffer(record, record.getMessage())
|
||
active = _bounded(data.get("active", 0), "active")
|
||
if active:
|
||
with open(log_dir / ACTIVE, "ab") as handle:
|
||
handle.write(b"x" * active)
|
||
rotated = data.get("rotated", [])
|
||
if not isinstance(rotated, list) or len(rotated) > 20:
|
||
raise web.HTTPBadRequest(text="rotated must be a list of sizes")
|
||
for number, size in enumerate(rotated, start=1):
|
||
moment = ROTATED_TIME - datetime.timedelta(hours=number)
|
||
_touch(log_dir / f"{ACTIVE}.{number}", _bounded(size, "size"), moment)
|
||
if data.get("link") and os.name != "nt":
|
||
target = Path(root) / "outside-log-target.txt"
|
||
target.write_bytes(b"not a log of this folder")
|
||
os.symlink(target, log_dir / f"{ACTIVE}.{len(rotated) + 1}")
|
||
if data.get("other"):
|
||
_touch(log_dir / f"{ACTIVE}.bak", 0, ROTATED_TIME)
|
||
|
||
def rotate() -> None:
|
||
manager = _manager()
|
||
log_dir = folder()
|
||
with manager._files_lock:
|
||
held = manager._acquire_file_handlers()
|
||
try:
|
||
for number in (2, 1):
|
||
source = log_dir / f"{ACTIVE}.{number}"
|
||
if source.is_file() and not source.is_symlink():
|
||
os.replace(source, log_dir / f"{ACTIVE}.{number + 1}")
|
||
_touch(log_dir / f"{ACTIVE}.1", 4096, datetime.datetime.now())
|
||
finally:
|
||
manager._release_handlers(held)
|
||
|
||
def state():
|
||
manager = _manager()
|
||
log_dir = folder()
|
||
entries = []
|
||
with os.scandir(log_dir) as listing:
|
||
found = sorted(listing, key=lambda item: item.name)
|
||
for entry in found:
|
||
info = entry.stat(follow_symlinks=False)
|
||
entries.append(
|
||
{"name": entry.name, "size": info.st_size, "link": entry.is_symlink()}
|
||
)
|
||
with manager._buffer_lock:
|
||
buffered = len(manager._log_buffer)
|
||
return {"entries": entries, "buffer": buffered, "config": manager.get_config()}
|
||
|
||
async def body(request):
|
||
data = await request.json() if request.can_read_body else {}
|
||
if not isinstance(data, dict):
|
||
raise web.HTTPBadRequest()
|
||
return data
|
||
|
||
@routes.post("/__test__/logs/reset")
|
||
async def logs_reset(request: web.Request) -> web.Response:
|
||
await api._run_in_executor(reset)
|
||
return web.json_response({"success": True})
|
||
|
||
@routes.post("/__test__/logs/seed")
|
||
async def logs_seed(request: web.Request) -> web.Response:
|
||
data = await body(request)
|
||
await api._run_in_executor(seed, data)
|
||
return web.json_response({"success": True})
|
||
|
||
@routes.post("/__test__/logs/rotate")
|
||
async def logs_rotate(request: web.Request) -> web.Response:
|
||
await api._run_in_executor(rotate)
|
||
return web.json_response({"success": True})
|
||
|
||
@routes.get("/__test__/logs/state")
|
||
async def logs_state(request: web.Request) -> web.Response:
|
||
return web.json_response(await api._run_in_executor(state))
|
||
|
||
return reset
|