After the faster walk, a read that found the listing stale still took
0.5-1.0 s against 0.07 s: it was served from the old listing but ran beside
the refresh walk it had just started, sharing the GIL. The refresh now starts
REFRESH_DELAY (1 s) after that read through a per-root rewalk_handle, so the
page's own requests finish first. One pending refresh per root and none while
a walk runs; any walk start (a cold read, Rescan, the warm-up) cancels a
pending one; on_cleanup cancels them before its first await.
The live test showed a read during a background refresh taking 2.2 s instead
of 0.07 s: the walk held the GIL for about 2 s in the server. py-spy split
that walk into two halves: Path.relative_to per file in folder_sets, and
Path.stat + Path.is_symlink + Path joins per file in the scan.
- _walk_media visits folders in os.walk's order with os.scandir entries:
the file type comes from the listing (no lstat per file) and the mtime
from entry.stat (one call, none on Windows). _iter_media_files is now a
thin wrapper, so its other caller is unchanged.
- folder_sets slices the path strings (in the path flavour's separator)
instead of calling relative_to, falling back to it for a path that does
not start with the root string.
On the maintainer's 19,935-file output folder: scan 0.386 s -> 0.112 s,
folder sets 0.452 s -> 0.007 s, both results identical to the previous code.
A reference copy of the os.walk version checks order, filters, the depth
cap, symlinks and the file cap.
The Images page, its sidebar counts, the subfolder list and the legacy
gallery each rebuilt the output listing when the 30 s cache expired, all
at once, and the subfolder list computed relative paths on the event
loop. Thumbnail writes also threw the listing away.
Keep one RootListing per output root (py/api/gallery_listing.py):
- one walk per root at a time; concurrent readers wait for it (shielded)
- a listing older than 30 s is served at once and refreshed in the
background
- the walk, the comparison with the previous listing and both folder
sets run in the executor; an unchanged walk keeps the same list object
- os.scandir probe: a missing root fails the walk and keeps the previous
listing (one warning per failure streak)
- Rescan waits for a walk started after the click and is a 500 when it
fails, even with a previous listing
- deletes keep the blocking invalidation; a generation counter that
survives drops discards walks that raced it
- thumbnail jobs, conversion and the legacy generate/clear routes no
longer drop the listing
- warm-up on the host's on_startup, cancelled on on_cleanup
Adds tools/vendor/perf/images_load_probe.mjs (maintainer tool, not CI)
and the TESTING.md section "Images listing cache (#348)".
Vito Sansevero