fix(thumbnails): keep going past the per-batch cap until every preview is made

The missing, rebuild and Convert outdated jobs stopped after MAX_THUMBNAIL_FILES (10,000) files and said "N more on the next run", but nothing ever started that next run. The cap now bounds one batch. After each batch the job plans again, leaving out every file it already worked on, and continues until nothing is left, the job is cancelled, or a batch makes no progress. That last case stops on files that fail every time instead of looping over them. Activity progress counts against the overall total, and the final line and the legacy stream's complete frame report every batch. Admission, worker threads, cancellation and the #316 rules are unchanged. The Thumbnail maintenance dialog now counts all the work instead of saying "more on the next run".
2026-10-24 16:57:03 -07:00
parent af0482a12d
commit 726b7b70da
9 changed files with 318 additions and 88 deletions
@@ -2164,7 +2164,7 @@ changed." and Retry, never an empty list.

 Thumbnail generation is an Activity job (S5, `py/api/thumbnail_jobs.py`). `GET
 /prompt_manager/images/thumbnails/plan?mode=missing|rebuild|convert`
-counts the work (`targets`, `fresh`, `outdated`, `remaining` past the 10,000-file cap,
+counts the work (`targets`, `fresh`, `outdated`, `remaining` past the 10,000-file batch cap,
 the source labels, and the server's `format`, `quality` and `size` it writes with) and
 writes nothing; an unknown mode is a 400 `invalid-request` that starts nothing. `size`
 and `quality` an older client sends are ignored (#317). `POST /prompt_manager/jobs/thumbnails {mode}` starts the
@@ -2178,6 +2178,12 @@ previews take the same settings: the OpenCV frame, ffmpeg's frame (no OpenCV; a
 stdout fitted inside the edge keeping its aspect ratio and never upscaled, `-vf
 scale='min(iw,E)':'min(ih,E)':force_original_aspect_ratio=decrease`) and the
 placeholder are encoded like an image's preview.
+MAX_THUMBNAIL_FILES (10,000) bounds one batch, not the job: after each batch the
+missing, rebuild and convert jobs plan again, leaving out every file a batch already
+worked on, and go on until nothing is left, the job is cancelled, or a batch made no
+progress (every file failed: the rest waits for the next run). Progress counts against
+the overall total and the final line reports every batch (`TestBatchesContinue`,
+`test_convert_goes_on_past_the_per_batch_cap`). The plan dialog counts all of it.
 `rebuild` writes fresh previews again, each through the same temp-file
 rename, so nothing is cleared first and a failed file keeps its old preview (a video
 whose frame cannot be decoded keeps its preview instead of getting the placeholder);
@@ -2185,7 +2191,7 @@ originals are only read (`tests/test_thumbnail_jobs.py`). Cancel is also checked
 file during the planning walk: a job cancelled there stops the walk at the next file,
 publishes no `start` frame and ends "Cancelled before any preview was written". Every
 exit sets a final status line: "N previews written · M errors" (plus "· R more on the
-next run" past the cap), "No previews to generate · N already up to date" (or "No
+next run" when a batch made no progress), "No previews to generate · N already up to date" (or "No
 images or videos to preview"), "Cancelled · N previews written", or the failure; a job
 with nothing to do never reads "Processing file 0 of 0" (`test_every_exit_sets_a_truthful_final_status_line`,
 and "a finished thumbnail job with nothing to do says so" in `workspace-activity.spec.mjs`).
@@ -2485,8 +2491,8 @@ Recorded deviations (plan D60–D67, D79–D81 renumbered D92–D99 and D111–D
  maintenance") with production copy. A `p.small[role=status]` line before the actions
  says what the server's read-only plan (`GET /images/thumbnails/plan`) counts for the
  chosen scope: "N previews to generate · M already up to date." or, for Rebuild all,
-  "Every preview is regenerated: N files.", with "· R more on the next run" past the
-  per-run cap; "Counting previews…" while it is read and "The previews could not be
+  "Every preview is regenerated: N files.", N counting the files past the per-batch
+  cap too (#317); "Counting previews…" while it is read and "The previews could not be
  counted." when the read fails (Start still starts). Each scope is read once per
  dialog, since size and quality change no count. The primary is "Start" (the
  concept's "Simulate thumbnail job"). Overview's Review thumbnails opens the same
@@ -1991,12 +1991,15 @@ class ImageRoutesMixin:
        mode: str = "missing",
        stop=None,
        settings=None,
+        exclude=None,
    ) -> dict:
-        """Plan every gallery root with one shared per-run generation cap.
+        """Plan every gallery root with one shared per-batch generation cap.

        See ``_plan_thumbnails``; ``fresh_files`` keeps at most
        MAX_FRESH_LISTED files across the roots. ``stop()`` true ends the walk
        at the next file (a cancelled job); the partial plan is returned.
+        *exclude* (a set of source paths) leaves files a job already worked
+        on out of a later batch's plan.
        """
        settings = writer_settings(settings)
        combined = {key: 0 for key in _PLAN_COUNTS}
@@ -2006,7 +2009,12 @@ class ImageRoutesMixin:
                break
            root = Path(root)
            plan = self._plan_thumbnails(
-                root, root / "thumbnails", mode=mode, stop=stop, settings=settings
+                root,
+                root / "thumbnails",
+                mode=mode,
+                stop=stop,
+                settings=settings,
+                exclude=exclude,
            )
            available = max(0, MAX_THUMBNAIL_FILES - len(combined["targets"]))
            combined["targets"].extend(plan["targets"][:available])
@@ -2054,7 +2062,13 @@ class ImageRoutesMixin:
            )

    def _plan_thumbnails(
-        self, output_path, thumbnails_dir, mode="missing", stop=None, settings=None
+        self,
+        output_path,
+        thumbnails_dir,
+        mode="missing",
+        stop=None,
+        settings=None,
+        exclude=None,
    ):
        """Decide which media files need a preview (blocking walk).

@@ -2071,9 +2085,10 @@ class ImageRoutesMixin:
        ``fresh_files`` lists the first MAX_FRESH_LISTED fresh files. The walk
        is bounded like the gallery listing (MAX_SCAN_DEPTH, MAX_GALLERY_FILES,
        no symlinks), each preview folder is listed once, and at most
-        MAX_THUMBNAIL_FILES targets are queued per run; ``remaining`` counts
-        the targets beyond that cap. ``stop()`` true ends the walk before the
-        next file.
+        MAX_THUMBNAIL_FILES targets are queued per batch (a job plans again
+        after each batch); ``remaining`` counts the targets beyond that cap.
+        Sources in *exclude* (worked on by an earlier batch) are left out.
+        ``stop()`` true ends the walk before the next file.
        """
        settings = writer_settings(settings)
        output_path = Path(output_path)
@@ -2084,6 +2099,8 @@ class ImageRoutesMixin:
        for src in _iter_media_files(output_path, MAX_GALLERY_FILES):
            if stop is not None and stop():
                break
+            if exclude and src in exclude:
+                continue
            rel_no_ext = src.relative_to(output_path).with_suffix("")
            stem = f"{rel_no_ext.name}{thumbnail_settings.THUMB_SUFFIX}"
            dst = thumbnails_dir / rel_no_ext.parent / (stem + settings.extension)
@@ -2109,8 +2126,8 @@ class ImageRoutesMixin:
            plan["targets"].append((src, dst, is_video))
        if plan["remaining"]:
            self.logger.warning(
-                f"Thumbnail run capped at {MAX_THUMBNAIL_FILES} new files; "
-                f"{plan['remaining']} more will be generated on the next run"
+                f"Thumbnail batch capped at {MAX_THUMBNAIL_FILES} files; "
+                f"{plan['remaining']} more are left for the next batch"
            )
        return plan

@@ -167,21 +167,14 @@ class ThumbnailConvertMixin:
        job.log_line(
            f"Conversion started · {folders} · to {options.settings.describe()}"
        )
-        plan = await self._run_in_executor(
-            self._plan_gallery_thumbnails,
-            output_dirs,
-            "convert",
-            stop=lambda: job.cancelled,
-            settings=options.settings,
-        )
+        plan = await self._plan_conversion(job, options, output_dirs)
        if job.cancelled:
            return self._end_conversion(job, 0, CANCELLED_BEFORE_CONVERTING)
-        targets = plan["targets"]
-        if not targets:
+        if not plan["targets"]:
            job.update(progress=100, status_line=NOTHING_OUTDATED)
            job.log_line(f"Conversion finished · {NOTHING_OUTDATED.lower()}")
            return "finished"
-        totals = await self._convert_targets(job, options, targets, output_dirs)
+        totals, left = await self._convert_in_batches(job, options, plan, output_dirs)
        self.invalidate_gallery_cache()
        if totals["stopped"]:
            job.update(status_line=FORMAT_CHANGED.format(converted=totals["converted"]))
@@ -196,7 +189,7 @@ class ThumbnailConvertMixin:
                totals["converted"],
                f"Cancelled · {_plural(totals['converted'], 'preview')} converted",
            )
-        line = _finished_line(totals, plan["remaining"])
+        line = _finished_line(totals, left)
        job.update(progress=100, status_line=line)
        job.log_line(f"Conversion finished · {line}")
        self.logger.info(f"Thumbnail conversion: {line}")
@@ -210,18 +203,59 @@ class ThumbnailConvertMixin:
        )
        return "cancelled"

-    async def _convert_targets(self, job, options, targets, output_dirs):
-        """Convert *targets* in worker batches of preview groups, recording
-        each file."""
-        roots = _scan_roots(output_dirs)
-        total = len(targets)
-        groups = _group_targets(targets)
-        job.update(total=total, status_line=_scan_status_line(0, total, 0, 0))
+    async def _plan_conversion(self, job, options, output_dirs, exclude=None):
+        """One batch's plan (at most MAX_THUMBNAIL_FILES targets), off-loop."""
+        return await self._run_in_executor(
+            self._plan_gallery_thumbnails,
+            output_dirs,
+            "convert",
+            stop=lambda: job.cancelled,
+            settings=options.settings,
+            exclude=exclude,
+        )
+
+    async def _convert_in_batches(self, job, options, plan, output_dirs):
+        """Convert ``plan``, then plan again after each batch until nothing
+        is left, the job is cancelled or stopped, or a batch converted
+        nothing (files that fail are never planned again). Returns
+        ``(totals across batches, files left)``."""
        totals = {"converted": 0, "failed": 0, "before": 0, "after": 0}
        totals["stopped"] = False
-        start = time.monotonic()
+        run = {
+            "index": 0,
+            "batches": 0,
+            "total": len(plan["targets"]) + plan["remaining"],
+        }
+        run["start"] = time.monotonic()
+        job.update(
+            total=run["total"], status_line=_scan_status_line(0, run["total"], 0, 0)
+        )
+        attempted = set()
+        while True:
+            before = totals["converted"]
+            await self._convert_targets(
+                job, options, plan["targets"], output_dirs, totals, run
+            )
+            attempted.update(src for src, _dst, _video in plan["targets"])
+            stop = job.cancelled or totals["stopped"]
+            if stop or not plan["remaining"] or totals["converted"] == before:
+                return totals, plan["remaining"]
+            plan = await self._plan_conversion(
+                job, options, output_dirs, exclude=attempted
+            )
+            if job.cancelled:
+                return totals, 0
+            run["total"] = run["index"] + len(plan["targets"]) + plan["remaining"]
+            job.update(total=run["total"])
+            if not plan["targets"]:
+                return totals, plan["remaining"]
+
+    async def _convert_targets(self, job, options, targets, output_dirs, totals, run):
+        """Convert one batch's *targets* in worker batches of preview groups,
+        recording each file against the job's overall ``run`` counters."""
+        roots = _scan_roots(output_dirs)
+        groups = _group_targets(targets)
        batch_size = _thumbnail_batch_size(_worker_threads())
-        index = batches = 0
        for batch_start in range(0, len(groups), batch_size):
            if job.cancelled or totals["stopped"]:
                break
@@ -235,17 +269,20 @@ class ThumbnailConvertMixin:
                        totals["stopped"] = True
                    if result["action"] in ("cancelled", "stopped"):
                        continue
-                    index += 1
+                    run["index"] += 1
+                    index, total = run["index"], max(run["total"], run["index"])
                    self._record_conversion(job, src, result, roots, totals)
                    job.update(processed=index, progress=index * 100 // total)
-            batches += 1
+            run["batches"] += 1
            job.update(
                status_line=_scan_status_line(
-                    index, total, batches, time.monotonic() - start
+                    run["index"],
+                    run["total"],
+                    run["batches"],
+                    time.monotonic() - run["start"],
                )
            )
            await asyncio.sleep(0)  # let other requests run between batches
-        return totals

    @staticmethod
    def _record_conversion(job, src, result, roots, totals):
@@ -253,42 +253,84 @@ class ThumbnailJobsMixin:
                {"phase": "scanning", "message": "Scanning for images and videos..."},
            )
        )
-        plan = await self._run_in_executor(
-            self._plan_gallery_thumbnails,
-            output_dirs,
-            options.mode,
-            stop=lambda: job.cancelled,
-            settings=options.settings,
-        )
+        plan = await self._plan_thumbnail_batch(job, options, output_dirs)
        if job.cancelled:  # M2: stopped during the walk, before any start frame
            return self._end_cancelled(job, 0, CANCELLED_BEFORE_WRITING)
        roots = _scan_roots(output_dirs)
        self._record_fresh(job, plan, roots)
-        stats, start = await self._generate_thumbnail_targets(job, options, plan, roots)
+        stats, start, left, attempted = await self._generate_in_batches(
+            job, options, plan, roots, output_dirs
+        )
        self.invalidate_gallery_cache()
        generated = _plural(stats["generated"], "preview")
        if job.cancelled:
            return self._end_cancelled(
                job, stats["generated"], f"Cancelled · {generated} written"
            )
+        overall = {"targets": attempted, "skipped": plan["skipped"], "remaining": left}
        payload = _thumbnail_complete_payload(
-            len(plan["targets"]) + plan["skipped"] + plan["remaining"],
+            len(attempted) + plan["skipped"] + left,
            stats,
            start,
-            remaining=plan["remaining"],
-            processed=len(plan["targets"]),
+            remaining=left,
+            processed=len(attempted),
        )
-        job.update(progress=100, status_line=_finished_line(plan, stats))
-        if plan["remaining"]:
-            job.log_line(
-                f"{plan['remaining']:,} more previews are left for the next run"
-            )
+        job.update(progress=100, status_line=_finished_line(overall, stats))
+        if left:
+            job.log_line(f"{left:,} more previews are left for the next run")
        errors = _plural(len(stats["errors"]), "error")
        job.log_line(f"Thumbnails finished · {generated} written · {errors}")
        self.logger.info(payload["message"])
        job.publish(_frame("complete", payload))
        return "finished"

+    async def _plan_thumbnail_batch(self, job, options, output_dirs, exclude=None):
+        """One batch's plan (at most MAX_THUMBNAIL_FILES targets), off-loop."""
+        return await self._run_in_executor(
+            self._plan_gallery_thumbnails,
+            output_dirs,
+            options.mode,
+            stop=lambda: job.cancelled,
+            settings=options.settings,
+            exclude=exclude,
+        )
+
+    async def _generate_in_batches(self, job, options, plan, roots, output_dirs):
+        """Work through ``plan``, then plan again after each batch until
+        nothing is left, the job is cancelled, or a batch made no progress
+        (every file failed: they are not planned again, and the rest waits
+        for the next run). Files worked on are never planned twice. Returns
+        ``(stats, start, files left, files worked on)``; progress counts
+        against the overall total."""
+        run = {
+            "index": 0,
+            "batches": 0,
+            "total": len(plan["targets"]) + plan["remaining"],
+        }
+        stats = _new_thumbnail_stats()
+        stats["skipped"] = plan["skipped"]
+        start = time.monotonic()
+        self._publish_thumbnail_start(job, plan, run["total"])
+        attempted = set()
+        while True:
+            before = stats["generated"] + stats["skipped"]
+            await self._generate_thumbnail_targets(
+                job, options, plan, roots, stats, run, start
+            )
+            attempted.update(src for src, _dst, _video in plan["targets"])
+            progressed = stats["generated"] + stats["skipped"] > before
+            if job.cancelled or not plan["remaining"] or not progressed:
+                return stats, start, plan["remaining"], attempted
+            plan = await self._plan_thumbnail_batch(
+                job, options, output_dirs, exclude=attempted
+            )
+            if job.cancelled:
+                return stats, start, 0, attempted
+            run["total"] = run["index"] + len(plan["targets"]) + plan["remaining"]
+            job.update(total=run["total"])
+            if not plan["targets"]:
+                return stats, start, plan["remaining"], attempted
+
    @staticmethod
    def _end_cancelled(job, generated, status_line):
        """Close a cancelled job: its status line, log line and error frame."""
@@ -316,10 +358,10 @@ class ThumbnailJobsMixin:
        if plan["skipped"]:
            job.log_line(f"= {plan['skipped']:,} previews already up to date")

-    async def _generate_thumbnail_targets(self, job, options, plan, roots):
-        """Generate ``plan``'s targets in worker batches, publishing each file."""
+    @staticmethod
+    def _publish_thumbnail_start(job, plan, total):
+        """The legacy stream's start frame, with the job's overall total."""
        targets = plan["targets"]
-        total = len(targets)
        video_count = sum(1 for _, _, is_video in targets if is_video)
        message = (
            f"Found {total - video_count} images and {video_count} videos to process"
@@ -335,7 +377,7 @@ class ThumbnailJobsMixin:
                {
                    "total_images": total,
                    "skipped": plan["skipped"],
-                    "remaining": plan["remaining"],
+                    "remaining": 0,
                    "phase": "processing",
                    "image_count": total - video_count,
                    "video_count": video_count,
@@ -343,17 +385,19 @@ class ThumbnailJobsMixin:
                },
            )
        )
-        stats = _new_thumbnail_stats()
-        stats["skipped"] = plan["skipped"]
-        start = time.monotonic()
+
+    async def _generate_thumbnail_targets(
+        self, job, options, plan, roots, stats, run, start
+    ):
+        """Generate one batch (``plan``'s targets) in worker batches,
+        publishing each file against the job's overall ``run`` counters."""
+        targets = plan["targets"]
        batch_size = _thumbnail_batch_size(_worker_threads())
        generate = {
            "rebuild": options.rebuild,
            "stop": lambda: job.cancelled,
        }
-        index = 0
-        batches = 0
-        for batch_start in range(0, total, batch_size):
+        for batch_start in range(0, len(targets), batch_size):
            if job.cancelled:
                break
            batch = targets[batch_start : batch_start + batch_size]
@@ -363,7 +407,8 @@ class ThumbnailJobsMixin:
            for (src, _dst, _video), result in zip(batch, results):
                if result["action"] == "cancelled":
                    continue
-                index += 1
+                run["index"] += 1
+                index, total = run["index"], max(run["total"], run["index"])
                _record_thumbnail_result(stats, result)
                self._record_thumbnail_outcome(job, src, result, roots)
                job.update(processed=index, progress=index * 100 // total)
@@ -380,14 +425,13 @@ class ThumbnailJobsMixin:
                        _thumbnail_progress_payload(index, total, stats, result, start),
                    )
                )
-            batches += 1
+            run["batches"] += 1
            job.update(
                status_line=_scan_status_line(
-                    index, total, batches, time.monotonic() - start
+                    run["index"], run["total"], run["batches"], time.monotonic() - start
                )
            )
            await asyncio.sleep(0)  # let other requests run between batches
-        return stats, start

    @staticmethod
    def _record_thumbnail_outcome(job, src, result, roots):
@@ -25,10 +25,10 @@ test('D131: the dialog names the server thumbnail settings instead of its own qu
    }
 });

-test('Convert outdated says how many previews in an older format it converts (#317)', async () => {
+test('Convert outdated says how many previews in an older format it converts, past the per-batch cap too (#317)', async () => {
    const { planText } = await reviewModule;
    assert.equal(planText({ mode: 'convert', targets: 17500, fresh: 2000, outdated: 17500, remaining: 7500 }),
-        '17,500 previews in an older format to convert · 7,500 more on the next run.');
+        '25,000 previews in an older format to convert.');
    assert.equal(planText({ mode: 'convert', targets: 1, fresh: 0, outdated: 1, remaining: 0 }), '1 preview in an older format to convert.');
    assert.equal(planText({ mode: 'convert', targets: 0, fresh: 9, outdated: 0, remaining: 0 }), 'No previews in an older format.');
 });
@@ -39,7 +39,7 @@ test('the plan line counts what a missing-only run generates and what is already
    assert.equal(planText({ mode: 'missing', targets: 1, fresh: 1, remaining: 0 }), '1 preview to generate · 1 already up to date.');
    assert.equal(planText({ mode: 'missing', targets: 0, fresh: 0, remaining: 0 }), '0 previews to generate · 0 already up to date.');
    assert.equal(planText({ mode: 'missing', targets: 10000, fresh: 5, remaining: 2500 }),
-        '10,000 previews to generate · 5 already up to date · 2,500 more on the next run.');
+        '12,500 previews to generate · 5 already up to date.');
 });

 test('Rebuild all says every preview is regenerated (D99)', async () => {
@@ -47,7 +47,7 @@ test('Rebuild all says every preview is regenerated (D99)', async () => {
    assert.equal(planText({ mode: 'rebuild', targets: 42, fresh: 30, remaining: 0 }), 'Every preview is regenerated: 42 files.');
    assert.equal(planText({ mode: 'rebuild', targets: 1, fresh: 1, remaining: 0 }), 'Every preview is regenerated: 1 file.');
    assert.equal(planText({ mode: 'rebuild', targets: 10000, fresh: 9000, remaining: 3 }),
-        'Every preview is regenerated: 10,000 files · 3 more on the next run.');
+        'Every preview is regenerated: 10,003 files.');
 });

 test('an answer without counts is a failed plan, never zeros', async () => {
@@ -497,7 +497,8 @@ class TestThumbnailProgressStream(ThumbnailTestCase):
        self.assertNotIn("_generate_one", calls)
        self.assertIn("_plan_gallery_thumbnails", calls)

-    async def test_capped_run_reports_remaining_and_the_next_run_finishes(self):
+    async def test_a_run_past_the_batch_cap_goes_on_to_the_end(self):
+        # The cap bounds one batch; the run plans again and finishes (#317).
        for name in ("a", "b", "c"):
            make_png(self.output_dir / f"{name}.png", size=(64, 64))

@@ -510,18 +511,19 @@ class TestThumbnailProgressStream(ThumbnailTestCase):
        )

        names = [e for e, _ in first]
-        self.assertEqual(names.count("progress"), 2)  # only pending files are work
+        self.assertEqual(names.count("progress"), 3)  # only pending files are work
+        self.assertEqual(dict(first)["start"]["total_images"], 3)
+        self.assertEqual(
+            [d["total_images"] for e, d in first if e == "progress"], [3] * 3
+        )
        done = dict(first)["complete"]
-        self.assertEqual(done["count"], 2)
-        self.assertEqual(done["remaining"], 1)
-        self.assertEqual(done["total_images"], 3)
-        self.assertIn("1 more", done["message"])
-        done = dict(second)["complete"]
-        self.assertEqual(done["count"], 1)
-        self.assertEqual(done["skipped"], 2)
+        self.assertEqual(done["count"], 3)
        self.assertEqual(done["remaining"], 0)
        self.assertEqual(done["total_images"], 3)
        self.assertNotIn("more", done["message"])
+        done = dict(second)["complete"]
+        self.assertEqual(done["count"], 0)
+        self.assertEqual(done["skipped"], 3)

    async def test_blocking_endpoint_reports_remaining(self):
        for name in ("a", "b", "c"):
@@ -2105,10 +2107,13 @@ class TestThumbnailStartupStatus(ThumbnailTestCase):
        make_png(second / "b.png")
        self.api._get_all_output_dirs = lambda: [self.output_dir, second]
        with patch.object(images_module, "MAX_THUMBNAIL_FILES", 1):
+            plan = self.api._plan_gallery_thumbnails([self.output_dir, second])
            response = await self.client.post(self.PROGRESS_URL)
            data = dict(parse_sse(await response.text()))["complete"]
-        self.assertEqual(data["count"], 1)
-        self.assertEqual(data["remaining"], 1)
+        # One batch holds one file across both roots; the run goes on (#317).
+        self.assertEqual((len(plan["targets"]), plan["remaining"]), (1, 1))
+        self.assertEqual(data["count"], 2)
+        self.assertEqual(data["remaining"], 0)
        self.assertEqual(data["total_images"], 2)

    async def test_equal_timestamps_do_not_regenerate_existing_thumbnails(self):
@@ -410,6 +410,19 @@ class TestConvertOutdated(ConvertCase):
            self.assertTrue(tc._same_file(a, self.output_dir / "." / "a_thumb.jpg"))
        self.assertFalse(tc._same_file(a, self.output_dir / "gone.jpg"))

+    async def test_convert_goes_on_past_the_per_batch_cap(self):
+        import py.api.images as images_module
+
+        olds = [self._legacy(f"c{i}") for i in range(8)]
+        with patch.object(images_module, "MAX_THUMBNAIL_FILES", 3):
+            _, job = await self._convert()
+        self.assertEqual(job.state, "finished")
+        self.assertEqual(job.counters["generated"], 8)
+        self.assertFalse(any(old.exists() for old in olds))
+        self.assertEqual(job.summary()["total"], 8)
+        self.assertRegex(job.status_line, r"^8 converted · 0 failed · ")
+        self.assertNotIn("next run", job.status_line)
+
    def test_byte_sizes_are_readable(self):
        from py.api.thumbnail_convert import readable_bytes

@@ -687,6 +687,113 @@ LEGACY_KEYS = {
 }


+class TestBatchesContinue(ThumbnailJobCase):
+    """MAX_THUMBNAIL_FILES bounds one batch, not the job: after each batch
+    the job plans again and goes on until nothing is left, it is cancelled,
+    or a batch makes no progress (files that fail every time are never
+    planned again)."""
+
+    def _sources(self, count, broken=False):
+        for i in range(count):
+            path = self.output_dir / f"s{i}.png"
+            if broken:
+                path.write_bytes(b"\x89PNG broken")
+            else:
+                make_png(path, size=(32, 32))
+
+    async def test_one_job_makes_every_preview_past_the_cap(self):
+        self._sources(8)
+        with patch.object(images_module, "MAX_THUMBNAIL_FILES", 3):
+            status, body = await self._start(mode="missing")
+            self.assertEqual(status, 202)
+            job = await self._finished(body["job"]["id"])
+        self.assertEqual(job.state, "finished")
+        self.assertEqual(job.counters["generated"], 8)
+        for i in range(8):
+            self.assertTrue(self._thumb_path(f"s{i}").is_file(), i)
+        summary = job.summary()
+        self.assertEqual((summary["total"], summary["processed"]), (8, 8))
+        self.assertEqual(summary["progress"], 100)
+        self.assertEqual(job.status_line, "8 previews written")
+
+    async def test_rebuild_goes_through_every_file_once(self):
+        self._sources(5)
+        calls = []
+        real = self.api._generate_batch
+
+        def counting(batch, *args, **kwargs):
+            calls.extend(src.name for src, _dst, _video in batch)
+            return real(batch, *args, **kwargs)
+
+        self.api._generate_batch = counting
+        with patch.object(images_module, "MAX_THUMBNAIL_FILES", 2):
+            _, body = await self._start(mode="rebuild")
+            job = await self._finished(body["job"]["id"])
+        self.assertEqual(sorted(calls), [f"s{i}.png" for i in range(5)])
+        self.assertEqual(job.counters["generated"], 5)
+
+    async def test_cancel_between_batches_stops_the_job(self):
+        self._sources(8)
+        real = self.api._plan_gallery_thumbnails
+        plans = []
+        job_ids = []
+
+        def cancelling(*args, **kwargs):
+            plans.append(1)
+            if len(plans) == 2:
+                self.assertTrue(self.api._jobs.cancel(job_ids[0]))
+            return real(*args, **kwargs)
+
+        self.api._plan_gallery_thumbnails = cancelling
+        with patch.object(images_module, "MAX_THUMBNAIL_FILES", 3):
+            _, body = await self._start(mode="missing")
+            job_ids.append(body["job"]["id"])
+            job = await self._finished(job_ids[0])
+        self.assertEqual(job.state, "cancelled")
+        self.assertEqual(job.counters["generated"], 3)
+        self.assertEqual(len(plans), 2)
+
+    async def test_a_batch_where_everything_fails_stops(self):
+        self._sources(8, broken=True)
+        plans = []
+        real = self.api._plan_gallery_thumbnails
+
+        def counting(*args, **kwargs):
+            plans.append(1)
+            return real(*args, **kwargs)
+
+        self.api._plan_gallery_thumbnails = counting
+        with patch.object(images_module, "MAX_THUMBNAIL_FILES", 3):
+            _, body = await self._start(mode="missing")
+            job = await asyncio.wait_for(self._finished(body["job"]["id"]), 10)
+        self.assertEqual(job.state, "finished")
+        self.assertEqual(len(plans), 1, "no second plan after a batch with no progress")
+        self.assertEqual(job.counters["errors"], 3)
+        self.assertIn("5 more on the next run", job.status_line)
+
+    async def test_failed_files_are_never_planned_again(self):
+        # One bad file per batch still lets the job go on, without retrying it.
+        for i in range(6):
+            path = self.output_dir / f"m{i}.png"
+            if i in (0, 3):
+                path.write_bytes(b"\x89PNG broken")
+            else:
+                make_png(path, size=(32, 32))
+        seen = []
+        real = self.api._generate_batch
+
+        def counting(batch, *args, **kwargs):
+            seen.extend(src.name for src, _dst, _video in batch)
+            return real(batch, *args, **kwargs)
+
+        self.api._generate_batch = counting
+        with patch.object(images_module, "MAX_THUMBNAIL_FILES", 2):
+            _, body = await self._start(mode="missing")
+            job = await self._finished(body["job"]["id"])
+        self.assertEqual(sorted(seen), sorted(f"m{i}.png" for i in range(6)))
+        self.assertEqual((job.counters["generated"], job.counters["errors"]), (4, 2))
+
+
 class TestLegacyStreamOnTheJob(ThumbnailJobCase):
    async def test_legacy_progress_events_unchanged_on_success(self):
        make_png(self.output_dir / "a.png", size=(400, 400))
@@ -11,8 +11,8 @@
 *   ("Previews are written as WebP · quality 80 · 512 px. …").
 * - D99: a `p.small[role=status]` line before the actions says what the server's plan counts for the chosen scope
 *   (`GET /images/thumbnails/plan?mode=`, read-only): "N previews to generate · M already up to date.", for a rebuild
- *   "Every preview is regenerated: N files.", for a conversion "N previews in an older format to convert.", plus
- *   "· R more on the next run" past the per-run cap. Each scope is read once per dialog. The primary is "Start" (the
+ *   "Every preview is regenerated: N files.", for a conversion "N previews in an older format to convert.". N counts
+ *   the files past the per-batch cap too: the job plans again after each batch (#317). Each scope is read once per dialog. The primary is "Start" (the
 *   concept's "Simulate thumbnail job"): it closes the dialog and starts the job through the Activity store, which
 *   toasts that it started; the job opens in Activity, where its progress lives. A start refused because another job
 *   holds the slot toasts the D57 busy link toast and stays.
@@ -50,13 +50,14 @@ export function planText(plan) {
    if (!plan || !MODES.has(plan.mode) || ![plan.targets, plan.fresh, plan.remaining].every(isCount)) {
        return PLAN_FAILED;
    }
-    const more = plan.remaining ? ` · ${count(plan.remaining)} more on the next run` : '';
-    if (plan.mode === 'rebuild') return `Every preview is regenerated: ${count(plan.targets)} ${noun(plan.targets, 'file')}${more}.`;
+    // #317: the cap bounds one batch, not the job, which plans again until nothing is left: the line counts it all.
+    const work = plan.targets + plan.remaining;
+    if (plan.mode === 'rebuild') return `Every preview is regenerated: ${count(work)} ${noun(work, 'file')}.`;
    if (plan.mode === 'convert') {
-        if (!plan.targets && !plan.remaining) return 'No previews in an older format.';
-        return `${count(plan.targets)} ${noun(plan.targets, 'preview')} in an older format to convert${more}.`;
+        if (!work) return 'No previews in an older format.';
+        return `${count(work)} ${noun(work, 'preview')} in an older format to convert.`;
    }
-    return `${count(plan.targets)} ${noun(plan.targets, 'preview')} to generate · ${count(plan.fresh)} already up to date${more}.`;
+    return `${count(work)} ${noun(work, 'preview')} to generate · ${count(plan.fresh)} already up to date.`;
 }

 /**