[bugfix]: coalesce in-flight arrivals when the batching deadline has expired (#1453)

When batching_delay_ms > 0, the collect deadline anchors to the first
job's enqueue time. Any job that waited behind an in-flight generation
arrives with an already-expired deadline and was dispatched as a batch
of 1, degrading dynamic batching to sequential under sustained load.
Fall through to the same non-blocking greedy drain the delay=0 path
uses so queued compatible jobs still coalesce up to batching_max_size.
This commit is contained in:
SolitaryThinker
2026-07-11 18:35:45 -07:00
parent cdd0f7f48f
commit dcd2cba956
+6 -6
View File
@@ -89,17 +89,17 @@ class VideoBatchScheduler:
deadline = first.enqueue_time + delay_s
while len(batch) < max_size:
if delay_s > 0:
timeout = deadline - time.perf_counter()
if timeout <= 0:
break
timeout = deadline - time.perf_counter()
if timeout > 0:
try:
candidate = await asyncio.wait_for(self._get_next_job(), timeout=timeout)
except TimeoutError:
break
else:
# delay=0 means "don't wait": greedily drain whatever is
# already queued so max_size still coalesces.
# delay=0 means "don't wait", and an already-expired deadline
# means the first job waited behind an in-flight generation:
# in both cases greedily drain whatever is already queued so
# max_size still coalesces instead of dispatching batches of 1.
if self._pending:
candidate = self._pending.popleft()
else: