Handle predecoded video in Modal LLM

docs/LEARNINGS.md

@@ -223,6 +223,7 @@
 - Prompt completion also should not wait for every seeded lane task to return. If faster lanes already drained the mapped item queue, the scheduler can return immediately, clean up only the lanes that actually seeded, and let any extra slow lane finish booting in the background so it becomes a warm worker for the next prompt instead of blocking the current one.
 - Modal snapshot identity should follow immutable cold-start state, not scheduler lane identity. If lane affinity is folded into the `@app.cls` parameters, Modal builds and caches one GPU snapshot per slot even when every lane loads the exact same ComfyUI runtime and model profile. Keeping only `snapshot_profile_key` and GPU-snapshot mode in the class parameters lets all lanes share the same expensive cold-start snapshot while lane assignment stays a local scheduling concern.
 - Deployed Modal class behavior that matters at `snap=True` should not be inferred from environment inside the remote container. The container does not automatically inherit the local ComfyUI launch env, so flags like GPU-snapshot mode need to be carried as deploy-time/class parameters if the snapshot hook logic depends on them.
+- Native ComfyUI video is already decoded and uniformly sampled before it reaches a multimodal processor. Passing that frame list to Transformers without `do_sample_frames=False` makes SmolVLM try to sample it again and fail because an in-memory frame list has no source-video metadata.
 - A remote proxy cannot treat every incoming list literal as just another scalar boundary input. Ordinary remote components also have to preserve ComfyUI's implicit zipped batch semantics across the boundary, or primitive widget sockets like `seed: INT` end up receiving whole lists such as `[10, 11, 12, 13]` instead of per-item values.
 - That implicit batching path also needs the same hybrid static-vs-mapped split as explicit `ModalMapInput`. If two remote execute nodes share a non-transportable upstream object like `MODEL` but only one of them depends on the multi-item boundary input, the sibling execute node still has to run once, not once per batch item.
 - Suppressed per-item progress streams should not overwrite `display_node_id` with the component representative. Lane ids are enough to separate concurrent runs, so progress bars can stay attached to the actual executing node instead of collapsing onto the first node in the remote island.

modal_llm_runtime.py

@@ -368,6 +368,26 @@ def _stopping_criteria(progress_callback: Callable[[int], None]) -> Any:
     return StoppingCriteriaList([ComfyProgressStoppingCriteria()])
 
 
+def _apply_multimodal_chat_template(
+    processor: Any,
+    messages: list[dict[str, Any]],
+    *,
+    has_predecoded_video: bool,
+) -> Any:
+    """Tokenize chat content without resampling video frames prepared by ComfyUI."""
+    processor_kwargs: dict[str, Any] = {}
+    if has_predecoded_video:
+        processor_kwargs["do_sample_frames"] = False
+    return processor.apply_chat_template(
+        messages,
+        add_generation_prompt=True,
+        tokenize=True,
+        return_dict=True,
+        return_tensors="pt",
+        processor_kwargs=processor_kwargs,
+    )
+
+
 class TransformersMultimodalBackend:
     """Run a curated image-text-to-text model through Hugging Face Transformers."""
 
@@ -424,12 +444,10 @@ class TransformersMultimodalBackend:
         """Tokenize multimodal messages and generate only the assistant continuation."""
         import torch
 
-        inputs = self.processor.apply_chat_template(
+        inputs = _apply_multimodal_chat_template(
+            self.processor,
             self._messages(prepared_inputs),
-            add_generation_prompt=True,
-            tokenize=True,
-            return_dict=True,
-            return_tensors="pt",
+            has_predecoded_video=prepared_inputs.video is not None,
         )
         inputs = _move_batch_to_device(inputs, "cuda")
         input_ids = inputs.get("input_ids")

tests/test_modal_llm.py

@@ -221,6 +221,36 @@ def test_transformers_stopping_criteria_propagates_cancellation(
     assert calls == [1, 2]
 
 
+def test_transformers_processor_does_not_resample_predecoded_video(
+    modal_llm_runtime_module: Any,
+) -> None:
+    """Native ComfyUI frames should bypass the processor's metadata-based sampler."""
+    calls: list[tuple[list[dict[str, Any]], dict[str, Any]]] = []
+
+    class FakeProcessor:
+        """Record the chat-template call without importing a real model processor."""
+
+        def apply_chat_template(
+            self,
+            messages: list[dict[str, Any]],
+            **kwargs: Any,
+        ) -> str:
+            """Capture processor options and return a tokenization sentinel."""
+            calls.append((messages, kwargs))
+            return "tokenized"
+
+    messages = [{"role": "user", "content": [{"type": "video", "video": [object()]}]}]
+
+    result = modal_llm_runtime_module._apply_multimodal_chat_template(
+        FakeProcessor(),
+        messages,
+        has_predecoded_video=True,
+    )
+
+    assert result == "tokenized"
+    assert calls[0][1]["processor_kwargs"] == {"do_sample_frames": False}
+
+
 @dataclass
 class _FakeBackend:
     """Record resident backend inference and unload behavior."""