Video input: on LTX a video picked as the scene start feeds its frames, not just the first one

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzJLJWVu7RZnbRqm3nYAnX

CHANGELOG.md

@@ -4,7 +4,7 @@
 
 ### Added (experimental)
 
-- **Video input** (Project and Scene tabs): reference videos from the Media bin can be got ready before the model sees them, like a video loader node: target width and height (fill, fit with bars, or stretch; rounded to a multiple), edge crop in percent, frame rate (project's, the video's own or a number), skip first N frames, cut last N, every Nth frame, frame cap. The sound is cut to match. One set per project and one per scene; a dropdown in Project settings says whose is used (Automatic: the project's when switched on, otherwise the scene's). Off by default, so existing projects load videos as before. Works for any model whose pipeline takes reference videos (MiniMax H3 today). Tested on made-up frames and an encoded file, not yet on a rental.
+- **Video input** (Project and Scene tabs): reference videos (and, on LTX, a video picked as the scene start, fed as frames) from the Media bin can be got ready before the model sees them, like a video loader node: target width and height (fill, fit with bars, or stretch; rounded to a multiple), edge crop in percent, frame rate (project's, the video's own or a number), skip first N frames, cut last N, every Nth frame, frame cap. The sound is cut to match. One set per project and one per scene; a dropdown in Project settings says whose is used (Automatic: the project's when switched on, otherwise the scene's). Off by default, so existing projects load videos as before. Works for any model whose pipeline takes reference videos (MiniMax H3 today). Tested on made-up frames and an encoded file, not yet on a rental.
 - **Attention explorer** (needs a Taste key): each run tries a small attention change (sharper, as is or looser) per depth group of blocks, and your ratings decide which changes stay; how far a change goes learns from ratings too (or type your own). The Log names the exact blocks made sharper or looser. Also learns, per run: how strongly each group attends to your prompt ("Prompt focus"), and the clip's four time bands and its sound separately ("Clip regions"). A rating counts more for runs of a similar prompt, and a combination you liked on this prompt is reused (with some groups re-drawn). Zero extra model passes. Untested on a real H3.
 - **Noise merging** (Settings > Engine > Sampling): "Merge" can also run a whole second video beside yours from the second seed ("Second run": one extra pass at every step up to the last one named, blended at the steps you name), or mix the run's starting noise with a second seed's (no extra cost, variance kept). Or, at the steps you name, the step is answered a second time on the same picture with different noise (a second seed), and the two pictures are blended at your strength. Merge order picks which one is the base. The second seed is random each run unless you type one. Picture and sound are both merged (the sound can be left out with "Merge sound too"). "Second pass new noise" sets how much of the step's own noise the second pass swaps for the second seed's (1 all, lower keeps more of your own). Costs one extra model pass per merged step. Untested on a real H3.
 - **SLA rising sparsity** (diffusion model loader, `sla_sparsity_start`): skip fewer attention blocks on early steps and more on late ones, in a straight line up to `sla_sparsity`. Off at 0. One real H3 render (4 steps, 0.70 to 0.90): quality held, perhaps slightly better. Idea from GSRS (arXiv 2610.21129); only the schedule is taken, not its other parts.

app/ui/features/generate/inputs.js

@@ -20,6 +20,17 @@ export async function buildInputs({ project, scene, slots, expand, nodeInfo, fra
   for (const slot of (slots || []).filter((s) => s.node === "FunUIShotCamera")) put(slot, "project", project.id || "");      // what the node keeps per project (the words focus infusion draws on), also for pipelines saved before it was asked for
   const source = rolesAt(slots, "assets.source_image")[0];
   if (source) put(source.slot, "media_id", project.generation_mode === "t2v" ? "" : scene.source_image || "");    // "from a prompt" starts every shot without a picture
+  // A pipeline that can take a whole video where it takes a picture (LTX) says so on the role: a picked VIDEO then
+  // supplies its frames (prepared as Video input says) to everything the picture fed, instead of only its first frame.
+  const startId = project.generation_mode === "t2v" ? "" : scene.source_image;
+  if (source && source.role.video_frames && startId && kinds.get(startId) === "video") {
+    put(source.slot, "full", true);
+    const use = activeVideoInput(project, scene);
+    if (use) put(source.slot, "prepare", videoInputJson(use, project, scene));
+    for (const s of slots || []) for (const [name, v] of Object.entries(s.inputs || {})) {
+      if (Array.isArray(v) && v[0] === source.slot.id && v[1] === 0) put(s, name, [source.slot.id, 1]);
+    }
+  }
   // A project saved before the preset said so still has this shape: a picture-resizing slot is enough to know a stand-in fits.
   const resizer = (slots || []).find((s) => s.node === "ImageTransformKJ"), enhancer = (slots || []).find((s) => s.node === "FunUIEnhancePrompt");
   const like = source && (source.role.blank_like || (resizer && { slot: resizer.id, width: "target_width", height: "target_height", ...(enhancer ? { unsend: { slot: enhancer.id, input: "use_image" } } : {}) }));

app/ui/features/generate/inputs.test.js

@@ -143,3 +143,19 @@ test("a reference video is prepared as the project says, else as the scene says,
   assert.equal((await run({ video_input_from: "project", video_input: { on: false } }, scene)).prepare, undefined);
   learn([]);
 });
+
+test("a video picked as the start feeds its frames to what the picture fed, when the pipeline says it can", async () => {
+  const { learn } = await import("../../shell/bin.js");
+  learn([{ id: "vid", name: "v.mp4", kind: "video" }, { id: "pic", name: "p.png", kind: "image" }]);
+  const mk = (flag) => [{ id: "src", node: "FunUILoadMedia", roles: [{ at: "assets.source_image", input: "media_id", ...(flag ? { video_frames: true } : {}) }], inputs: {} },
+    { id: "anchor", node: "X", inputs: { image: ["src", 0], other: ["src", 1] } }];
+  const run = (slots, id, project = {}) => buildInputs({ project: { video: {}, frame_rate: 25, ...project }, scene: { text: "x", source_image: id }, slots, expand: async () => null });
+  const on = await run(mk(true), "vid", { video_input: { on: true, skip_first: 3 } });
+  assert.equal(on.inputs.src.full, true);
+  assert.equal(JSON.parse(on.inputs.src.prepare).skip_first, 3);
+  assert.deepEqual(on.inputs.anchor.image, ["src", 1]);
+  assert.equal((await run(mk(true), "vid")).inputs.src.prepare, undefined);          // no preparation switched on: whole video as before
+  assert.equal((await run(mk(false), "vid")).inputs.src.full, undefined);          // the pipeline did not say: first frame only
+  assert.equal((await run(mk(true), "pic")).inputs.src.full, undefined);          // a picture stays a picture
+  learn([]);
+});

modules/models/ltx/pipeline.py

@@ -47,7 +47,8 @@ def _base(image_to_video: bool, as_guide: bool = False):
     if image_to_video:
         slots += [
             {"id": "source_image", "group": "Reference media", "node": "FunUILoadMedia",
-             "roles": [{"at": "assets.source_image", "input": "media_id"}],
+             # video_frames: a video picked as the start is fed whole (prepared as Video input says), not as its first frame.
+             "roles": [{"at": "assets.source_image", "input": "media_id", "video_frames": True}],
              "inputs": {"media_id": ""}},
             {"id": "split", "group": "Preparation", "node": "LTXVSeparateAVLatent",
              "inputs": {"av_latent": ["latent", 0]}},