Compare commits

...
Author SHA1 Message Date
ausboss cecffcc592 Camera Move: keep moving through keys, steady the depth
The camera stopped at every key: each stretch between two keys eased in
and out on its own. Each value now follows a monotone cubic through the
keys, so it keeps its speed through a key, never runs past one, and only
eases at the first and last key.

The moved picture shook. The depth model guesses the lens again for every
frame, the depth-scale smoothing kept the frame-to-frame wobble instead of
removing it, and still things got a slightly different depth each frame.
Every frame now uses the clip's median lens, each frame's median depth
follows the smoothed line, and the depth of spots that hold still is
averaged over nearby frames. With the camera held at one angle the guide
now moves 0.02 px a frame between frames, against 0.28 before and 0.04 in
the source clip.
2026-11-09 09:41:40 -06:00
ausboss 2edd8c0859 Camera Move: move the camera of your own clip
A new node, Camera Move. Press Show my clip and it shows the clip in 3D on
the node: drag to turn the camera, scroll or pinch to pull back or push in,
and click the time bar to set keys. It turns the clip's depth (core MoGe
Inference) into a guide clip where what the old camera never saw is pink,
for MiniMax H3 with the CrossView-Warp LoRA to paint in.

The face also works on a subgraph box: once the camera input is brought up
to the box, the box shows the shot, its settings card and the LoRA rows.

Adds the MiniMax H3 Camera Move example (Load Video, one box, Save Video),
the help page, the Chinese names, tests for the camera maths, and the
panel and wheel guards for the new face.
2026-11-07 17:45:19 -06:00
19 changed files with 5459 additions and 1 deletions
+7
View File
@@ -4,6 +4,13 @@ All notable changes to ComfyUI-AusBoss are documented here.
## Unreleased
- **New node: Camera Move 🆎.** Moves the camera of your own clip. Drag the
picture on the node to turn the camera, pull back or push in, and set keys
on a time bar. What the old camera never saw comes out pink, ready for
MiniMax H3 with the CrossView-Warp LoRA to paint in. The new MiniMax H3
Camera Move example puts it all in one box between Load Video and Save
Video.
- **Mask by Touch 🆎 no longer fills holes unless you ask.** Fill small
holes, under More, now starts off, so a hole you pick stays a hole. Saved
workflows keep their setting.
+6
View File
@@ -167,6 +167,12 @@ Upload a sound file, or pick one from the Source list, and the node shows it as
It returns the `audio` you kept and its length in `seconds`. Only that part is decoded. [Help page](js/docs/AUSBOSS_NODES_LoadAudio.md).
### Camera Move
Moves the camera of your own clip. Press **Show my clip** and the node shows your clip in 3D. Drag the picture to turn the camera, scroll or pinch to pull back or push in, and click the time bar to set keys. What the old camera never saw comes out pink, and MiniMax H3 with Cseti's CrossView-Warp LoRA paints it in. Stay within about 45° left or right, 30° up and 20° down. Your keys add to any move your clip's camera already makes.
It needs depth from the core **MoGe Inference** node and returns the `guide` clip for H3 and a `holes` mask. [Help page](js/docs/AUSBOSS_NODES_CameraMove.md). [Open the MiniMax H3 Camera Move workflow →](example_workflows/MiniMax%20H3%20Camera%20Move%20%28AusBoss%29.json)
### Video Crop + Rotate + Pad → Frame
Picks one frame from a video, then gives you the same rotate, crop and pad controls as the image node, plus an optional megapixel resize. Scrub to the frame with the playhead on the node; the editor adds frame-by-frame stepping and playback. It returns the transformed `image`, a `mask` of the added area, a `stitcher`, the chosen `original` frame, and the output `width` and `height`. For a whole clip, use the Clip version below.
+1
View File
@@ -20,6 +20,7 @@ NODE_MODULES = [
"node_align_image",
"node_batch_utils",
"node_callout",
"node_camera_move",
"node_color_match",
"node_compare",
"node_frame_interpolate",
Binary file not shown.

After

Width:  |  Height:  |  Size: 118 KiB

File diff suppressed because it is too large Load Diff
+92
View File
@@ -0,0 +1,92 @@
// The settings brought up to a Camera Move box, as one compact card above
// the shot: what people change per clip on top (size, seed), the model files
// folded under "Models", since they are set once. A LoRA Loader's rows keep
// their own panel, with its strength bars; the camera keys are the shot.
//
// A box row shares its options with the setting inside the box, and Nodes 2.0
// reads "hidden" from there, so a row the card takes over is hidden inside the
// box too in that renderer (the classic canvas still shows it). Its value is
// set from the box either way.
import { mountWidgetCard } from "../shared/widget_card.mjs";
import { boxRowInputs } from "../shared/lora_box.mjs";
import { hideWidget } from "../shared/widget_visibility.mjs";
const STASH = "__ausbossVisibilityStash"; // widget_visibility.mjs: a row already hidden
const MODEL_FILE = /\.(safetensors|sft|ckpt|pt|pth|bin|gguf|onnx)$/i;
const SUFFIX = { megapixels: "MP" };
const ROW_TYPES = new Set(["number", "slider", "combo", "toggle", "text"]);
const isLoraLoader = (node) => node?.type === "AUSBOSS_NODES_LoraLoader" || node?.comfyClass === "AUSBOSS_NODES_LoraLoader";
const isModelRow = (widget) => {
if (widget?.type !== "combo") return false;
if (MODEL_FILE.test(String(widget.value ?? ""))) return true;
const values = typeof widget.options?.values === "function" ? [] : widget.options?.values;
return Array.isArray(values) && values.some((value) => MODEL_FILE.test(String(value)));
};
function labelOf(box, widget) {
const input = (box.inputs ?? []).find((slot) => slot?.widget?.name === widget.name);
const text = String(input?.label || widget.label || widget.name).replace(/_/g, " ");
return text.charAt(0).toUpperCase() + text.slice(1);
}
// The rows a box's card shows: every setting brought up to it except `skip`
// (the camera keys), a LoRA Loader's rows and anything with its own panel.
export function boxCardRows(box, skip) {
const loraRows = new Set(boxRowInputs(box, isLoraLoader).map((entry) => entry.name));
const top = [], models = [];
for (const widget of box.widgets ?? []) {
if (!widget || widget === skip || widget.element || !ROW_TYPES.has(String(widget.type)) || loraRows.has(widget.name)) continue;
if (widget[STASH] && !widget.__ausbossCameraCard) continue;
const row = { widget: widget.name, label: labelOf(box, widget) };
if (isModelRow(widget)) models.push({ ...row, kind: "select", group: "models" });
else top.push(SUFFIX[innerName(box, widget)] ? { ...row, suffix: SUFFIX[innerName(box, widget)] } : row);
}
return models.length ? [...top, { group: "models", label: `Models · ${models.length} files` }, ...models] : top;
}
// The name of the setting inside the box that a box row stands for.
function innerName(box, widget) {
const input = (box.subgraph?.inputs ?? []).find((slot) => slot?.name === widget.name);
for (const id of input?.linkIds ?? []) {
const link = box.subgraph.getLink?.(id) ?? box.subgraph.links?.get?.(id) ?? box.subgraph.links?.[id];
const node = link && (box.subgraph.getNodeById?.(link.target_id));
const name = node?.inputs?.[link.target_slot]?.widget?.name ?? node?.inputs?.[link.target_slot]?.name;
if (name) return name;
}
return widget.name;
}
// The frontend now and then makes a box's rows afresh (a new object under
// the same name), so the card hides its rows again whenever it finds one
// showing. Cheap: a handful of names.
export function rehideBoxCard(box) {
let changed = false;
for (const name of box.__ausbossCameraCardRows ?? []) {
const widget = (box.widgets ?? []).find((w) => w?.name === name);
if (!widget || widget[STASH]) continue;
hideWidget(widget);
widget.__ausbossCameraCard = true;
changed = true;
}
return changed;
}
// Mount the card once per box. Settings brought up later show as plain rows.
// `tidy` puts the box's panels back in order and says whether it moved one.
export function mountBoxCard(box, skip, tidy) {
if (box.__ausbossCameraCard) { rehideBoxCard(box); return false; }
const rows = boxCardRows(box, skip);
if (!rows.length) return false;
box.__ausbossCameraCardRows = rows.filter((row) => row.widget).map((row) => row.widget);
rehideBoxCard(box);
mountWidgetCard(box, { rows, minWidth: 300, first: true });
box.__ausbossCameraCard = true;
const draw = box.onDrawForeground;
box.onDrawForeground = function (...args) {
if (rehideBoxCard(this) | Boolean(tidy?.(this))) this.setDirtyCanvas?.(true, true);
return draw?.apply(this, args);
};
return true;
}
+183
View File
@@ -0,0 +1,183 @@
// Camera Move: the camera maths the face shares with the node.
//
// The same keys, pose and easing as nodes/_camera_move_helpers.py, so the
// picture on the node is the shot the warp makes. No DOM and no app import:
// tests/camera_move.test.mjs runs it under node:test.
//
// Axes are OpenCV's: x right, y down, z into the picture. A key is
// { t, az, el, back, px, py }: t from 0 to 1 over the clip, turn and rise in
// degrees, back and the pan in distances to the pivot.
export const LIM = { az: [-90, 90], el: [-30, 60], back: [-0.6, 2], px: [-1, 1], py: [-1, 1] };
export const K0 = { az: 0, el: 0, back: 0, px: 0, py: 0 };
export const FIELDS = Object.keys(K0);
// Two keys this close on the clip are the same key.
export const SAME_KEY = 0.015;
export const MAX_KEYS = 64;
export const LORA_RANGE = { az: 45, up: 30, down: -20 };
export const DEFAULT_KEYS = [{ t: 0, ...K0 }, { t: 1, ...K0, az: 30, el: 5 }];
export const clamp = (v, a, b) => Math.max(a, Math.min(b, v));
const num = (v, fallback = 0) => {
const n = Number(v);
return Number.isFinite(n) ? n : fallback;
};
export function cleanKey(raw) {
const key = { t: clamp(num(raw?.t), 0, 1) };
for (const name of FIELDS) key[name] = clamp(num(raw?.[name]), ...LIM[name]);
return key;
}
// The camera a node holds: { keys, pivot }. Unreadable text gives the default move.
export function parseCamera(text) {
let data = null;
if (text && typeof text === "object") data = text;
else if (typeof text === "string" && text.trim()) {
try { data = JSON.parse(text); } catch { data = null; }
}
const fallback = !data || !Array.isArray(data.keys) || !data.keys.length;
if (fallback) data = { keys: DEFAULT_KEYS, pivot: null };
const keys = [];
for (const raw of data.keys.slice(0, MAX_KEYS)) {
const key = cleanKey(raw);
const same = keys.find((k) => Math.abs(k.t - key.t) < SAME_KEY);
if (same) Object.assign(same, key);
else keys.push(key);
}
keys.sort((a, b) => a.t - b.t);
const p = data.pivot;
const pivot = p && typeof p === "object" && "u" in p && "v" in p
? { u: clamp(num(p.u, 0.5), 0, 1), v: clamp(num(p.v, 0.5), 0, 1), t: clamp(num(p.t), 0, 1) }
: null;
return { keys, pivot, fallback };
}
const round = (v, places) => Math.round(v * 10 ** places) / 10 ** places;
export function cameraText({ keys, pivot }) {
const out = {
v: 1,
keys: keys.map((k) => ({ t: round(k.t, 4), az: round(k.az, 2), el: round(k.el, 2), back: round(k.back, 3), px: round(k.px, 3), py: round(k.py, 3) })),
pivot: pivot ? { u: round(pivot.u, 4), v: round(pivot.v, 4), t: round(pivot.t, 4) } : null,
};
return JSON.stringify(out);
}
// How fast one field moves as it passes each key: zero at the first and last
// key (a gentle start and end) and where the field turns back or holds (it
// never runs past a key); elsewhere it keeps its speed through the key, so a
// move made of several keys does not slow down at each one.
function keySlopes(ts, ys) {
const n = ys.length, m = new Array(n).fill(0);
for (let k = 1; k < n - 1; k++) {
const h0 = ts[k] - ts[k - 1], h1 = ts[k + 1] - ts[k];
if (h0 <= 0 || h1 <= 0) continue;
const d0 = (ys[k] - ys[k - 1]) / h0, d1 = (ys[k + 1] - ys[k]) / h1;
if (d0 * d1 > 0) m[k] = 3 * (h0 + h1) / ((2 * h1 + h0) / d0 + (h1 + 2 * h0) / d1);
}
return m;
}
// Where the camera is at moment t: through every key, gently at the ends.
// Same maths as camera_at in nodes/_camera_move_helpers.py.
export function camAt(keys, t) {
if (!keys.length) return { ...K0 };
const n = keys.length;
const pick = (k) => { const o = {}; for (const f of FIELDS) o[f] = k[f] || 0; return o; };
if (n === 1 || t <= keys[0].t) return pick(keys[0]);
if (t >= keys[n - 1].t) return pick(keys[n - 1]);
const ts = keys.map((k) => k.t);
for (let i = 0; i < n - 1; i++) {
const a = keys[i], b = keys[i + 1];
if (t >= a.t && t <= b.t) {
const h = Math.max(1e-6, b.t - a.t), s = (t - a.t) / h;
const h00 = 2 * s ** 3 - 3 * s ** 2 + 1, h10 = s ** 3 - 2 * s ** 2 + s, h01 = -2 * s ** 3 + 3 * s ** 2, h11 = s ** 3 - s ** 2;
const o = {};
for (const f of FIELDS) {
const m = keySlopes(ts, keys.map((k) => k[f] || 0));
o[f] = h00 * (a[f] || 0) + h10 * h * m[i] + h01 * (b[f] || 0) + h11 * h * m[i + 1];
}
return o;
}
}
return pick(keys[n - 1]);
}
export const keyAt = (keys, t) => keys.findIndex((k) => Math.abs(k.t - t) < SAME_KEY);
// Moving the camera at the time bar sets the key there, or adds one.
export function setKeyAt(keys, t, cam) {
const c = {};
for (const f of FIELDS) c[f] = clamp(num(cam[f]), ...LIM[f]);
const i = keyAt(keys, t);
if (i >= 0) { Object.assign(keys[i], c); return keys; }
if (keys.length >= MAX_KEYS) return keys;
keys.push({ t: clamp(t, 0, 1), ...c });
keys.sort((a, b) => a.t - b.t);
return keys;
}
export const pastLoraRange = (c) => Math.abs(c.az) > LORA_RANGE.az || c.el > LORA_RANGE.up || c.el < LORA_RANGE.down;
// R = Ry(-az) Rx(-el), row-major.
export function rot(az, el) {
const th = -az * Math.PI / 180, ph = -el * Math.PI / 180;
const ct = Math.cos(th), st = Math.sin(th), cp = Math.cos(ph), sp = Math.sin(ph);
return [ct, st * sp, st * cp, 0, cp, -sp, -st, ct * sp, ct * cp];
}
export const mv = (R, v) => [R[0] * v[0] + R[1] * v[1] + R[2] * v[2], R[3] * v[0] + R[4] * v[1] + R[5] * v[2], R[6] * v[0] + R[7] * v[1] + R[8] * v[2]];
// The moved camera's rotation R and centre C. Pan slides the camera in its own frame.
export function pose(cam, pivot) {
const R = rot(cam.az, cam.el), p = pivot, Rp = mv(R, p), d = Math.hypot(p[0], p[1], p[2]);
const b = mv(R, [(cam.px || 0) * d, (cam.py || 0) * d, -(cam.back || 0) * d]);
return { R, C: [p[0] - Rp[0] + b[0], p[1] - Rp[1] + b[1], p[2] - Rp[2] + b[2]] };
}
// A world point in the moved camera: R^T (P - C).
export function toCam(R, C, P) {
const d = [P[0] - C[0], P[1] - C[1], P[2] - C[2]];
return [R[0] * d[0] + R[3] * d[1] + R[6] * d[2], R[1] * d[0] + R[4] * d[1] + R[7] * d[2], R[2] * d[0] + R[5] * d[1] + R[8] * d[2]];
}
// The node's depth picture: red is the high byte, green the low one, on a log
// scale from zmin to zmax; 0 is no depth (sky), which sits at `far`.
export function unpackDepth(rgba, count, zmin, zmax, far) {
const z = new Float32Array(count), valid = new Uint8Array(count);
const lmin = Math.log(zmin), lr = Math.log(zmax) - lmin;
for (let i = 0; i < count; i++) {
const q = rgba[i * 4] * 256 + rgba[i * 4 + 1];
if (q === 0) z[i] = far;
else { z[i] = Math.exp(lmin + (q - 1) / 65534 * lr); valid[i] = 1; }
}
return { z, valid };
}
// The 3D spot under a pixel of a frame: the middle of the nearest depth found
// within three pixels, as the node does it. Null over sky.
export function pivotFromPixel(frame, u, v) {
const { W, H, z, valid, fx, fy, cx, cy } = frame;
const x = clamp(Math.floor(u * W), 0, W - 1), y = clamp(Math.floor(v * H), 0, H - 1);
for (let r = 0; r <= 3; r++) {
const found = [];
for (let yy = Math.max(0, y - r); yy <= Math.min(H - 1, y + r); yy++) {
for (let xx = Math.max(0, x - r); xx <= Math.min(W - 1, x + r); xx++) if (valid[yy * W + xx]) found.push(z[yy * W + xx]);
}
if (found.length) {
found.sort((a, b) => a - b);
const depth = found.length % 2 ? found[found.length >> 1] : (found[found.length / 2 - 1] + found[found.length / 2]) / 2;
return [(x + 0.5 - cx) / fx * depth, (y + 0.5 - cy) / fy * depth, depth];
}
}
return null;
}
// Words for where the camera is.
export function fmtAz(v) { const r = Math.round(v); return r === 0 ? "0°" : `${Math.abs(r)}° ${r > 0 ? "right" : "left"}`; }
export function fmtEl(v) { const r = Math.round(v); return r === 0 ? "0°" : `${Math.abs(r)}° ${r > 0 ? "up" : "down"}`; }
export function describe(c) {
const part = (v, neg, pos) => (Math.abs(v) < 0.005 ? "" : `${Math.abs(v).toFixed(2)} ${v > 0 ? pos : neg}`);
const pan = [part(c.px, "left", "right"), part(c.py, "up", "down")].filter(Boolean).join(", ");
return `Turn ${fmtAz(c.az)} · Rise ${fmtEl(c.el)} · ${c.back < -0.004 ? "In" : "Back"} ${Math.abs(c.back).toFixed(2)}×${pan ? ` · Pan ${pan}` : ""}`;
}
+900
View File
@@ -0,0 +1,900 @@
// Camera Move's face: the shot on the node, steered by grabbing it.
//
// Drag the picture to turn the camera round, pinch or scroll to pull back,
// two fingers or Shift to pan, tap a spot to circle round it. Top right, as
// in Blender: an axis ball, a magnifier, a hand, and a camera that puts you
// back on your own shot. Under it the time bar with its keys.
//
// No app import: index.js hands in where the camera is kept, how to run the
// node and how to reach a file, so the same face sits on the node or on the
// box that holds it.
//
// The stage is the one part of the node that keeps the mouse wheel (it zooms
// the camera, as in Blender); its root carries no "ausboss-" class, so the
// pack's wheel hand-off leaves it alone. The controls under it do carry one,
// and the wheel there zooms the graph as everywhere else.
import { makeScrubInput } from "../shared/scrub_input.mjs";
import {
K0, LIM, SAME_KEY, camAt, cameraText, clamp, describe, keyAt, mv, parseCamera, pastLoraRange,
pivotFromPixel, pose, rot, setKeyAt, toCam, unpackDepth,
} from "./camera_math.mjs";
export const MIN_WIDTH = 380; // the play row on one line, with "243 / 243" frames
export const MIN_STAGE = 184;
export const CONTROLS_HEIGHT = 168;
const FPS = 24; // H3 renders at 24 frames a second; only playback speed uses it
const MAG = 0xFFFF00FF;
const CSS = `
.acm-face{position:relative;display:flex;flex-direction:column;width:100%;height:100%;box-sizing:border-box;
--acm-teal:#4fb3a9;--acm-teal-dim:#2b605b;--acm-line:#34413f;--acm-fg:#e3e9e8;--acm-muted:#92a2a1;--acm-amber:#e2a94c;
--acm-stage:#0c0f10;--acm-panel:#1f2629;--acm-head:#283236;color:var(--acm-fg);font:12.5px system-ui,-apple-system,"Segoe UI",Roboto,sans-serif}
.acm-face *{box-sizing:border-box}
.acm-stage{position:relative;flex:1 1 auto;min-height:${MIN_STAGE}px;background:var(--acm-stage);border-radius:6px;overflow:hidden;
touch-action:none;user-select:none;-webkit-user-select:none}
.acm-stage>canvas.acm-pic{position:absolute;display:block;cursor:grab}
.acm-stage>canvas.acm-pic:active{cursor:grabbing}
.acm-pivot{position:absolute;width:18px;height:18px;margin:-9px 0 0 -9px;border:2px solid var(--acm-teal);border-radius:50%;pointer-events:none;box-shadow:0 0 0 1px rgba(0,0,0,.6)}
.acm-pivot::after{content:"";position:absolute;left:50%;top:50%;width:3px;height:3px;margin:-1.5px;background:var(--acm-teal);border-radius:50%}
.acm-nav{position:absolute;top:6px;right:6px;display:flex;flex-direction:column;align-items:center;gap:5px}
.acm-nav canvas{width:64px;height:64px;cursor:grab;touch-action:none;margin-bottom:1px}
.acm-nav button,.acm-helpbtn{width:30px;height:30px;border-radius:50%;border:1px solid rgba(146,162,161,.4);background:rgba(12,15,16,.62);
color:var(--acm-fg);padding:0;display:grid;place-items:center;touch-action:none;box-shadow:0 1px 6px rgba(0,0,0,.45);cursor:pointer}
.acm-nav button svg{width:18px;height:18px;fill:none;stroke:currentColor;stroke-width:1.8;stroke-linecap:round;stroke-linejoin:round}
.acm-nav button:hover,.acm-helpbtn:hover{border-color:var(--acm-teal)}
.acm-nav button.on,.acm-helpbtn[aria-expanded="true"]{background:var(--acm-teal-dim);border-color:var(--acm-teal)}
.acm-nav .pan{cursor:move}.acm-nav .zoom{cursor:ns-resize}
.acm-helpbtn{position:absolute;top:6px;left:6px;width:24px;height:24px;font:600 12px system-ui,sans-serif}
.acm-inset{position:absolute;left:6px;bottom:6px;width:36%;border:1px solid var(--acm-line);border-radius:5px;overflow:hidden;background:var(--acm-stage);
box-shadow:0 2px 10px rgba(0,0,0,.5);pointer-events:none}
.acm-inset canvas{display:block;width:100%;height:auto}
.acm-tag{position:absolute;top:4px;left:5px;font-size:10px;color:var(--acm-muted);background:rgba(12,15,16,.7);padding:1px 5px;border-radius:4px;pointer-events:none}
.acm-empty{position:absolute;inset:0;display:flex;flex-direction:column;align-items:center;justify-content:center;gap:8px;padding:16px;text-align:center;color:var(--acm-muted);line-height:1.4}
.acm-empty b{color:var(--acm-fg);font-weight:600;font-size:13px}
.ausboss-camera-move-controls{flex:0 0 auto;display:flex;flex-direction:column;gap:6px;padding:7px 0 2px}
.acm-readout{display:flex;justify-content:space-between;gap:8px;font-size:11.5px;color:var(--acm-fg);font-variant-numeric:tabular-nums;min-height:15px}
.acm-readout .acm-where{flex:1 1 auto;min-width:0;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}
.acm-readout .acm-pink{color:var(--acm-muted);flex:0 0 auto}
.acm-warn{color:var(--acm-amber);font-size:11.5px;line-height:15px;height:15px;white-space:nowrap;overflow:hidden;text-overflow:ellipsis}
.acm-back{display:flex;align-items:center;gap:8px}
.acm-back>span{width:40px;color:var(--acm-muted)}
.acm-track{position:relative;height:34px;background:var(--acm-stage);border:1px solid var(--acm-line);border-radius:6px;touch-action:none;cursor:pointer}
.acm-track .acm-fill{position:absolute;top:15px;height:2px;left:8px;right:8px;background:var(--acm-line)}
.acm-track .acm-inner{position:absolute;left:8px;right:8px;top:0;bottom:0}
.acm-track .acm-key{position:absolute;top:10px;width:12px;height:12px;margin-left:-6px;transform:rotate(45deg);background:var(--acm-teal);border:1px solid #0c0f10;cursor:ew-resize}
.acm-track .acm-key.on{background:#bfe9e4}
.acm-track .acm-head{position:absolute;top:2px;bottom:2px;width:2px;margin-left:-1px;background:var(--acm-fg);pointer-events:none}
.acm-track .acm-head::before{content:"";position:absolute;top:-2px;left:-4px;width:10px;height:6px;background:var(--acm-fg);border-radius:2px}
.acm-row{display:flex;align-items:center;gap:6px}
.acm-row>.acm-keynav,.acm-row>.acm-btn{flex:none;white-space:nowrap}
.acm-btn{border:1px solid var(--acm-line);background:var(--acm-head);color:var(--acm-fg);font:inherit;font-size:12.5px;padding:6px 10px;border-radius:6px;cursor:pointer;line-height:1.1}
.acm-btn:disabled{opacity:.4;cursor:default}
.acm-btn.primary{background:var(--acm-teal-dim);border-color:var(--acm-teal-dim)}
.acm-keynav{display:inline-flex;border:1px solid var(--acm-line);border-radius:6px;background:var(--acm-head);overflow:hidden}
.acm-keynav button{width:32px;height:29px;border:0;background:transparent;color:var(--acm-fg);padding:0;display:grid;place-items:center;cursor:pointer}
.acm-keynav button+button{border-left:1px solid var(--acm-line)}
.acm-keynav button:disabled{color:var(--acm-muted);opacity:.35;cursor:default}
.acm-keynav button:not(:disabled):hover{background:var(--acm-panel)}
.acm-keynav svg{width:14px;height:14px}
.acm-keynav .dia svg{width:16px;height:16px;fill:none;stroke:currentColor;stroke-width:1.6}
.acm-keynav .dia.on svg{fill:#bfe9e4;stroke:#bfe9e4}
.acm-keynav .arr svg{fill:currentColor}
.acm-keynav.transport .play{width:36px;background:var(--acm-teal-dim);font-size:12px}
.acm-keynav.transport .play:hover{background:var(--acm-teal)}
.acm-keynav.transport .loop{color:var(--acm-muted)}
.acm-keynav.transport .loop svg{width:16px;height:16px;fill:none;stroke:currentColor;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round}
.acm-keynav.transport .loop[aria-pressed="true"]{color:#bfe9e4;background:var(--acm-teal-dim)}
.acm-time{margin-left:auto;font-size:11.5px;color:var(--acm-muted);font-variant-numeric:tabular-nums;white-space:nowrap}
.acm-help{position:absolute;top:36px;left:6px;right:6px;z-index:5;background:#172023;border:1px solid var(--acm-line);border-radius:8px;
padding:10px 12px 12px;box-shadow:0 8px 28px rgba(0,0,0,.55);max-height:calc(100% - 44px);overflow:auto}
.acm-help-h{display:flex;justify-content:space-between;align-items:center;font-weight:600;font-size:13px;margin-bottom:6px}
.acm-help-x{border:0;background:transparent;color:var(--acm-muted);font-size:20px;line-height:1;cursor:pointer;padding:0 2px}
.acm-help dl{margin:0;display:grid;grid-template-columns:auto 1fr;gap:7px 12px;font-size:12.5px;line-height:1.4}
.acm-help dt{color:var(--acm-teal);font-weight:600}.acm-help dd{margin:0}
.acm-nav,.acm-keynav,.acm-btn,.acm-helpbtn{-webkit-touch-callout:none;-webkit-user-select:none;user-select:none}
.acm-tip{position:fixed;z-index:10000;max-width:250px;background:#0a0d0e;color:#e3e9e8;border:1px solid #34413f;border-radius:6px;
padding:5px 8px;font:12px/1.35 system-ui,sans-serif;pointer-events:none;box-shadow:0 4px 14px rgba(0,0,0,.5)}
.acm-face [hidden],.acm-tip[hidden]{display:none!important}
`;
function ensureCss() {
if (document.getElementById("ausboss-camera-move-css")) return;
const style = document.createElement("style");
style.id = "ausboss-camera-move-css";
style.textContent = CSS;
document.head.append(style);
}
const el = (tag, cls, attrs = {}) => {
const e = document.createElement(tag);
if (cls) e.className = cls;
for (const [k, v] of Object.entries(attrs)) {
if (k === "text") e.textContent = v;
else if (k === "html") e.innerHTML = v;
else e.setAttribute(k, v);
}
return e;
};
const NAV_ICONS = {
zoom: '<svg viewBox="0 0 24 24"><circle cx="10.5" cy="10.5" r="6"/><path d="M15 15l5 5"/></svg>',
pan: '<svg viewBox="0 0 24 24"><path d="M7.5 12.5V6.3a1.25 1.25 0 0 1 2.5 0v5M10 11.3V4.8a1.25 1.25 0 0 1 2.5 0v6.5M12.5 11.3V5.6a1.25 1.25 0 0 1 2.5 0v6M15 11.8V8a1.25 1.25 0 0 1 2.5 0v5.5c0 3.9-2.6 6.8-6.3 6.8h-.6c-2.1 0-3.6-.9-4.8-2.6l-2.4-3.5a1.35 1.35 0 0 1 2.1-1.7l2 2.3"/></svg>',
shot: '<svg viewBox="0 0 24 24"><rect x="2.8" y="7.3" width="12.6" height="9.4" rx="1.8"/><path d="M15.4 10.6 21 7.8v8.4l-5.6-2.8"/></svg>',
};
const NAV_TIPS = { zoom: "Zoom: drag up to push in, down to pull back", pan: "Pan: drag to slide the camera", shot: "Back to your own shot" };
const GZ = 64;
// Blender's names for the scene's directions: X to the right, Y into the picture, Z up.
const AXES = [
{ n: "X", v: [1, 0, 0], col: "#e05561", pos: { az: 90, el: 0 }, neg: { az: -90, el: 0 } },
{ n: "Y", v: [0, 0, 1], col: "#86b84a", pos: null, neg: { az: 0, el: 0 } }, // nothing to see from behind the scene
{ n: "Z", v: [0, -1, 0], col: "#4b8fdf", pos: { az: 0, el: LIM.el[1] }, neg: { az: 0, el: LIM.el[0] } },
];
const SUB = [[-0.25, -0.25], [0.25, -0.25], [-0.25, 0.25], [0.25, 0.25]];
const VIEW = { az: -58, el: 28 }; // the corner map looks at the scene from the left
// ---------- tooltips: hover on a computer, press and hold a button on a phone ----------
let tipEl = null, tipTimer = 0, tipFor = null, tipXY = [0, 0], tipHeld = false;
function tip() {
if (!tipEl || !tipEl.isConnected) {
tipEl = el("div", "acm-tip", { role: "tooltip" });
tipEl.hidden = true;
document.body.append(tipEl);
}
return tipEl;
}
function placeTip(target) {
const t = target.dataset.tip, box = tip();
if (!t || !target.isConnected) return hideTip();
box.textContent = t;
box.hidden = false;
const r = target.getBoundingClientRect(), tw = box.offsetWidth, th = box.offsetHeight, atPtr = target.dataset.tipAt === "pointer";
if (target.closest(".acm-nav")) { // the column top right: the tip opens to its left and covers no other button
box.style.left = `${Math.max(8, r.left - tw - 8)}px`;
box.style.top = `${r.top + r.height / 2 - th / 2}px`;
return;
}
const ax = atPtr ? tipXY[0] : r.left + r.width / 2;
const below = atPtr ? tipXY[1] + 22 : r.bottom + 8, above = atPtr ? tipXY[1] - th - 12 : r.top - th - 8;
box.style.left = `${clamp(ax - tw / 2, 8, innerWidth - tw - 8)}px`;
box.style.top = `${below + th > innerHeight - 8 ? above : below}px`;
}
function hideTip() { clearTimeout(tipTimer); if (tipEl) tipEl.hidden = true; tipFor = null; }
const armTip = (target, ms) => { clearTimeout(tipTimer); tipFor = target; tipTimer = setTimeout(() => placeTip(target), ms); };
function wireTips(root) {
const targetOf = (e) => e.target?.closest?.("[data-tip]") ?? null;
root.addEventListener("pointermove", (e) => {
if (e.pointerType !== "mouse") return;
tipXY = [e.clientX, e.clientY];
if (e.buttons) { if (tipFor) hideTip(); return; }
const t = targetOf(e);
if (!t) { if (tipFor) hideTip(); return; }
if (t !== tipFor) { hideTip(); armTip(t, t.dataset.tipAt === "pointer" ? 900 : 450); }
else if (t.dataset.tipAt === "pointer") { hideTip(); armTip(t, 900); } // over a picture the tip waits until the mouse rests
}, true);
root.addEventListener("pointerleave", hideTip);
root.addEventListener("pointerdown", (e) => {
hideTip();
if (e.pointerType === "mouse") return;
const t = targetOf(e);
if (!t || t.dataset.tipAt === "pointer") return;
const x0 = e.clientX, y0 = e.clientY;
armTip(t, 500);
const done = () => { removeEventListener("pointermove", move, true); removeEventListener("pointerup", up, true); removeEventListener("pointercancel", up, true); };
const move = (m) => { if (Math.hypot(m.clientX - x0, m.clientY - y0) > 8) { clearTimeout(tipTimer); done(); } };
const up = () => {
if (tipEl && !tipEl.hidden) { tipHeld = true; setTimeout(() => { tipHeld = false; }, 400); setTimeout(hideTip, 1600); } else clearTimeout(tipTimer);
done();
};
addEventListener("pointermove", move, true); addEventListener("pointerup", up, true); addEventListener("pointercancel", up, true);
}, true);
// A hold that showed a tip is not a click.
root.addEventListener("click", (e) => { if (tipHeld) { tipHeld = false; e.preventDefault(); e.stopPropagation(); } }, true);
root.addEventListener("contextmenu", (e) => { if (e.target.closest?.(".acm-nav, .acm-keynav, .acm-btn")) e.preventDefault(); });
}
// ---------- one frame of the clip, as points ----------
async function loadFrame(entry, info, view, blobOf) {
const W = info.w, H = info.h;
const picture = new Image();
picture.decoding = "async";
const pictureReady = new Promise((ok, fail) => { picture.onload = ok; picture.onerror = () => fail(new Error("picture")); });
picture.src = view(entry.color);
const depthBlob = await blobOf(entry.depth);
const depthBitmap = await createImageBitmap(depthBlob, { colorSpaceConversion: "none", premultiplyAlpha: "none" });
await pictureReady;
const canvas = document.createElement("canvas");
canvas.width = W; canvas.height = H;
const g = canvas.getContext("2d", { willReadFrequently: true });
g.drawImage(picture, 0, 0, W, H);
const rgba = new Uint32Array(g.getImageData(0, 0, W, H).data.buffer.slice(0));
g.clearRect(0, 0, W, H);
g.drawImage(depthBitmap, 0, 0, W, H);
const { z, valid } = unpackDepth(g.getImageData(0, 0, W, H).data, W * H, info.zmin, info.zmax, info.far);
// the corner map leaves out what is very far away
const zs = [];
for (let i = 0; i < W * H; i += 5) if (valid[i]) zs.push(z[i]);
zs.sort((a, b) => a - b);
const zcap = Math.min(zs[Math.floor(zs.length * 0.97)] || info.far, (zs[zs.length >> 1] || 1) * 3.5);
return { ...entry, W, H, z, valid, rgba, zcap, picture };
}
// ---------- the face ----------
// opts: read() -> camera text, write(text), run() -> Promise<boolean>,
// view(fileEntry) -> url, onSize() when the floor changes.
export function mountCameraFace(opts) {
ensureCss();
const root = el("div", "acm-face");
const stage = el("div", "acm-stage");
const pic = el("canvas", "acm-pic", { "data-tip": "Drag to turn · Shift-drag to pan · wheel to zoom · click a spot to circle round it", "data-tip-at": "pointer" });
const picCtx = pic.getContext("2d");
const pivotMark = el("div", "acm-pivot");
const nav = el("div", "acm-nav");
const gizmo = el("canvas", "", { "data-tip": "Drag to turn. Click an axis to look from that side.", role: "img", "aria-label": "Axis ball: drag to turn the camera, tap an axis to look from that side" });
gizmo.width = GZ * 2; gizmo.height = GZ * 2;
const gctx = gizmo.getContext("2d");
nav.append(gizmo);
const inset = el("div", "acm-inset");
const out = el("canvas");
out.width = 360; out.height = 225;
const outCtx = out.getContext("2d");
const outImg = outCtx.createImageData(out.width, out.height);
const outZ = new Float32Array(out.width * out.height);
inset.append(out, el("span", "acm-tag", { text: "where the camera is" }));
const helpBtn = el("button", "acm-helpbtn", { text: "?", "aria-label": "How to steer", "aria-expanded": "false", "data-tip": "How to steer" });
const empty = el("div", "acm-empty");
stage.append(pic, pivotMark, inset, nav, helpBtn, empty);
const controls = el("div", "ausboss-camera-move-controls");
const readout = el("div", "acm-readout");
const where = el("span", "acm-where");
const pink = el("span", "acm-pink", { "data-tip": "How much of the frame the camera never saw. The LoRA paints it in." });
readout.append(where, pink);
// Always one line tall, empty or not, so a warning never pushes the controls down.
const warn = el("div", "acm-warn");
const backRow = el("div", "acm-back");
const back = makeScrubInput({
value: 0, min: LIM.back[0], max: LIM.back[1], step: 0.05, fineStep: 0.01, decimals: 2, unit: "×", unitWidth: 14, width: 132,
title: "How far the camera pulls back from what it circles. Below 0 pushes in.",
onChange: (v) => { stopPlay(); setCam({ ...camAt(keys, p), back: v }); },
});
backRow.append(el("span", "", { text: "Back" }), back.root);
const track = el("div", "acm-track", { "data-tip": "Click to put a key here. Drag to move the time bar, or drag a key to change its time.", "data-tip-at": "pointer" });
const inner = el("div", "acm-inner");
const head = el("div", "acm-head");
inner.append(head);
track.append(el("div", "acm-fill"), inner);
const row = el("div", "acm-row");
const transport = el("span", "acm-keynav transport", { role: "group", "aria-label": "Playback" });
const toStartBtn = el("button", "arr", { "aria-label": "Back to the start", "data-tip": "Back to the start", html: '<svg viewBox="0 0 14 14"><path d="M2.5 2.5h1.6v9H2.5zM11.5 2.5v9L5 7z"/></svg>' });
const play = el("button", "play", { text: "▶", "aria-label": "Play", "data-tip": "Play" });
const loop = el("button", "loop", { "aria-label": "Play on a loop", "aria-pressed": "false", "data-tip": "Play on a loop", html: '<svg viewBox="0 0 16 16"><path d="M3 9.5V7a2.5 2.5 0 0 1 2.5-2.5H12M10 2.5l2 2-2 2M13 6.5V9a2.5 2.5 0 0 1-2.5 2.5H4M6 13.5l-2-2 2-2"/></svg>' });
transport.append(toStartBtn, play, loop);
const keynav = el("span", "acm-keynav", { role: "group", "aria-label": "Keys" });
const prevKey = el("button", "arr", { "aria-label": "Previous key", "data-tip": "Previous key", html: '<svg viewBox="0 0 14 14"><path d="M10 2.5v9L3.5 7z"/></svg>' });
const toggleKey = el("button", "dia", { "aria-label": "Add a key here", "data-tip": "Add a key here", html: '<svg viewBox="0 0 16 16"><path d="M8 1.8 14.2 8 8 14.2 1.8 8z"/></svg>' });
const nextKey = el("button", "arr", { "aria-label": "Next key", "data-tip": "Next key", html: '<svg viewBox="0 0 14 14"><path d="M4 2.5v9L10.5 7z"/></svg>' });
keynav.append(prevKey, toggleKey, nextKey);
const reset = el("button", "acm-btn", { text: "Start over", "data-tip": "One key again, on your own shot" });
const time = el("span", "acm-time", { "data-tip": "Frame at the time bar" });
row.append(transport, keynav, reset, time);
controls.append(readout, warn, backRow, track, row);
const help = el("div", "acm-help");
help.hidden = true;
root.append(stage, controls, help);
wireTips(root);
// ---------- state ----------
let keys = [], pivotSpec = null, lastText = null, p = 0, playing = false, lastT = 0, dirty = true, alive = true;
let info = null, frames = [], shown = null, pivot = [0, 0, 2], pinkPct = 0, status = "idle", statusNote = "";
let flashText = "", flashUntil = 0, gBalls = [], gHot = false, hovered = false, overFace = false, looping = false, needsWrite = false;
let buf = null; // z, colour and source pixel of the moved view
const fallbackLens = { W: 416, H: 234, fx: 380, fy: 380, cx: 208, cy: 117 };
function readCamera() {
const text = opts.read();
if (text === lastText) return;
lastText = text;
const parsed = parseCamera(text);
keys = parsed.keys;
pivotSpec = parsed.pivot;
resolvePivot();
dirty = true;
}
function writeSoon() { needsWrite = true; dirty = true; }
function flushWrite() {
if (!needsWrite) return;
needsWrite = false;
const text = cameraText({ keys, pivot: pivotSpec });
lastText = text;
if (text !== opts.read()) opts.write(text);
}
function setCam(cam) { setKeyAt(keys, p, cam); writeSoon(); }
function lens() { return shown ?? (info ? { ...fallbackLens, ...info.frames[0], W: info.w, H: info.h } : fallbackLens); }
function resolvePivot() {
if (!info) { pivot = [0, 0, 2]; return; }
if (!pivotSpec) { pivot = info.pivot_auto?.slice() ?? [0, 0, 2]; return; }
if (info.pivot_from && Math.abs(info.pivot_from.u - pivotSpec.u) < 1e-4 && Math.abs(info.pivot_from.v - pivotSpec.v) < 1e-4
&& Math.abs(info.pivot_from.t - pivotSpec.t) < 1e-4) { pivot = info.pivot.slice(); return; }
const f = nearestLoaded(pivotSpec.t);
const found = f && pivotFromPixel(f, pivotSpec.u, pivotSpec.v);
if (found) pivot = found;
}
// ---------- frames ----------
function nearestFrame(t) {
let best = null;
for (const f of frames) if (!best || Math.abs(f.t - t) < Math.abs(best.t - t)) best = f;
return best;
}
function nearestLoaded(t) {
let best = null;
for (const f of frames) if (f.data && (!best || Math.abs(f.t - t) < Math.abs(best.t - t))) best = f;
return best?.data ?? null;
}
async function loadAll(payload) {
const order = [...frames].sort((a, b) => Math.abs(a.t - p) - Math.abs(b.t - p));
let failed = 0, next = 0;
// Four at a time, the frames nearest the time bar first.
const worker = async () => {
while (next < order.length) {
const f = order[next++];
if (info !== payload) return;
try { f.data = await loadFrame(f.entry, payload, opts.view, opts.blob); } catch { failed++; }
if (info !== payload) return;
if (f.data && status !== "ready") { status = "ready"; resolvePivot(); }
dirty = true;
}
};
await Promise.all([worker(), worker(), worker(), worker()]);
if (info !== payload) return;
if (failed === frames.length) { info = null; frames = []; shown = null; status = "idle"; statusNote = "The last run's pictures are gone. Run it again to see your clip."; dirty = true; }
}
// ---------- the camera's view ----------
function fitPicture() {
const L = lens(), sw = stage.clientWidth, sh = stage.clientHeight;
if (!sw || !sh) return null;
const s = Math.min(sw / L.W, sh / L.H), w = L.W * s, h = L.H * s, x = (sw - w) / 2, y = (sh - h) / 2;
Object.assign(pic.style, { left: `${x}px`, top: `${y}px`, width: `${w}px`, height: `${h}px` });
inset.hidden = !shown || sw < 300 || sh < MIN_STAGE;
return { x, y, w, h };
}
function renderPreview(cam, fit) {
const f = shown;
if (!f) { pic.hidden = true; pivotMark.hidden = true; return; }
pic.hidden = false;
const { W, H, z, rgba, fx, fy, cx, cy } = f;
if (!buf || buf.W !== W || buf.H !== H) {
buf = { W, H, zbuf: new Float32Array(W * H), cbuf: new Uint32Array(W * H), ibuf: new Int32Array(W * H), fill: new Uint8Array(W * H), img: picCtx.createImageData(W, H) };
pic.width = W; pic.height = H;
buf.img = picCtx.createImageData(W, H);
}
const { zbuf, cbuf, ibuf, fill, img } = buf;
const { R, C } = pose(cam, pivot);
const [r0, r1, r2, r3, r4, r5, r6, r7, r8] = R, [Cx, Cy, Cz] = C;
zbuf.fill(Infinity); ibuf.fill(-1);
for (let y = 0, i = 0; y < H; y++) for (let x = 0; x < W; x++, i++) {
const zz = z[i], c = rgba[i];
for (let s = 0; s < 4; s++) {
const X = (x + 0.5 + SUB[s][0] - cx) / fx * zz - Cx, Y = (y + 0.5 + SUB[s][1] - cy) / fy * zz - Cy, Z = zz - Cz;
const zc = r2 * X + r5 * Y + r8 * Z;
if (zc <= 1e-3) continue;
const u = fx * (r0 * X + r3 * Y + r6 * Z) / zc + cx, v = fy * (r1 * X + r4 * Y + r7 * Z) / zc + cy;
if (u < 0 || v < 0 || u >= W || v >= H) continue;
const idx = (v | 0) * W + (u | 0);
if (zc < zbuf[idx]) { zbuf[idx] = zc; cbuf[idx] = c; ibuf[idx] = i; }
}
}
for (let i = 0; i < W * H; i++) fill[i] = zbuf[i] < Infinity ? 1 : 0;
// fill one-pixel cracks: a gap mostly surrounded by picture takes their average (two passes, like the node)
for (let pass = 0; pass < 2; pass++) {
const was = fill.slice(), src = cbuf.slice();
let any = false;
for (let y = 1; y < H - 1; y++) for (let x = 1; x < W - 1; x++) {
const i = y * W + x;
if (was[i]) continue;
let n = 0, r = 0, g = 0, b = 0;
for (let oy = -1; oy <= 1; oy++) for (let ox = -1; ox <= 1; ox++) {
const q = i + oy * W + ox;
if (!was[q]) continue;
const c = src[q]; n++; r += c & 255; g += (c >>> 8) & 255; b += (c >>> 16) & 255;
}
if (n >= 5) { cbuf[i] = 0xFF000000 | ((b / n) << 16) | ((g / n) << 8) | (r / n); fill[i] = 1; any = true; }
}
if (!any) break;
}
const o = new Uint32Array(img.data.buffer);
let holes = 0;
for (let i = 0; i < W * H; i++) { if (fill[i]) o[i] = cbuf[i]; else { o[i] = MAG; holes++; } }
picCtx.putImageData(img, 0, 0);
pinkPct = 100 * holes / (W * H);
const q = toCam(R, C, pivot);
if (fit && q[2] > 1e-3) {
pivotMark.hidden = false;
pivotMark.style.left = `${fit.x + fit.w * (fx * q[0] / q[2] + cx) / W}px`;
pivotMark.style.top = `${fit.y + fit.h * (fy * q[1] / q[2] + cy) / H}px`;
} else pivotMark.hidden = true;
}
// ---------- the corner map: your scene from the side, and the camera in it ----------
function viewer() {
const P = pivot, d = Math.hypot(P[0], P[1], P[2]), T = [P[0] * 0.55, P[1] * 0.55, P[2] * 0.55];
const R = rot(VIEW.az, VIEW.el), b = mv(R, [0, 0, -d * 2.3]);
return { R, C: [T[0] + b[0], T[1] + b[1], T[2] + b[2]], f: 0.95 * out.width };
}
function proj(V, P) {
const q = toCam(V.R, V.C, P);
return q[2] <= 1e-3 ? null : [V.f * q[0] / q[2] + out.width / 2, V.f * q[1] / q[2] + out.height / 2, q[2]];
}
function frustum(V, cam, color, width) {
const L = lens(), { R, C } = pose(cam, pivot), s = Math.hypot(...pivot) * 0.3;
const corners = [[0, 0], [L.W, 0], [L.W, L.H], [0, L.H]].map(([x, y]) => {
const l = mv(R, [(x - L.cx) / L.fx * s, (y - L.cy) / L.fy * s, s]);
return proj(V, [C[0] + l[0], C[1] + l[1], C[2] + l[2]]);
});
const a = proj(V, C);
if (!a || corners.some((c) => !c)) return;
outCtx.strokeStyle = color; outCtx.lineWidth = width; outCtx.beginPath();
for (const c of corners) { outCtx.moveTo(a[0], a[1]); outCtx.lineTo(c[0], c[1]); }
outCtx.moveTo(corners[0][0], corners[0][1]);
for (const c of [...corners.slice(1), corners[0]]) outCtx.lineTo(c[0], c[1]);
outCtx.stroke();
// a small roof on the top edge, so up and down never mix
outCtx.fillStyle = color; outCtx.beginPath(); outCtx.moveTo(corners[0][0], corners[0][1]); outCtx.lineTo(corners[1][0], corners[1][1]);
const up = mv(R, [0, -L.H / 2 / L.fy * s * 1.35, s]), ut = proj(V, [C[0] + up[0], C[1] + up[1], C[2] + up[2]]);
if (ut) { outCtx.lineTo(ut[0], ut[1]); outCtx.closePath(); outCtx.globalAlpha = 0.6; outCtx.fill(); outCtx.globalAlpha = 1; }
}
function renderOutside(cam) {
if (inset.hidden) return;
const w = out.width, h = out.height, V = viewer(), o = new Uint32Array(outImg.data.buffer);
o.fill(0xFF100D0C); outZ.fill(Infinity);
const f = shown;
if (f) {
const { W, H, z, valid, rgba, zcap, fx, fy, cx, cy } = f, [r0, r1, r2, r3, r4, r5, r6, r7, r8] = V.R, [Cx, Cy, Cz] = V.C;
for (let y = 0; y < H; y += 2) for (let x = 0; x < W; x += 2) {
const i = y * W + x, zz = z[i];
if (!valid[i] || zz > zcap) continue;
const dx = (x + 0.5 - cx) / fx * zz - Cx, dy = (y + 0.5 - cy) / fy * zz - Cy, dz = zz - Cz, zc = r2 * dx + r5 * dy + r8 * dz;
if (zc <= 1e-3) continue;
const u = V.f * (r0 * dx + r3 * dy + r6 * dz) / zc + w / 2, v = V.f * (r1 * dx + r4 * dy + r7 * dz) / zc + h / 2;
const sz = clamp(Math.round(V.f * (zz / fx * 2.2) / zc), 1, 4), x0 = (u - sz / 2) | 0, y0 = (v - sz / 2) | 0;
for (let yy = y0; yy < y0 + sz; yy++) {
if (yy < 0 || yy >= h) continue;
for (let xx = x0; xx < x0 + sz; xx++) { if (xx < 0 || xx >= w) continue; const q = yy * w + xx; if (zc < outZ[q]) { outZ[q] = zc; o[q] = rgba[i]; } }
}
}
}
outCtx.putImageData(outImg, 0, 0);
frustum(V, { ...K0 }, "rgba(200,210,210,.55)", 1.5);
const project = (c) => proj(V, pose(c, pivot).C);
outCtx.strokeStyle = "rgba(79,179,169,.9)"; outCtx.lineWidth = 2; outCtx.setLineDash([5, 4]); outCtx.beginPath();
let first = true;
for (let i = 0; i <= 48; i++) {
const q = project(camAt(keys, i / 48));
if (!q) { first = true; continue; }
if (first) outCtx.moveTo(q[0], q[1]); else outCtx.lineTo(q[0], q[1]);
first = false;
}
outCtx.stroke(); outCtx.setLineDash([]);
for (const k of keys) {
const q = project(k);
if (!q) continue;
outCtx.save(); outCtx.translate(q[0], q[1]); outCtx.rotate(Math.PI / 4);
outCtx.fillStyle = Math.abs(k.t - p) < SAME_KEY ? "#bfe9e4" : "#4fb3a9"; outCtx.strokeStyle = "#0c0f10"; outCtx.lineWidth = 1.5;
outCtx.fillRect(-5, -5, 10, 10); outCtx.strokeRect(-5, -5, 10, 10); outCtx.restore();
}
const pv = proj(V, pivot), cc = project(cam);
if (pv && cc) { outCtx.strokeStyle = "rgba(79,179,169,.45)"; outCtx.lineWidth = 1; outCtx.beginPath(); outCtx.moveTo(cc[0], cc[1]); outCtx.lineTo(pv[0], pv[1]); outCtx.stroke(); }
frustum(V, cam, "#4fb3a9", 2.5);
}
// ---------- the axis ball ----------
function drawGizmo(cam) {
const g = gctx, c = GZ / 2, L = 21, R = rot(cam.az, cam.el);
g.setTransform(2, 0, 0, 2, 0, 0); g.clearRect(0, 0, GZ, GZ);
g.fillStyle = gHot ? "rgba(146,162,161,.26)" : "rgba(12,15,16,.42)"; g.beginPath(); g.arc(c, c, c - 1, 0, 7); g.fill();
gBalls = [];
for (const a of AXES) for (const sg of [1, -1]) {
const w = [a.v[0] * sg, a.v[1] * sg, a.v[2] * sg]; // the direction as the camera sees it
const q = [R[0] * w[0] + R[3] * w[1] + R[6] * w[2], R[1] * w[0] + R[4] * w[1] + R[7] * w[2], R[2] * w[0] + R[5] * w[1] + R[8] * w[2]];
gBalls.push({ a, sg, x: c + q[0] * L, y: c + q[1] * L, z: q[2], view: sg > 0 ? a.pos : a.neg });
}
gBalls.sort((m, n) => n.z - m.z); // far balls first, near ones on top
for (const b of gBalls) {
if (b.sg > 0) { g.strokeStyle = b.a.col; g.lineWidth = 2; g.beginPath(); g.moveTo(c, c); g.lineTo(b.x, b.y); g.stroke(); }
g.beginPath(); g.arc(b.x, b.y, b.sg > 0 ? 7 : 6, 0, 7);
if (b.sg > 0) {
g.fillStyle = b.a.col; g.fill();
g.fillStyle = "#0c0f10"; g.font = "bold 9px system-ui, sans-serif"; g.textAlign = "center"; g.textBaseline = "middle"; g.fillText(b.a.n, b.x, b.y + 0.5);
} else { g.globalAlpha = 0.35; g.fillStyle = b.a.col; g.fill(); g.globalAlpha = 1; g.strokeStyle = b.a.col; g.lineWidth = 1.2; g.stroke(); }
}
}
// ---------- gestures ----------
const picRect = () => pic.getBoundingClientRect();
function panBy(cam, dx, dy) { // the spot you circle stays under your finger
const L = lens(), r = picRect(), d = Math.hypot(...pivot), zz = Math.max(0.05 * d, pivot[2] + cam.back * d);
const du = dx * L.W / (r.width || 1), dv = dy * L.H / (r.height || 1);
return { px: (cam.px || 0) - du * zz / (L.fx * d), py: (cam.py || 0) - dv * zz / (L.fy * d) };
}
function dragTo(kind, cam, dx, dy) {
const r = picRect(), w = r.width || stage.clientWidth || 1, h = r.height || stage.clientHeight || 1;
if (kind === "orbit") setCam({ ...cam, az: cam.az - dx * 90 / w, el: cam.el + dy * 90 / w }); // the picture follows your finger
else if (kind === "pan") setCam({ ...cam, ...panBy(cam, dx, dy) });
else if (kind === "zoom") setCam({ ...cam, back: cam.back + dy / h * 2.4 }); // up = in, down = back
}
const kindFor = (e) => (e.shiftKey ? "pan" : (e.ctrlKey || e.metaKey) ? "zoom" : "orbit");
(() => {
const pts = new Map();
let g = null;
const begin = (kind, carry) => {
const v = [...pts.values()], cam = camAt(keys, p);
if (v.length >= 2) {
const [a, b] = v;
g = { kind: "pinch", cam, d: Math.max(8, Math.hypot(a[0] - b[0], a[1] - b[1])), mx: (a[0] + b[0]) / 2, my: (a[1] + b[1]) / 2, moved: true, multi: true };
} else if (v.length === 1) g = { kind, cam, x: v[0][0], y: v[0][1], moved: Boolean(carry), multi: Boolean(carry) };
else g = null;
};
pic.addEventListener("mousedown", (e) => { if (e.button === 1) e.preventDefault(); }); // no auto-scroll on middle click
pic.addEventListener("pointerdown", (e) => {
if (e.button > 1) return;
e.preventDefault(); e.stopPropagation();
pic.setPointerCapture(e.pointerId); pts.set(e.pointerId, [e.clientX, e.clientY]); stopPlay();
begin(kindFor(e));
});
pic.addEventListener("pointermove", (e) => {
if (!pts.has(e.pointerId) || !g) return;
pts.set(e.pointerId, [e.clientX, e.clientY]);
if (g.kind === "pinch") {
const [a, b] = [...pts.values()], d = Math.max(8, Math.hypot(a[0] - b[0], a[1] - b[1]));
const c = { ...g.cam, back: g.cam.back - Math.log(d / g.d) * 0.9 };
Object.assign(c, panBy(g.cam, (a[0] + b[0]) / 2 - g.mx, (a[1] + b[1]) / 2 - g.my));
setCam(c);
return;
}
const dx = e.clientX - g.x, dy = e.clientY - g.y;
if (!g.moved && Math.hypot(dx, dy) < 5) return;
g.moved = true;
dragTo(g.kind, g.cam, dx, dy);
});
const up = (e) => {
if (!pts.has(e.pointerId)) return;
pts.delete(e.pointerId);
if (pts.size === 0) {
if (g && !g.moved && !g.multi && g.kind === "orbit" && e.type === "pointerup" && e.button === 0) pickPivot(e);
g = null;
return;
}
begin("orbit", true); // a finger left: carry on from here
};
pic.addEventListener("pointerup", up);
pic.addEventListener("pointercancel", up);
})();
function pickPivot(e) { // tap: circle round what you tapped
const f = shown, r = picRect();
if (!f || !buf || !r.width) return;
const x = ((e.clientX - r.left) / r.width * f.W) | 0, y = ((e.clientY - r.top) / r.height * f.H) | 0;
if (x < 0 || y < 0 || x >= f.W || y >= f.H) return;
let best = -1;
for (let rr = 0; rr <= 3 && best < 0; rr++) for (let oy = -rr; oy <= rr && best < 0; oy++) for (let ox = -rr; ox <= rr; ox++) {
const xx = x + ox, yy = y + oy;
if (xx < 0 || yy < 0 || xx >= f.W || yy >= f.H) continue;
const i = buf.ibuf[yy * f.W + xx];
if (i >= 0) { best = i; break; }
}
if (best < 0 || !f.valid[best]) return; // sky: nothing to circle
const spec = { u: ((best % f.W) + 0.5) / f.W, v: (Math.floor(best / f.W) + 0.5) / f.H, t: f.t };
const found = pivotFromPixel(f, spec.u, spec.v);
if (!found) return;
pivotSpec = spec; pivot = found;
writeSoon();
}
stage.addEventListener("wheel", (e) => {
e.preventDefault(); e.stopPropagation(); stopPlay();
const unit = e.deltaMode === 1 ? 33 : e.deltaMode === 2 ? 400 : 1, c = camAt(keys, p);
if (e.shiftKey) { const a = (e.deltaY || e.deltaX) * unit; setCam({ ...c, ...panBy(c, 0, -a * 0.4) }); return; } // Shift + wheel pans up and down
setCam({ ...c, back: c.back + e.deltaY * unit * (e.ctrlKey ? 0.01 : 0.0008) }); // wheel, and a trackpad pinch (Ctrl), zoom
}, { passive: false });
stage.addEventListener("gesturestart", (e) => e.preventDefault()); // Safari: never zoom the page instead
stage.addEventListener("pointerdown", (e) => { if (e.button === 1) { e.preventDefault(); e.stopPropagation(); } });
stage.addEventListener("pointerenter", () => { hovered = true; });
stage.addEventListener("pointerleave", () => { hovered = false; });
// Blender's numpad, while the pointer is over the picture.
const onKey = (e) => {
if (!hovered || !alive || e.altKey || e.target?.closest?.("input, textarea, select, [contenteditable]")) return;
const c = camAt(keys, p), w = picRect().width || 300, step = 15, pan = e.ctrlKey || e.metaKey;
const go = { Numpad4: [-1, 0], Numpad6: [1, 0], Numpad8: [0, 1], Numpad2: [0, -1] }[e.code];
if (go) setCam(pan ? { ...c, ...panBy(c, -go[0] * w * 0.1, go[1] * w * 0.1) } : { ...c, az: c.az + go[0] * step, el: c.el + go[1] * step });
else if (e.code === "NumpadAdd" || e.code === "NumpadSubtract") setCam({ ...c, back: c.back + (e.code === "NumpadAdd" ? -0.1 : 0.1) });
else if (e.code === "Numpad1" && !pan) setCam({ ...c, az: 0, el: 0 }); // front
else if (e.code === "Numpad3") setCam({ ...c, az: pan ? -90 : 90, el: 0 }); // right, Ctrl: left
else if (e.code === "Numpad7") setCam({ ...c, az: 0, el: pan ? LIM.el[0] : LIM.el[1] }); // top, Ctrl: bottom
else if (e.code === "Numpad0") setCam({ ...K0 }); // your own shot
else if (e.key === "Escape" && !help.hidden) { openHelp(false); }
else return;
e.preventDefault(); e.stopPropagation(); stopPlay();
};
window.addEventListener("keydown", onKey, true);
// Space plays and pauses, Home goes to the start, while the pointer is over the face.
// Held there, Space would otherwise start a canvas pan.
root.addEventListener("pointerenter", () => { overFace = true; });
root.addEventListener("pointerleave", () => { overFace = false; });
const onPlayKey = (e) => {
if (!overFace || !alive || e.altKey || e.ctrlKey || e.metaKey || e.repeat && e.code === "Space") return;
if (e.target?.closest?.("input, textarea, select, button, [contenteditable]")) return;
if (e.key === "Home") toStart();
else if (e.code === "Space") play.click();
else return;
e.preventDefault(); e.stopPropagation();
};
window.addEventListener("keydown", onPlayKey, true);
// the axis ball: drag turns, a tap on an axis looks from that side
(() => {
let g = null;
gizmo.addEventListener("pointerenter", () => { gHot = true; dirty = true; });
gizmo.addEventListener("pointerleave", () => { if (!g) { gHot = false; dirty = true; } });
gizmo.addEventListener("pointerdown", (e) => {
if (e.button > 1) return;
e.preventDefault(); e.stopPropagation();
gizmo.setPointerCapture(e.pointerId); stopPlay(); gHot = true; dirty = true;
g = { id: e.pointerId, x: e.clientX, y: e.clientY, cam: camAt(keys, p), moved: false };
});
gizmo.addEventListener("pointermove", (e) => {
if (!g || e.pointerId !== g.id) return;
const dx = e.clientX - g.x, dy = e.clientY - g.y;
if (!g.moved && Math.hypot(dx, dy) < 4) return;
g.moved = true;
dragTo("orbit", g.cam, dx * 1.5, dy * 1.5);
});
const end = (e) => {
if (!g || e.pointerId !== g.id) return;
if (!g.moved && e.type === "pointerup" && (e.pointerType === "mouse" || !tipEl || tipEl.hidden)) {
const r = gizmo.getBoundingClientRect(), x = (e.clientX - r.left) * GZ / r.width, y = (e.clientY - r.top) * GZ / r.height;
let hit = null;
for (const b of gBalls) if (Math.hypot(b.x - x, b.y - y) <= 9) hit = b;
if (hit?.view) setCam({ ...camAt(keys, p), ...hit.view });
else if (hit) flash("Nothing behind: the camera never saw the back.");
}
g = null;
if (e.pointerType !== "mouse") gHot = false;
dirty = true;
};
gizmo.addEventListener("pointerup", end);
gizmo.addEventListener("pointercancel", end);
})();
for (const kind of ["zoom", "pan", "shot"]) {
const b = el("button", kind, { html: NAV_ICONS[kind], "data-tip": NAV_TIPS[kind], "aria-label": NAV_TIPS[kind] });
nav.append(b);
if (kind === "shot") { b.onclick = (e) => { e.stopPropagation(); stopPlay(); setCam({ ...K0 }); }; continue; }
let g = null;
b.addEventListener("pointerdown", (e) => {
if (e.button > 1) return;
e.preventDefault(); e.stopPropagation();
b.setPointerCapture(e.pointerId); stopPlay(); b.classList.add("on");
g = { id: e.pointerId, x: e.clientX, y: e.clientY, cam: camAt(keys, p) };
});
b.addEventListener("pointermove", (e) => {
if (!g || e.pointerId !== g.id) return;
const m = kind === "pan" ? 1 : 1.5;
dragTo(kind, g.cam, (e.clientX - g.x) * m, (e.clientY - g.y) * m);
});
const end = (e) => { if (g && e.pointerId === g.id) { g = null; b.classList.remove("on"); } };
b.addEventListener("pointerup", end);
b.addEventListener("pointercancel", end);
}
function flash(text) { flashText = text; flashUntil = performance.now() + 2600; dirty = true; setTimeout(() => { dirty = true; }, 2700); }
// ---------- the ? : the whole how-to, out of the way until asked for ----------
function buildHelp() {
help.innerHTML = "";
const h = el("div", "acm-help-h", { text: "How to steer" });
const x = el("button", "acm-help-x", { text: "×", "aria-label": "Close" });
x.onclick = () => openHelp(false);
h.append(x);
const dl = el("dl");
const rows = [
["Picture", "Drag to turn the camera round. Pinch or scroll to pull back or push in; two fingers or Shift pan. Tap a spot to circle round it."],
["Top right", "Drag the axis ball to turn, or tap an axis to look from that side. Drag the magnifier to zoom and the hand to pan. The camera puts you back on your own shot."],
["Keys", "Click the time bar to put a key there, or drag the time bar and move the camera. ◆ adds or removes a key, ◀ ▶ jump between keys. Drag a key along the bar to change its time."],
["Play", "⏮ goes back to the start, ▶ plays, and the loop button plays it round and round."],
["Mouse", "Like Blender: middle-drag turns, Shift pans, Ctrl zooms, the wheel zooms. Numpad 4 6 8 2 turn, + − zoom, 1 3 7 front, right, top, 0 your own shot. Space plays, Home goes to the start."],
["Pink", "What the camera never saw. The CrossView-Warp LoRA paints it in."],
];
for (const [k, v] of rows) dl.append(el("dt", "", { text: k }), el("dd", "", { text: v }));
help.append(h, dl);
}
function openHelp(open) { if (open) buildHelp(); help.hidden = !open; helpBtn.setAttribute("aria-expanded", String(open)); }
helpBtn.addEventListener("pointerdown", (e) => e.stopPropagation());
helpBtn.onclick = (e) => { e.stopPropagation(); openHelp(help.hidden); };
const closeHelp = (e) => { if (!help.hidden && !help.contains(e.target) && !helpBtn.contains(e.target)) openHelp(false); };
document.addEventListener("pointerdown", closeHelp, true);
// ---------- the time bar ----------
let keyEls = [];
function drawTrack() {
for (const k of keyEls) k.remove();
keyEls = [];
for (const k of keys) {
const d = el("div", `acm-key${Math.abs(k.t - p) < SAME_KEY ? " on" : ""}`);
d.style.left = `${k.t * 100}%`;
inner.insertBefore(d, head);
keyEls.push(d);
}
head.style.left = `${p * 100}%`;
const count = info?.count ?? 0;
time.textContent = count ? `${Math.round(p * (count - 1)) + 1} / ${count}` : `${Math.round(p * 100)}%`;
// the diamond is filled while the time bar sits on a key, hollow when it does not
const on = keyAt(keys, p) >= 0;
toggleKey.classList.toggle("on", on);
const tipText = !on ? "Add a key here" : keys.length > 1 ? "Remove this key" : "The last key stays";
toggleKey.dataset.tip = tipText; toggleKey.setAttribute("aria-label", tipText);
toggleKey.disabled = on && keys.length < 2;
prevKey.disabled = !keys.some((k) => k.t < p - SAME_KEY);
nextKey.disabled = !keys.some((k) => k.t > p + SAME_KEY);
}
(() => {
let drag = null;
const tAt = (e) => { const r = inner.getBoundingClientRect(); return clamp((e.clientX - r.left) / (r.width || 1), 0, 1); };
const set = (e) => { let t = tAt(e); for (const k of keys) if (Math.abs(k.t - t) < 0.03) t = k.t; p = t; dirty = true; };
track.addEventListener("pointerdown", (e) => {
if (e.button !== 0) return;
e.stopPropagation();
track.setPointerCapture(e.pointerId); stopPlay();
const r = inner.getBoundingClientRect();
let hit = null, best = 11; // press on a key to grab it; anywhere else moves the time bar
for (const k of keys) { const d = Math.abs(r.left + k.t * r.width - e.clientX); if (d < best) { best = d; hit = k; } }
drag = { key: hit, x: e.clientX, moved: false };
if (hit) { p = hit.t; dirty = true; } else set(e);
});
track.addEventListener("pointermove", (e) => {
if (!drag) return;
if (!drag.key) { if (Math.abs(e.clientX - drag.x) >= 4) drag.moved = true; set(e); return; }
if (!drag.moved && Math.abs(e.clientX - drag.x) < 4) return;
drag.moved = true; // a key slides between its neighbours and never jumps over one
const i = keys.indexOf(drag.key), lo = i > 0 ? keys[i - 1].t + 0.02 : 0, hi = i < keys.length - 1 ? keys[i + 1].t - 0.02 : 1;
drag.key.t = clamp(tAt(e), lo, hi); p = drag.key.t;
writeSoon();
});
// A click on an empty spot of the bar puts a key there; a drag only moves the time bar.
const end = (e) => {
if (drag && !drag.key && !drag.moved && e.type === "pointerup" && keyAt(keys, p) < 0) setCam(camAt(keys, p));
drag = null;
};
track.addEventListener("pointerup", end);
track.addEventListener("pointercancel", end);
})();
const playLabel = (on) => { play.textContent = on ? "❚❚" : "▶"; play.dataset.tip = on ? "Pause" : "Play"; play.setAttribute("aria-label", play.dataset.tip); };
function stopPlay() { if (playing) { playing = false; playLabel(false); } }
play.onclick = () => {
if (playing) { stopPlay(); return; }
if (p >= 0.999) p = 0;
playing = true; lastT = performance.now(); playLabel(true);
};
loop.onclick = () => {
looping = !looping;
loop.setAttribute("aria-pressed", String(looping));
loop.dataset.tip = looping ? "Looping: click to play once" : "Play on a loop";
loop.setAttribute("aria-label", loop.dataset.tip);
};
const toStart = () => { p = 0; lastT = performance.now(); dirty = true; }; // playing carries on from the start
toStartBtn.onclick = toStart;
toggleKey.onclick = () => { // add a key at the time bar, or remove the one it sits on
stopPlay();
const i = keyAt(keys, p);
if (i < 0) setCam(camAt(keys, p));
else if (keys.length > 1) { keys.splice(i, 1); writeSoon(); }
};
const jump = (dir) => {
stopPlay();
const k = dir < 0 ? [...keys].reverse().find((x) => x.t < p - SAME_KEY) : keys.find((x) => x.t > p + SAME_KEY);
if (k) { p = k.t; dirty = true; }
};
prevKey.onclick = () => jump(-1);
nextKey.onclick = () => jump(1);
reset.onclick = () => {
stopPlay();
keys.length = 0; keys.push({ t: 0, ...K0 });
pivotSpec = null; resolvePivot(); p = 0;
writeSoon();
};
for (const b of [play, prevKey, toggleKey, nextKey, reset]) b.addEventListener("pointerdown", (e) => e.stopPropagation());
// ---------- before there is a clip to look at ----------
function drawEmpty() {
const show = status !== "ready";
empty.hidden = !show;
if (!show) return;
const key = `${status}|${statusNote}`;
if (empty.dataset.key === key) return;
empty.dataset.key = key;
empty.innerHTML = "";
if (status === "running") {
empty.append(el("b", "", { text: "Reading the depth…" }), el("span", "", { text: "MoGe looks at every frame once. A long clip takes a while." }));
return;
}
if (status === "loading") { empty.append(el("b", "", { text: "Loading your clip…" })); return; }
const go = el("button", "acm-btn primary", { text: "Show my clip", "data-tip": "Runs this node and the depth before it, not the whole workflow" });
go.addEventListener("pointerdown", (e) => e.stopPropagation());
go.onclick = async (e) => {
e.stopPropagation();
status = "running"; statusNote = ""; dirty = true;
let ok = false;
try { ok = await opts.run(); } catch { ok = false; }
if (!ok && status === "running") { status = "idle"; statusNote = "It did not run. Pick a clip in Load Video, and check MoGe Inference is connected."; dirty = true; }
};
empty.append(el("b", "", { text: "Steer the camera on your clip" }), el("span", "", { text: statusNote || "Run the depth once to see your first frame here." }), go);
}
// ---------- loop ----------
function frame(now) {
if (!alive) return;
requestAnimationFrame(frame);
if (!root.isConnected) return;
readCamera();
if (playing) {
const dur = Math.max(1, (info?.count ?? 121) / FPS);
p += (now - lastT) / 1000 / dur; lastT = now; dirty = true;
if (p >= 1) { if (looping) p %= 1; else { p = 1; stopPlay(); } }
}
flushWrite();
if (!dirty) return;
if (root.offsetParent === null && !root.getClientRects().length) return;
dirty = false;
const cam = camAt(keys, p);
const want = nearestLoaded(p);
// a new run's frames take over as they load; until then the last ones stay up
if (want && want !== shown) shown = want;
if (!info) shown = null;
const fit = fitPicture();
drawEmpty();
renderPreview(cam, fit);
drawGizmo(cam);
renderOutside(cam);
drawTrack();
where.textContent = describe(cam);
pink.textContent = shown ? `pink ${pinkPct.toFixed(0)}%` : "";
const over = pastLoraRange(cam), fl = performance.now() < flashUntil;
const warnText = fl ? flashText : over ? "Past the LoRA's range: it learned up to 45° of turn, 30° up and 20° down." : "";
if (warn.textContent !== warnText) { warn.textContent = warnText; warn.dataset.tip = warnText; }
if (!warnText) delete warn.dataset.tip;
back.set(cam.back);
}
const resize = new ResizeObserver(() => { dirty = true; });
resize.observe(stage);
requestAnimationFrame(frame);
return {
root,
// What a run of the node sent back: the frames to look at.
takeScene(payload) {
if (!payload || !Array.isArray(payload.frames) || !payload.frames.length) return;
info = payload;
frames = payload.frames.map((entry) => ({ t: entry.t, entry, data: null }));
shown = null;
if (status !== "ready") status = "loading";
resolvePivot();
dirty = true;
loadAll(payload);
},
runFailed(note) {
if (status !== "running") return;
status = "idle"; statusNote = note || "It did not run. Pick a clip in Load Video, and check MoGe Inference is connected."; dirty = true;
},
running() { if (status !== "ready") { status = "running"; dirty = true; } },
refresh() { lastText = null; dirty = true; },
get hasScene() { return Boolean(info); },
// The least height the face needs: the picture's least, and the controls as they measure.
floor() { return MIN_STAGE + (controls.offsetHeight || CONTROLS_HEIGHT); },
destroy() {
alive = false;
resize.disconnect();
window.removeEventListener("keydown", onKey, true);
window.removeEventListener("keydown", onPlayKey, true);
document.removeEventListener("pointerdown", closeHelp, true);
hideTip();
},
};
}
+331
View File
@@ -0,0 +1,331 @@
// Camera Move: the shot on the node, and on the box (subgraph node) that holds it.
//
// The face itself is face.mjs. This file mounts it, keeps it pointed at
// wherever the camera keys are kept, runs the node alone when the face asks
// for the clip, and hands it what a run sent back.
//
// Where the keys are kept:
// - on the node, in its own `camera` setting;
// - once `camera` is brought up to a box (a subgraph input linked to it),
// in the box's own copy of the setting, one copy per box. The face then
// sits on the box too, and the node inside shows that box's keys.
import { api } from "../../../scripts/api.js";
import { app } from "../../../scripts/app.js";
import { chainCallback, keepDomWidgetWidthAuto, notifyAusbossChange, showToast } from "../shared/index.mjs";
import { executionIdOf, isSwitchedOff } from "../shared/graph_ids.mjs";
import { SUBGRAPH_INPUT_ID, boxesOf, hostBox, workflowNodes } from "../shared/lora_box.mjs";
import { WIDGET_FRAME, ensureNodeMinHeight, fillNodeHeight } from "../shared/panel_layout.mjs";
import { hideWidget } from "../shared/widget_visibility.mjs";
import { CONTROLS_HEIGHT, MIN_STAGE, MIN_WIDTH, mountCameraFace } from "./face.mjs";
import { mountBoxCard } from "./box_card.mjs";
import { CARD_WIDGET } from "../shared/widget_card.mjs";
const NODE_CLASS = "AUSBOSS_NODES_CameraMove";
const CAMERA = "camera";
const UI_KEY = "ausboss_camera_scene";
const FACE_WIDGET = "ausboss_camera_move";
const OPENING_WIDTH = 460;
const OPENING_ROOM = 90; // a new node opens this much taller than its floor, so a wide clip fills the picture
const MIN_HEIGHT = MIN_STAGE + CONTROLS_HEIGHT + WIDGET_FRAME;
const mounts = new Set();
const rootGraph = () => app.rootGraph ?? app.graph;
const isMove = (node) => node?.type === NODE_CLASS || node?.comfyClass === NODE_CLASS;
const same = (a, b) => a != null && b != null && String(a) === String(b);
const linkOf = (graph, id) => (id == null || !graph ? null : graph.getLink?.(id) ?? graph.links?.get?.(id) ?? graph.links?.[id] ?? null);
const nodeOf = (graph, id) => graph?.getNodeById?.(id) ?? (graph?.nodes ?? graph?._nodes ?? []).find((node) => same(node?.id, id)) ?? null;
const ownCamera = (node) => node?.widgets?.find((widget) => widget?.name === CAMERA) ?? null;
const boxCopy = (box, name) => (box?.widgets ?? []).find((widget) => widget?.name === name && typeof widget.value === "string") ?? null;
// The subgraph input that brings a node's camera in from its box, or null.
function cameraInputName(node) {
const graph = node?.graph;
const index = (node?.inputs ?? []).findIndex((input) => input?.name === CAMERA);
if (!graph || index < 0) return null;
const link = linkOf(graph, node.inputs[index].link);
if (!link || !same(link.origin_id, SUBGRAPH_INPUT_ID)) return null;
return graph.inputs?.[link.origin_slot]?.name ?? null;
}
// The first Camera Move inside a box whose camera comes up to the box: { name, inner }.
function boxCamera(box) {
const subgraph = box?.subgraph;
for (const input of subgraph?.inputs ?? []) {
for (const id of input?.linkIds ?? []) {
const link = linkOf(subgraph, id);
const inner = link && nodeOf(subgraph, link.target_id);
if (isMove(inner) && inner.inputs?.[link.target_slot]?.name === CAMERA) return { name: input.name, inner };
}
}
return null;
}
// Where a face reads and writes the keys, and which run of the node it shows.
function sourceOf(mount) {
const root = rootGraph();
if (mount.box) {
const widget = boxCopy(mount.box, mount.inputName);
if (widget) hideWidget(widget);
const at = executionIdOf(root, mount.box);
return { widget, id: at ? `${at}:${mount.inner.id}` : null, holder: mount.box };
}
const node = mount.inner;
const fed = cameraInputName(node);
if (fed) {
const box = hostBox(node.graph, boxesOf(node.graph, workflowNodes(root)), null);
const widget = box && boxCopy(box, fed);
const at = box && executionIdOf(root, box);
if (widget) return { widget, id: at ? `${at}:${node.id}` : executionIdOf(root, node), holder: box };
}
return { widget: ownCamera(node), id: executionIdOf(root, node), holder: node };
}
const viewRoute = (file) => `/view?${new URLSearchParams({ filename: file.filename, type: file.type || "temp", subfolder: file.subfolder || "" })}`;
const viewUrl = (file) => api.apiURL(viewRoute(file));
// The depth picture's exact bytes, through ComfyUI's own request helper.
const viewBlob = async (file) => {
const response = await api.fetchApi(viewRoute(file));
if (!response.ok) throw new Error(`depth ${response.status}`);
return response.blob();
};
function locatorOf(node) {
const graph = node?.graph;
if (!graph || node.id == null) return null;
const root = graph.rootGraph ?? graph;
return graph !== root && graph.id ? `${graph.id}:${node.id}` : String(node.id);
}
function restore(mount) {
if (mount.face.hasScene) return;
const scene = app.nodeOutputs?.[locatorOf(mount.inner)]?.[UI_KEY]?.[0];
if (scene) mount.face.takeScene(scene);
}
function mount(host, { box = null, inputName = null, inner = host } = {}) {
const entry = { host, box, inputName, inner, face: null, widget: null };
const face = mountCameraFace({
read: () => String(sourceOf(entry).widget?.value ?? ""),
write: (text) => {
const widget = sourceOf(entry).widget;
if (!widget) return;
widget.value = text;
app.canvas?.setDirty?.(true, false);
notifyAusbossChange();
},
view: viewUrl,
blob: viewBlob,
run: async () => {
const { id, holder } = sourceOf(entry);
if (!id || typeof app.queuePrompt !== "function") return false;
if (isSwitchedOff(rootGraph(), holder) || isSwitchedOff(rootGraph(), inner)) {
showToast({ severity: "warn", summary: "Camera Move", detail: "This node is switched off (bypassed or muted), so it cannot run." });
return false;
}
entry.askedAt = performance.now();
// At the front of the queue: the clip shows up once the job running now is done.
const queued = await app.queuePrompt(-1, 1, [id]);
// Refused: an empty loader, a missing link, or a check that stopped it (it says why in a note).
// Nothing was queued, so the face must not wait for a picture that never comes.
if (queued === false) return false;
return true;
},
});
entry.face = face;
const widget = host.addDOMWidget(box ? `${FACE_WIDGET}:${inputName}` : FACE_WIDGET, FACE_WIDGET, face.root, { serialize: false, hideOnZoom: false });
keepDomWidgetWidthAuto(widget);
// Never saved with the workflow: options.serialize only keeps it out of the prompt.
widget.serialize = false;
// The floor follows the controls as they measure on this screen (fonts differ between phones).
fillNodeHeight(widget, { minWidth: MIN_WIDTH, minHeight: () => face.floor() + WIDGET_FRAME, minNodeSize: [MIN_WIDTH, MIN_HEIGHT], exactMinWidth: true });
entry.widget = widget;
mounts.add(entry);
restore(entry);
holdFloor(host);
return entry;
}
// A node or box kept shorter than the face needs (a saved size, a font that
// runs taller on this screen) would cut the controls off at its bottom edge.
function holdFloor(host) {
requestAnimationFrame(() => requestAnimationFrame(() => {
if (ensureNodeMinHeight(host)) host.setDirtyCanvas?.(true, true);
}));
}
function unmount(entry) {
mounts.delete(entry);
entry.face.destroy();
try { entry.host.removeWidget?.(entry.widget); } catch { entry.widget?.onRemove?.(); }
}
// A new node or box opens with room for the picture; a saved size wins.
function openRoomy(host) {
const floor = Number(host.computeSize?.()?.[1]);
if (!Number.isFinite(floor) || host.size?.[1] > floor + 40) return;
host.setSize?.([Math.max(host.size?.[0] ?? 0, OPENING_WIDTH), floor + OPENING_ROOM]);
}
function build(node) {
if (node.__ausbossCameraMove) return node.__ausbossCameraMove;
hideWidget(ownCamera(node));
node.__ausbossCameraMove = mount(node);
return node.__ausbossCameraMove;
}
// The settings card and any LoRA rows sit on top; the shot comes last and
// takes the rest of the box, so its buttons stay on the bottom edge.
function faceLast(box) {
// A box makes `widgets` afresh on every read (its settings, then the panels
// added to it), so the panels are put in order in their own list: any other
// panel (a LoRA Loader's rows), the settings card, then the face. Nodes 2.0
// lays them out in the order they were made, which is the same.
const widgets = Array.isArray(box._extraWidgets) ? box._extraWidgets : box.widgets ?? [];
const order = [widgets.find((w) => w?.name === CARD_WIDGET), box.__ausbossCameraBox?.widget].filter(Boolean);
const want = [...widgets.filter((w) => !order.includes(w)), ...order];
if (want.every((w, i) => widgets[i] === w)) return false;
widgets.splice(0, widgets.length, ...want);
box.setDirtyCanvas?.(true, true);
return true;
}
// A box row reads its options from the widget inside. A node with a card
// (Image Resize) hides its own rows by setting options.hidden there, and
// Nodes 2.0 would then hide the box's row too. The box keeps every setting
// brought up to it in view, except the camera text the face stands in for.
const SHOWN = new WeakMap();
function shownOptions(options) {
if (!options || typeof options !== "object" || !options.hidden) return options;
let view = SHOWN.get(options);
if (!view) {
view = new Proxy(options, { get: (target, key) => (key === "hidden" ? false : Reflect.get(target, key)) });
SHOWN.set(options, view);
}
return view;
}
function showSettings(box, keep) {
for (const widget of box.widgets ?? []) {
if (widget === keep || widget.__ausbossCameraShown || !widget.options?.hidden) continue;
const own = Object.getOwnPropertyDescriptor(widget, "options");
if (!own?.get || !own.configurable || widget.__ausbossVisibilityStash) continue;
Object.defineProperty(widget, "options", { ...own, get() { return shownOptions(own.get.call(this)); } });
widget.__ausbossCameraShown = true;
}
}
// A box shows the face while the camera of a Camera Move inside comes up to it.
function syncBox(box) {
if (!box?.isSubgraphNode?.() || !box.graph) return;
const found = boxCamera(box);
const have = box.__ausbossCameraBox;
if (have && (!found || have.inner !== found.inner || have.inputName !== found.name || !boxCopy(box, found.name))) {
unmount(have);
box.__ausbossCameraBox = null;
box.setSize?.([box.size[0], box.computeSize?.()[1] ?? box.size[1]]);
}
if (box.__ausbossCameraBox) {
if (mountBoxCard(box, boxCopy(box, box.__ausbossCameraBox.inputName), faceLast)) holdFloor(box);
faceLast(box);
showSettings(box, boxCopy(box, box.__ausbossCameraBox.inputName));
return;
}
if (!found || !boxCopy(box, found.name)) return;
hideWidget(boxCopy(box, found.name));
// The card before the face: Nodes 2.0 lays a box's panels out in the order they were made.
mountBoxCard(box, boxCopy(box, found.name), faceLast);
box.__ausbossCameraBox = mount(box, { box, inputName: found.name, inner: found.inner });
faceLast(box);
showSettings(box, boxCopy(box, found.name));
if (!box.__ausbossCameraSized) openRoomy(box);
box.__ausbossCameraSized = true;
box.setDirtyCanvas?.(true, true);
}
function watchBox(box) {
if (!box?.isSubgraphNode?.() || box.__ausbossCameraWatch) return;
const abort = new AbortController();
box.__ausbossCameraWatch = abort;
const later = () => requestAnimationFrame(() => syncBox(box));
for (const type of ["widget-promoted", "widget-demoted", "input-added", "removing-input"]) {
box.subgraph?.events?.addEventListener?.(type, later, { signal: abort.signal });
}
chainCallback(box, "onConfigure", (info) => { if (info?.size) box.__ausbossCameraSized = true; later(); holdFloor(box); });
chainCallback(box, "onRemoved", () => {
abort.abort();
if (box.__ausbossCameraBox) unmount(box.__ausbossCameraBox);
box.__ausbossCameraBox = null;
});
later();
}
// What a run sent back reaches every face that shows that run of the node.
api.addEventListener("executed", ({ detail }) => {
try {
const scene = detail?.output?.[UI_KEY]?.[0];
if (!scene) return;
const ids = new Set([String(detail.node ?? ""), String(detail.display_node ?? "")]);
for (const entry of mounts) if (ids.has(String(sourceOf(entry).id))) entry.face.takeScene(scene);
} catch {
// Never in the way of a run.
}
});
const stopped = (note) => { for (const entry of mounts) entry.face.runFailed(note); };
api.addEventListener("execution_error", ({ detail }) => {
if (/OutOfMemory/i.test(String(detail?.exception_type ?? ""))) {
stopped("The graphics card is full. Close other jobs using it, then try again.");
return;
}
const why = String(detail?.exception_message ?? "").split("\n")[0].slice(0, 160);
stopped(why ? `The run stopped: ${why}` : null);
});
api.addEventListener("execution_interrupted", () => stopped("The run was stopped."));
// A run that ended without sending the clip (it never reached the node)
// would leave a face waiting for good. Once the queue is empty, it stops.
api.addEventListener("status", ({ detail }) => {
if (detail?.exec_info?.queue_remaining !== 0) return;
const now = performance.now();
for (const entry of mounts) if (now - (entry.askedAt ?? 0) > 1500) entry.face.runFailed();
});
app.registerExtension({
name: "ausboss.camera_move",
onNodeOutputsUpdated() {
for (const entry of mounts) restore(entry);
},
nodeCreated(node) {
if (node?.isSubgraphNode?.()) watchBox(node);
},
loadedGraphNode(node) {
if (node?.isSubgraphNode?.()) watchBox(node);
},
beforeRegisterNodeDef(nodeType, nodeData) {
if (nodeData?.name !== NODE_CLASS) return;
chainCallback(nodeType.prototype, "onNodeCreated", function () {
build(this);
});
chainCallback(nodeType.prototype, "onAdded", function () {
if (this.__ausbossCameraSized) return;
this.__ausbossCameraSized = true;
openRoomy(this);
});
chainCallback(nodeType.prototype, "onConfigure", function (info) {
if (info?.size) this.__ausbossCameraSized = true;
const entry = build(this);
hideWidget(ownCamera(this));
queueMicrotask(() => { entry.face.refresh(); restore(entry); });
holdFloor(this);
});
chainCallback(nodeType.prototype, "onConnectionsChange", function () {
this.__ausbossCameraMove?.face.refresh();
// The camera may have just come up to a box, or gone back inside.
for (const box of boxesOf(this.graph, workflowNodes(rootGraph()))) requestAnimationFrame(() => syncBox(box));
});
chainCallback(nodeType.prototype, "onRemoved", function () {
const entry = this.__ausbossCameraMove;
if (entry) unmount(entry);
this.__ausbossCameraMove = null;
});
},
});
+82
View File
@@ -0,0 +1,82 @@
# Camera Move
Moves the camera of your clip. You steer it on the node's picture, the way
you turn a model in a 3D viewer, and set keys on the time bar. What the old
camera never saw comes out pink, and the CrossView-Warp LoRA for MiniMax H3
paints it in.
## How to use it
1. Load your clip, scale it to the size you will render, and run it through
**Load MoGe Model** and **MoGe Inference** (core ComfyUI nodes).
2. Connect MoGe Inference's geometry to Camera Move.
3. Press **Show my clip** on the node. It runs the depth once and shows your
first frame.
4. Steer:
- Drag the picture to turn the camera round.
- Pinch or scroll to pull back or push in. Two fingers, or Shift-drag, pan.
- Tap a spot to circle round it.
- Top right: drag the axis ball to turn, or tap an axis to look from that
side. Drag the magnifier to zoom and the hand to pan. The camera button
puts you back on your own shot.
5. Keys: click the time bar to put a key there, using the camera at that
moment. Or drag the time bar, then move the camera, and a key appears
there. The diamond adds or removes a key; the arrows either side jump
between keys. Drag a key along the bar to change its time.
⏮ goes back to the start, ▶ plays, and the loop button plays it round
and round.
6. Run the workflow. `guide` goes to the H3 guide at frame 0.
The keys move the camera from where your clip's own camera was. If your
clip's camera already moves (walks down an aisle, for example), that move
stays, and your keys add to it. After the last key the camera holds that
turn, but it still travels along with your clip.
The line under the picture turns amber when a key goes past where the LoRA
was trained: 45° of turn, 30° up, 20° down. Past that it may leave pink.
### Mouse and keys (like Blender)
Middle-drag turns, Shift + middle-drag pans, Ctrl + middle-drag zooms, and
the wheel zooms. On the numpad: 4 6 8 2 turn, Ctrl with them pans, + and −
zoom, 1 is the front, 3 the right side (Ctrl: left), 7 the top (Ctrl:
bottom), 0 your own shot. Space plays and pauses and Home goes back to the
start, while the pointer is over the node.
The wheel over the picture zooms the camera, not the graph. Everywhere else
on the node it zooms the graph as usual.
## Inputs and outputs
| | |
|---|---|
| `moge_geometry` | Depth from MoGe Inference, one per frame. |
| `camera` | The keys, written by the node's picture. You never type here. |
| `guide` | Your clip from the moved camera, pink where nothing was seen. |
| `holes` | White where the guide is pink. |
## Technical details
- Each key holds a moment `t` (0 to 1 over the clip), a turn `az` and a rise
`el` in degrees, `back` (how far the camera pulls back, in distances to the
pivot; below 0 pushes in), and a pan `px`, `py` (same unit). The camera
passes through each key without slowing down, and only eases at the first
and last key. Each value follows a monotone cubic through the keys
(Fritsch-Butland slopes), so it never runs past a key: a value that turns
back or holds at a key stops there.
- The camera turns round the pivot: a spot you tap (saved as a place on one
frame, `pivot: {u, v, t}`) or, by default, the nearest fifth of what sits in
the middle of the first frame, never further than 3 m.
- `R = Ry(-az) · Rx(-el)`, `C = pivot − R·pivot + R·[px·d, py·d, −back·d]`
with `d = |pivot|`, in OpenCV axes (x right, y down, z forward).
- MoGe's depth scale and lens wobble from frame to frame, which shook the
moved picture. Every frame is scaled so its median depth follows a
smoothed line (15 frames wide) through all of them, and every frame uses
the clip's median lens.
- Each pixel is drawn as 4 points, nearest wins, and gaps mostly surrounded
by picture (5 of 8 neighbours, two passes) are filled. Everything else
stays magenta (1, 0, 1).
- The node is an output node so its picture can run it on its own. Each run
writes up to 61 small frames (416 px on the long side) with their depth to
ComfyUI's temp folder; the depth is a PNG with a log scale in red (high
byte) and green (low byte), 0 for no depth.
+20
View File
@@ -1591,5 +1591,25 @@
"AUSBOSS_NODES_WorkflowNoteTile": {
"display_name": "工作流说明卡片 🆎",
"description": "点击打开完整说明的小按钮。点击铅笔编辑,拖动角落调整大小。旧版节点包会将它显示为缺失节点,请更新 ComfyUI-AusBoss。"
},
"AUSBOSS_NODES_CameraMove": {
"display_name": "镜头移动 🆎",
"description": "移动视频的镜头。在节点画面上拖动来转动镜头,捏合或滚轮拉远推近,并在时间轴上设置关键帧。原镜头从未拍到的地方显示为粉色,交给 CrossView-Warp LoRA 补画。需要 MoGe Inference 的深度。",
"inputs": {
"moge_geometry": {
"name": "MoGe 几何"
},
"camera": {
"name": "镜头关键帧"
}
},
"outputs": {
"0": {
"name": "引导"
},
"1": {
"name": "空洞"
}
}
}
}
+428
View File
@@ -0,0 +1,428 @@
"""Camera Move: the camera keys, the camera's pose and the depth warp.
The keys are what the node's face writes: a list of moments on the clip
(t from 0 to 1) and where the camera is at each one. The camera passes
through every key without slowing down, and starts and ends gently. The face draws its picture with the
same maths, so what you steer on the node is what the warp makes.
Axes are OpenCV's: x right, y down, z into the picture. Each frame's points
stay where the original camera saw them; only the camera moves.
Technical details
-----------------
A key holds az (turn, degrees, right is positive), el (rise, degrees, up is
positive), back (pull back, in distances to the pivot; negative pushes in),
px and py (slide right and down, in distances to the pivot).
R = Ry(-az) . Rx(-el)
C = pivot - R.pivot + R.[px.d, py.d, -back.d] d = |pivot|
A point P lands in the new camera at R^T (P - C). The pivot is the spot the
camera circles: picked by a tap on the face, else what sits in the middle
of the first frame.
"""
from __future__ import annotations
import json
import math
LIMITS = {"az": (-90.0, 90.0), "el": (-30.0, 60.0), "back": (-0.6, 2.0), "px": (-1.0, 1.0), "py": (-1.0, 1.0)}
FIELDS = tuple(LIMITS)
HOME = {name: 0.0 for name in FIELDS}
# Where the LoRA was trained to work well. Past this the face warns.
LORA_RANGE = {"az": 45.0, "el_up": 30.0, "el_down": -20.0}
# Two keys within this much of the clip are the same key.
SAME_KEY = 0.015
MAX_KEYS = 64
MAGENTA = (1.0, 0.0, 1.0)
# The nearest thing in the middle of the first frame, never further than this
# (in the depth model's metres): the move stays a move, not a flight.
PIVOT_CAP = 3.0
SCENE_LONG_SIDE = 416
# Every second frame of a 5-second clip, so the face plays smoothly (about 190 KB a frame).
SCENE_FRAMES = 61
DEFAULT_CAMERA = {
"v": 1,
"keys": [{"t": 0.0, **HOME}, {"t": 1.0, **HOME, "az": 30.0, "el": 5.0}],
"pivot": None,
}
def default_camera_text() -> str:
return json.dumps(DEFAULT_CAMERA, separators=(",", ":"))
def _number(value, fallback=0.0) -> float:
try:
number = float(value)
except (TypeError, ValueError):
return fallback
return number if math.isfinite(number) else fallback
def _clamp(value, low, high):
return max(low, min(high, value))
def clean_key(raw) -> dict:
"""One key with every field present, inside its limits."""
raw = raw if isinstance(raw, dict) else {}
key = {"t": _clamp(_number(raw.get("t")), 0.0, 1.0)}
for name, (low, high) in LIMITS.items():
key[name] = _clamp(_number(raw.get(name)), low, high)
return key
def parse_camera(text) -> dict:
"""The camera the face wrote. Anything unreadable falls back to the default move.
Returns {"keys": [...], "pivot": None | {"u", "v", "t"}, "fallback": bool}.
Keys come back sorted by time, with keys at the same moment merged (the
later one wins), and never empty.
"""
data = None
if isinstance(text, dict):
data = text
elif isinstance(text, str) and text.strip():
try:
data = json.loads(text)
except (ValueError, RecursionError):
data = None
fallback = not isinstance(data, dict) or not isinstance(data.get("keys"), list) or not data["keys"]
if fallback:
data = DEFAULT_CAMERA
keys = []
for raw in data["keys"][:MAX_KEYS]:
key = clean_key(raw)
same = next((k for k in keys if abs(k["t"] - key["t"]) < SAME_KEY), None)
if same is not None:
same.update(key)
else:
keys.append(key)
keys.sort(key=lambda k: k["t"])
pivot = data.get("pivot")
if isinstance(pivot, dict) and all(name in pivot for name in ("u", "v")):
pivot = {
"u": _clamp(_number(pivot.get("u"), 0.5), 0.0, 1.0),
"v": _clamp(_number(pivot.get("v"), 0.5), 0.0, 1.0),
"t": _clamp(_number(pivot.get("t"), 0.0), 0.0, 1.0),
}
else:
pivot = None
return {"keys": keys, "pivot": pivot, "fallback": fallback}
def _key_slopes(ts, ys):
"""How fast one field moves as it passes each key.
Zero at the first and last key, so the move starts and ends gently. Zero
at a key where the field turns back or holds, so it never runs past a
key. Elsewhere it keeps its speed through the key (Fritsch-Butland), so
a long move made of several keys does not slow down at each one.
"""
n = len(ys)
slopes = [0.0] * n
for k in range(1, n - 1):
h0, h1 = ts[k] - ts[k - 1], ts[k + 1] - ts[k]
if h0 <= 0 or h1 <= 0:
continue
d0, d1 = (ys[k] - ys[k - 1]) / h0, (ys[k + 1] - ys[k]) / h1
if d0 * d1 > 0:
slopes[k] = 3.0 * (h0 + h1) / ((2.0 * h1 + h0) / d0 + (h1 + 2.0 * h0) / d1)
return slopes
def camera_at(keys, t: float) -> dict:
"""Where the camera is at moment t (0..1): through every key, gently at the ends."""
if not keys:
return dict(HOME)
first, last = keys[0], keys[-1]
if len(keys) == 1 or t <= first["t"]:
return {name: first[name] for name in FIELDS}
if t >= last["t"]:
return {name: last[name] for name in FIELDS}
ts = [k["t"] for k in keys]
for i in range(len(keys) - 1):
a, b = keys[i], keys[i + 1]
if a["t"] <= t <= b["t"]:
h = max(1e-6, b["t"] - a["t"])
s = (t - a["t"]) / h
h00, h10, h01, h11 = 2 * s**3 - 3 * s**2 + 1, s**3 - 2 * s**2 + s, -2 * s**3 + 3 * s**2, s**3 - s**2
out = {}
for name in FIELDS:
m = _key_slopes(ts, [k[name] for k in keys])
out[name] = h00 * a[name] + h10 * h * m[i] + h01 * b[name] + h11 * h * m[i + 1]
return out
return {name: last[name] for name in FIELDS}
def frame_moments(count: int) -> list[float]:
"""The moment (0..1) of each frame. One frame sits at the start."""
if count <= 1:
return [0.0] * max(0, count)
return [i / (count - 1) for i in range(count)]
def rotation(az: float, el: float) -> list[list[float]]:
"""R = Ry(-az) . Rx(-el): az > 0 swings the camera right, el > 0 lifts it."""
th, ph = math.radians(-az), math.radians(-el)
ct, st, cp, sp = math.cos(th), math.sin(th), math.cos(ph), math.sin(ph)
return [[ct, st * sp, st * cp], [0.0, cp, -sp], [-st, ct * sp, ct * cp]]
def _mv(R, v):
return [sum(R[r][c] * v[c] for c in range(3)) for r in range(3)]
def pose(cam: dict, pivot) -> tuple[list[list[float]], list[float]]:
"""The moved camera's rotation R and centre C, for a key and a pivot point."""
R = rotation(cam["az"], cam["el"])
d = math.sqrt(sum(x * x for x in pivot))
Rp = _mv(R, pivot)
b = _mv(R, [cam.get("px", 0.0) * d, cam.get("py", 0.0) * d, -cam["back"] * d])
return R, [pivot[i] - Rp[i] + b[i] for i in range(3)]
def past_lora_range(cam: dict) -> bool:
return abs(cam["az"]) > LORA_RANGE["az"] or cam["el"] > LORA_RANGE["el_up"] or cam["el"] < LORA_RANGE["el_down"]
def range_report(keys) -> str:
"""One plain line when a key goes past where the LoRA was trained, else empty."""
over = [k for k in keys if past_lora_range(k)]
if not over:
return ""
when = ", ".join(f"{k['t']:.2f}" for k in over[:4])
return (
f"Camera Move: {len(over)} key(s) past the LoRA's range (at {when} of the clip). "
"It learned up to 45 degrees of turn, 30 up and 20 down; more may leave pink."
)
def scene_frame_indices(count: int, most: int = SCENE_FRAMES) -> list[int]:
"""Which frames the face gets to look at: evenly spread, first and last included."""
if count <= 0:
return []
if count <= most:
return list(range(count))
return sorted({round(i * (count - 1) / (most - 1)) for i in range(most)})
def scene_size(width: int, height: int, long_side: int = SCENE_LONG_SIDE) -> tuple[int, int]:
scale = min(1.0, long_side / max(width, height))
return max(1, round(width * scale)), max(1, round(height * scale))
def pack_depth(z, valid, zmin: float, zmax: float):
"""Depth as two bytes a pixel (red high, green low) on a log scale; 0 is no depth.
z and valid are numpy arrays. Returns an (H, W, 3) uint8 array.
"""
import numpy as np
lo, hi = math.log(max(zmin, 1e-6)), math.log(max(zmax, zmin * 1.0001, 1e-6))
q = np.zeros(z.shape, dtype=np.uint32)
if valid.any():
zz = np.clip(z[valid], math.exp(lo), math.exp(hi))
q[valid] = 1 + np.round((np.log(zz) - lo) / max(1e-9, hi - lo) * 65534).astype(np.uint32)
out = np.zeros(z.shape + (3,), dtype=np.uint8)
out[..., 0] = (q >> 8) & 255
out[..., 1] = q & 255
return out
def unpack_depth(rgb, zmin: float, zmax: float, far: float):
"""The face's reading of pack_depth, for tests."""
import numpy as np
q = rgb[..., 0].astype(np.uint32) * 256 + rgb[..., 1].astype(np.uint32)
lo, hi = math.log(zmin), math.log(zmax)
z = np.exp(lo + (q.astype(np.float64) - 1) / 65534 * (hi - lo))
return np.where(q == 0, far, z), q > 0
# -- the warp (torch) ---------------------------------------------------------
def pixel_intrinsics(K, width: int, height: int):
"""MoGe's lens as pixels; it may come normalised to the picture size."""
K = K.clone().float()
if K[0, 2] < 4:
K[0, :] *= width
K[1, :] *= height
return K
def steady_depth(z, valid, image, radius: int = 4, colour_tol: float = 0.05):
"""Calm the depth of things that hold still.
The depth model looks at every frame afresh, so a wall that never moves
still gets a slightly different depth each frame, and the moved camera
shakes. A spot whose colour stays the same a few frames either side has
its depth averaged over those frames. Anything that changes in that
window, such as a person moving, keeps its own depth.
"""
import torch
count = z.shape[0]
if count < 2 or radius < 1:
return z
logz = torch.where(valid, z.clamp(min=1e-6).log(), torch.zeros_like(z))
out = z.clone()
inv = 1.0 / (2.0 * colour_tol * colour_tol)
for i in range(count):
lo, hi = max(0, i - radius), min(count, i + radius + 1)
change = ((image[lo:hi] - image[i]) ** 2).sum(-1).amax(0)
held = valid[lo:hi].all(0)
still = torch.exp(-change * inv) * held
mean = logz[lo:hi].sum(0) / (hi - lo)
calm = (still * mean + (1 - still) * logz[i]).exp()
out[i] = torch.where(valid[i] & held, calm, z[i])
return out
def prepare_geometry(geometry: dict, device=None):
"""Points, colours and lens from a MOGE_GEOMETRY, with every frame held to one depth scale.
The depth model's scale and lens wobble from frame to frame, which shakes
the moved picture. Each frame is scaled so its median depth follows a
smoothed line through all of them, every frame gets the clip's median
lens, and the depth of whatever holds still is calmed (steady_depth).
"""
import torch
import torch.nn.functional as F
points = geometry.get("points")
intrinsics = geometry.get("intrinsics")
if points is None or intrinsics is None:
raise ValueError(
"Camera Move needs depth from MoGe Inference with its camera lens. "
"Connect MoGe Inference's geometry, not a panorama."
)
points = points.float()
count, height, width, _ = points.shape
image = geometry["image"][..., :3].float()
if image.shape[1:3] != (height, width):
image = F.interpolate(image.movedim(-1, 1), size=(height, width), mode="bilinear", align_corners=False).movedim(1, -1)
z = points[..., 2]
valid = torch.isfinite(points).all(-1) & (z > 0)
mask = geometry.get("mask")
if mask is not None:
valid &= mask.bool().to(valid.device)
z = torch.where(valid, z, torch.zeros_like(z))
meds = torch.stack([z[i][valid[i]].median() if valid[i].any() else torch.tensor(1.0) for i in range(count)])
win = min(15, count) | 1
padded = F.pad(meds.log()[None, None], (win // 2, win // 2), mode="replicate")
smooth = F.avg_pool1d(padded, win, 1)[0, 0]
z = z * (smooth - meds.log()).exp()[:, None, None]
if device is not None:
z = steady_depth(z.to(device), valid.to(device), image.to(device)).to(z.device)
else:
z = steady_depth(z, valid, image)
# The depth model guesses the lens again for every frame and the guess
# wobbles a few percent, which shakes the warp. One lens for the clip.
K = torch.stack([pixel_intrinsics(intrinsics[i], width, height) for i in range(count)]).median(dim=0).values
Ks = K[None].expand(count, 3, 3).clone()
v0 = valid[0]
far = float(torch.quantile(z[0][v0][::7], 0.98)) * 2.5 if v0.any() else float(meds.median()) * 10.0
return {"z": z, "valid": valid, "image": image, "K": Ks, "far": far, "count": count, "width": width, "height": height}
def unproject(K, x: float, y: float, depth: float) -> list[float]:
fx, fy, cx, cy = float(K[0, 0]), float(K[1, 1]), float(K[0, 2]), float(K[1, 2])
return [(x - cx) / fx * depth, (y - cy) / fy * depth, depth]
def auto_pivot(scene) -> list[float]:
"""What sits in the middle of the first frame: the nearest fifth of the centre box."""
import torch
z0, v0, H, W = scene["z"][0], scene["valid"][0], scene["height"], scene["width"]
box = (slice(int(H * 0.35), int(H * 0.65)), slice(int(W * 0.4), int(W * 0.6)))
vb = v0[box]
if vb.any():
depth = float(torch.quantile(z0[box][vb].float(), 0.2))
elif v0.any():
depth = float(z0[v0].median())
else:
depth = 1.0
return [0.0, 0.0, min(depth, PIVOT_CAP)]
def resolve_pivot(scene, pivot) -> list[float]:
"""The 3D spot the camera circles: the tapped one, else the automatic one."""
if not pivot:
return auto_pivot(scene)
count, H, W = scene["count"], scene["height"], scene["width"]
i = min(count - 1, max(0, round(pivot["t"] * (count - 1))))
x = min(W - 1, max(0, int(pivot["u"] * W)))
y = min(H - 1, max(0, int(pivot["v"] * H)))
z, valid = scene["z"][i], scene["valid"][i]
for r in range(0, 4):
y0, y1, x0, x1 = max(0, y - r), min(H, y + r + 1), max(0, x - r), min(W, x + r + 1)
patch = valid[y0:y1, x0:x1]
if patch.any():
depth = float(z[y0:y1, x0:x1][patch].median())
return unproject(scene["K"][i], x + 0.5, y + 0.5, depth)
return auto_pivot(scene)
def _fill_cracks(img, filled, passes=2):
"""Fill one-pixel gaps that are mostly surrounded by picture."""
import torch
import torch.nn.functional as F
k = torch.ones(1, 1, 3, 3, device=img.device)
for _ in range(passes):
f = filled.float()[None, None]
cnt = F.conv2d(f, k, padding=1)[0, 0]
s = F.conv2d((img * filled.float())[:, None], k, padding=1)[:, 0]
take = (~filled) & (cnt >= 5)
if not take.any():
break
img = torch.where(take[None], s / cnt.clamp(min=1), img)
filled = filled | take
return img, filled
def warp_frame(scene, i: int, cam: dict, pivot, device):
"""One frame seen from the moved camera. Returns (colour (H,W,3), hole (H,W) bool) on the CPU."""
import torch
H, W = scene["height"], scene["width"]
K = scene["K"][i].to(device)
Kinv = torch.linalg.inv(K)
valid = scene["valid"][i].to(device)
zi = torch.where(valid, scene["z"][i].to(device), torch.full((H, W), scene["far"], device=device))
col = scene["image"][i].to(device)
sub = torch.tensor([[-0.25, -0.25], [0.25, -0.25], [-0.25, 0.25], [0.25, 0.25]], device=device)
vv, uu = torch.meshgrid(torch.arange(H, device=device, dtype=torch.float32), torch.arange(W, device=device, dtype=torch.float32), indexing="ij")
u = (uu[None] + 0.5 + sub[:, 0, None, None]).reshape(-1)
v = (vv[None] + 0.5 + sub[:, 1, None, None]).reshape(-1)
zz = zi[None].expand(4, H, W).reshape(-1)
cc = col[None].expand(4, H, W, 3).reshape(-1, 3)
P = (torch.stack([u, v, torch.ones_like(u)], -1) @ Kinv.T) * zz[:, None]
R_list, C_list = pose(cam, pivot)
R = torch.tensor(R_list, dtype=torch.float32, device=device)
C = torch.tensor(C_list, dtype=torch.float32, device=device)
Pc = (P - C) @ R # R^T (P - C), row vectors
zc = Pc[:, 2]
ok = zc > 1e-3
uv = Pc[:, :2] / zc.clamp(min=1e-3)[:, None]
up = K[0, 0] * uv[:, 0] + K[0, 2]
vp = K[1, 1] * uv[:, 1] + K[1, 2]
ui, vi = up.floor().long(), vp.floor().long()
ok &= (ui >= 0) & (ui < W) & (vi >= 0) & (vi < H)
lin, zc, cc = vi[ok] * W + ui[ok], zc[ok], cc[ok]
zbuf = torch.full((H * W,), float("inf"), device=device)
zbuf.scatter_reduce_(0, lin, zc, reduce="amin", include_self=True)
win = zc <= zbuf[lin] * (1 + 1e-5)
magenta = torch.tensor(MAGENTA, device=device).view(3, 1, 1)
frame = magenta.expand(3, H, W).clone()
frame.view(3, -1)[:, lin[win]] = cc[win].T
filled = torch.isfinite(zbuf).view(H, W)
frame, filled = _fill_cracks(frame, filled)
frame = torch.where(filled[None], frame, magenta)
return frame.movedim(0, -1).clamp(0, 1).cpu(), (~filled).cpu()
+170
View File
@@ -0,0 +1,170 @@
"""Camera Move 🆎: move the camera of a clip, steered with keys on the node."""
from __future__ import annotations
from pathlib import Path
from ._camera_move_helpers import (
camera_at,
default_camera_text,
frame_moments,
pack_depth,
parse_camera,
prepare_geometry,
range_report,
resolve_pivot,
auto_pivot,
scene_frame_indices,
scene_size,
warp_frame,
)
UI_KEY = "ausboss_camera_scene"
def _device():
try:
import comfy.model_management as mm
return mm.get_torch_device()
except Exception:
import torch
return torch.device("cuda" if torch.cuda.is_available() else "cpu")
class AusBossCameraMove:
CATEGORY = "🆎 AusBoss/Video"
DESCRIPTION = (
"Moves the camera of your clip. Steer it on the node: drag the picture "
"to turn, pinch or scroll to pull back, and set keys on the time bar. "
"Places the old camera never saw come out pink, ready for the "
"CrossView-Warp LoRA to paint in. Needs depth from MoGe Inference."
)
SEARCH_ALIASES = [
"camera move",
"move camera",
"orbit camera",
"camera orbit video",
"crossview warp",
"novel view",
"dolly",
"camera keyframes",
"ausboss",
]
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"moge_geometry": (
"MOGE_GEOMETRY",
{"tooltip": "Depth of your clip from MoGe Inference. Every frame needs it."},
),
"camera": (
"STRING",
{
"default": default_camera_text(),
"multiline": False,
"tooltip": "The camera keys. Set them on the node's picture; you never type here.",
},
),
}
}
RETURN_TYPES = ("IMAGE", "MASK")
RETURN_NAMES = ("guide", "holes")
OUTPUT_TOOLTIPS = (
"Your clip seen from the moved camera. Pink is what the old camera never saw.",
"White where the guide is pink: the parts to paint in.",
)
OUTPUT_NODE = True
FUNCTION = "run"
def run(self, moge_geometry, camera):
import torch
parsed = parse_camera(camera)
if parsed["fallback"]:
print("[AusBoss] Camera Move: could not read the camera keys, using the default move.")
report = range_report(parsed["keys"])
if report:
print("[AusBoss] " + report)
device = _device()
scene = prepare_geometry(moge_geometry, device)
pivot = resolve_pivot(scene, parsed["pivot"])
try:
import comfy.utils
bar = comfy.utils.ProgressBar(scene["count"])
except Exception:
bar = None
frames, holes = [], []
with torch.no_grad():
for i, t in enumerate(frame_moments(scene["count"])):
frame, hole = warp_frame(scene, i, camera_at(parsed["keys"], t), pivot, device)
frames.append(frame)
holes.append(hole)
if bar is not None:
bar.update(1)
guide = torch.stack(frames)
mask = torch.stack(holes).float()
info = self._scene(scene, pivot)
info["pivot_from"] = parsed["pivot"]
return {"ui": {UI_KEY: [info]}, "result": (guide, mask)}
# -- what the node's face looks at ----------------------------------------
def _scene(self, scene, pivot):
"""A few frames, small, with their depth, written to ComfyUI's temp folder."""
import folder_paths
import numpy as np
from PIL import Image
W, H, count = scene["width"], scene["height"], scene["count"]
w, h = scene_size(W, H)
ys = np.clip(((np.arange(h) + 0.5) * H / h).astype(int), 0, H - 1)
xs = np.clip(((np.arange(w) + 0.5) * W / w).astype(int), 0, W - 1)
picks = scene_frame_indices(count)
small = []
for i in picks:
z = scene["z"][i].cpu().numpy()[ys][:, xs]
valid = scene["valid"][i].cpu().numpy()[ys][:, xs]
small.append((i, z, valid))
every = np.concatenate([z[v] for _, z, v in small if v.any()]) if any(v.any() for _, _, v in small) else np.array([1.0])
zmin, zmax = float(every.min()), float(max(every.max(), every.min() * 1.001))
folder, name, counter, subfolder, _ = folder_paths.get_save_image_path(
"ausboss_camera_move", folder_paths.get_temp_directory(), w, h
)
frames = []
for j, (i, z, valid) in enumerate(small):
colour = (scene["image"][i].cpu().numpy().clip(0, 1) * 255).astype(np.uint8)
stem = f"{name}_{counter:05}_{j:02}"
Image.fromarray(colour).resize((w, h), Image.LANCZOS).save(Path(folder) / f"{stem}_c.jpg", quality=88)
Image.fromarray(pack_depth(z, valid, zmin, zmax)).save(Path(folder) / f"{stem}_d.png", compress_level=1)
K = scene["K"][i]
frames.append({
"i": i,
"t": i / (count - 1) if count > 1 else 0.0,
"color": {"filename": f"{stem}_c.jpg", "subfolder": subfolder, "type": "temp"},
"depth": {"filename": f"{stem}_d.png", "subfolder": subfolder, "type": "temp"},
"fx": float(K[0, 0]) * w / W,
"fy": float(K[1, 1]) * h / H,
"cx": float(K[0, 2]) * w / W,
"cy": float(K[1, 2]) * h / H,
})
return {
"w": w,
"h": h,
"count": count,
"zmin": zmin,
"zmax": zmax,
"far": scene["far"],
"pivot": [float(x) for x in pivot],
"pivot_auto": [float(x) for x in auto_pivot(scene)],
"frames": frames,
}
NODE_CLASS_MAPPINGS = {"AUSBOSS_NODES_CameraMove": AusBossCameraMove}
NODE_DISPLAY_NAME_MAPPINGS = {"AUSBOSS_NODES_CameraMove": "Camera Move 🆎"}
+1
View File
@@ -52,6 +52,7 @@ PUBLIC_NODE_IDS = {
"AUSBOSS_NODES_AlignImage",
"AUSBOSS_NODES_Callout",
"AUSBOSS_NODES_CalloutTile",
"AUSBOSS_NODES_CameraMove",
"AUSBOSS_NODES_ColorMatch",
"AUSBOSS_NODES_Compare",
"AUSBOSS_NODES_CropForInpaint",
+94
View File
@@ -0,0 +1,94 @@
// Camera Move's face maths must match the node's (nodes/_camera_move_helpers.py).
import assert from "node:assert/strict";
import test from "node:test";
import {
DEFAULT_KEYS, K0, LIM, camAt, cameraText, keyAt, parseCamera, pastLoraRange, pivotFromPixel, pose, setKeyAt, toCam, unpackDepth,
} from "../js/camera_move/camera_math.mjs";
const close = (a, b, eps = 1e-9) => assert.ok(Math.abs(a - b) < eps, `${a} != ${b}`);
test("pose matches the node's numbers", () => {
// From nodes/_camera_move_helpers.pose with the same key and pivot.
const { R, C } = pose({ az: 33, el: -12, back: 0.4, px: 0.2, py: -0.1 }, [0.3, -0.2, 2.0]);
const R_py = [0.838670567945424, -0.11323682265532711, -0.532737365365924, 0, 0.9781476007338057, -0.20791169081775934,
0.5446390350150271, 0.17436941582062362, 0.8203436038418747];
R.forEach((v, i) => close(v, R_py[i]));
[1.8881747404128695, 0.3816804134905334, -0.250127706778845].forEach((v, i) => close(C[i], v));
});
test("easing between keys matches the node's", () => {
const { keys } = parseCamera({ keys: [{ t: 0 }, { t: 0.4, az: 20, el: 10, back: 0.5 }, { t: 1, az: -10, px: 0.3 }] });
const c = camAt(keys, 0.3);
close(c.az, 16.875); close(c.el, 8.4375); close(c.back, 0.421875);
});
test("the camera keeps moving through a key, the same as the node", () => {
const { keys } = parseCamera({ keys: [{ t: 0 }, { t: 0.3, az: 12 }, { t: 0.6, az: 25, el: 5 }, { t: 1, az: 40, el: 5, back: 0.4 }] });
// From nodes/_camera_move_helpers.camera_at with the same keys.
close(camAt(keys, 0.45).az, 18.547068965517244); close(camAt(keys, 0.8).az, 34.51724137931035);
close(camAt(keys, 0.8).back, 0.2);
const steady = parseCamera({ keys: [{ t: 0 }, { t: 0.5, az: 20 }, { t: 1, az: 40 }] }).keys;
assert.ok((camAt(steady, 0.51).az - camAt(steady, 0.49).az) / 0.02 > 35);
});
test("the pivot stays where it was on screen", () => {
const pivot = [0.3, -0.2, 2], d = Math.hypot(...pivot);
const cam = { az: 40, el: 15, back: 0.5, px: 0, py: 0 };
const { R, C } = pose(cam, pivot);
const q = toCam(R, C, pivot);
close(q[0], 0.3); close(q[1], -0.2); close(q[2], 2 + 0.5 * d);
});
test("unreadable camera text falls back to the default move", () => {
for (const text of ["", "nope", "[]", '{"keys":[]}', null]) {
const parsed = parseCamera(text);
assert.equal(parsed.fallback, true);
assert.equal(parsed.keys.length, DEFAULT_KEYS.length);
}
});
test("camera text round-trips, clamped and sorted", () => {
const parsed = parseCamera({ keys: [{ t: 0.9, az: 300 }, { t: 0.1, el: -90, back: 5 }], pivot: { u: 0.25, v: 2, t: 0.5 } });
assert.deepEqual(parsed.keys.map((k) => k.t), [0.1, 0.9]);
assert.equal(parsed.keys[1].az, LIM.az[1]);
assert.equal(parsed.keys[0].el, LIM.el[0]);
assert.deepEqual(parsed.pivot, { u: 0.25, v: 1, t: 0.5 });
const again = parseCamera(cameraText(parsed));
assert.deepEqual(again.keys, parsed.keys);
assert.deepEqual(again.pivot, parsed.pivot);
});
test("moving the camera at the time bar sets or adds a key", () => {
const keys = [{ t: 0, ...K0 }, { t: 1, ...K0 }];
setKeyAt(keys, 0.5, { az: 10 });
assert.equal(keys.length, 3);
assert.equal(keys[1].t, 0.5);
setKeyAt(keys, 0.505, { az: 99 });
assert.equal(keys.length, 3);
assert.equal(keys[1].az, 90);
assert.equal(keyAt(keys, 0.51), 1);
assert.equal(keyAt(keys, 0.6), -1);
});
test("the LoRA's range", () => {
assert.equal(pastLoraRange({ az: 45, el: 30 }), false);
assert.equal(pastLoraRange({ az: 46, el: 0 }), true);
assert.equal(pastLoraRange({ az: 0, el: -21 }), true);
});
test("the depth picture reads back as the node wrote it", () => {
// q = 1 is zmin, q = 65535 is zmax, 0 is sky.
const rgba = new Uint8ClampedArray([0, 1, 0, 255, 255, 255, 0, 255, 0, 0, 0, 255]);
const { z, valid } = unpackDepth(rgba, 3, 0.5, 40, 99);
close(z[0], 0.5, 1e-6); close(z[1], 40, 1e-4); assert.equal(z[2], 99);
assert.deepEqual([...valid], [1, 1, 0]);
});
test("a tapped pixel becomes a point in front of the camera", () => {
const W = 4, H = 2, z = new Float32Array([2, 2, 2, 2, 2, 2, 2, 2]), valid = new Uint8Array([0, 0, 0, 0, 0, 1, 0, 0]);
const frame = { W, H, z, valid, fx: 2, fy: 2, cx: 2, cy: 1 };
const p = pivotFromPixel(frame, 0.1, 0.9);
assert.ok(p);
close(p[2], 2);
assert.equal(pivotFromPixel({ ...frame, valid: new Uint8Array(8) }, 0.5, 0.5), null);
});
+1
View File
@@ -108,6 +108,7 @@ test("only the listed parts of the pack listen for the wheel", () => {
const js = join(dirname(fileURLToPath(import.meta.url)), "..", "js");
const listed = [
"callout/index.js wrap", // the text box while a note is being edited
"camera_move/face.mjs stage", // the shot: the wheel pulls the camera back or in
"canvas_passthrough/index.js window", // hands the wheel to the graph
"image_folder/index.js root", // the copy question, a window over the page
"image_folder/index.js root", // the folder list, a window over the page
+17
View File
@@ -40,6 +40,23 @@
"optional": [],
"outputs": []
},
"AUSBOSS_NODES_CameraMove": {
"required": [
"moge_geometry",
"camera"
],
"optional": [],
"outputs": [
[
"IMAGE",
"guide"
],
[
"MASK",
"holes"
]
]
},
"AUSBOSS_NODES_ColorMatch": {
"required": [
"image",
+6 -1
View File
@@ -28,6 +28,8 @@ const JS_ROOT = join(dirname(fileURLToPath(import.meta.url)), "..", "js");
// A panel whose DOM is built by a shared module, so a second view can reuse
// it, is audited where it is built.
const SHARED_PANELS = { mask_by_touch: join("shared", "touch_stage.mjs") };
// A panel whose face lives in its own module beside index.js is audited with it.
const PANEL_PARTS = { camera_move: [join("camera_move", "face.mjs")] };
function domWidgetEntries() {
const entries = [];
@@ -38,7 +40,8 @@ function domWidgetEntries() {
} catch {
continue;
}
const source = readFileSync(path, "utf-8");
const source = [path, ...(PANEL_PARTS[name] ?? []).map((part) => join(JS_ROOT, part))]
.map((file) => readFileSync(file, "utf-8")).join("\n");
if (source.includes("addDOMWidget")) entries.push({ name, source });
}
return entries;
@@ -110,6 +113,7 @@ test("a panel that should follow the node's height never declares computeSize",
// is this a stage that should grow, or a fixed row?
const mustGrow = new Set([
"callout",
"camera_move",
"resolution",
"compare",
"image_crop_rotate_pad",
@@ -397,6 +401,7 @@ test("every panel root class carries border-box and an overflow clip", () => {
show_text: ".ausboss-show-text{",
step_preview: ".ausboss-steppv{",
callout: ".ausboss-callout{",
camera_move: ".acm-face{",
workflow_note: ".ausboss-note{",
seed: ".ausboss-seed{",
run_timer: ".ausboss-timer{",
+319
View File
@@ -0,0 +1,319 @@
"""Offline tests for Camera Move: keys, pose, depth packing and the warp."""
from __future__ import annotations
import json
import math
import sys
import tempfile
import types
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT))
if "nodes" in sys.modules and not hasattr(sys.modules["nodes"], "__path__"):
del sys.modules["nodes"]
from nodes._camera_move_helpers import (
DEFAULT_CAMERA,
LIMITS,
camera_at,
default_camera_text,
frame_moments,
pack_depth,
parse_camera,
past_lora_range,
pose,
range_report,
rotation,
scene_frame_indices,
scene_size,
unpack_depth,
)
try:
import numpy as np
import torch
except ImportError: # pragma: no cover - the warp tests need ComfyUI's Python
np = torch = None
def _apply(R, v):
return [sum(R[r][c] * v[c] for c in range(3)) for r in range(3)]
def _apply_t(R, v):
return [sum(R[c][r] * v[c] for c in range(3)) for r in range(3)]
class ParseCameraTests(unittest.TestCase):
def test_default_text_reads_back_as_the_default(self):
parsed = parse_camera(default_camera_text())
self.assertFalse(parsed["fallback"])
self.assertEqual([k["t"] for k in parsed["keys"]], [0.0, 1.0])
self.assertEqual(parsed["keys"][1]["az"], DEFAULT_CAMERA["keys"][1]["az"])
self.assertIsNone(parsed["pivot"])
def test_unreadable_text_falls_back_to_the_default_move(self):
for text in ("", "not json", "[]", '{"keys": []}', '{"keys": "x"}', None, 12):
parsed = parse_camera(text)
self.assertTrue(parsed["fallback"], text)
self.assertEqual(len(parsed["keys"]), 2)
def test_keys_are_clamped_sorted_and_merged(self):
text = json.dumps({"keys": [
{"t": 0.9, "az": 500, "el": -99, "back": 9},
{"t": 0.1, "az": "12", "px": -5},
{"t": 0.905, "az": 20},
{"t": 7, "el": float("nan")},
]})
keys = parse_camera(text)["keys"]
self.assertEqual([round(k["t"], 3) for k in keys], [0.1, 0.905, 1.0])
self.assertEqual(keys[0]["az"], 12.0)
self.assertEqual(keys[0]["px"], LIMITS["px"][0])
# The later key at the same moment wins.
self.assertEqual(keys[1]["az"], 20.0)
self.assertEqual(keys[2]["el"], 0.0)
for key in keys:
for name, (low, high) in LIMITS.items():
self.assertTrue(low <= key[name] <= high)
def test_pivot_is_kept_inside_the_picture(self):
parsed = parse_camera({"keys": [{"t": 0}], "pivot": {"u": 1.4, "v": -1, "t": 0.5}})
self.assertEqual(parsed["pivot"], {"u": 1.0, "v": 0.0, "t": 0.5})
self.assertIsNone(parse_camera({"keys": [{"t": 0}], "pivot": {"u": 0.2}})["pivot"])
class CameraAtTests(unittest.TestCase):
keys = [{"t": 0.0, "az": 0, "el": 0, "back": 0, "px": 0, "py": 0},
{"t": 0.5, "az": 40, "el": 10, "back": 1, "px": 0, "py": 0},
{"t": 1.0, "az": 40, "el": 10, "back": 1, "px": 0.5, "py": 0}]
def test_holds_before_the_first_and_after_the_last_key(self):
self.assertEqual(camera_at(self.keys, -1)["az"], 0)
self.assertEqual(camera_at(self.keys, 2)["px"], 0.5)
def test_eases_between_keys(self):
self.assertAlmostEqual(camera_at(self.keys, 0.25)["az"], 20.0)
self.assertAlmostEqual(camera_at(self.keys, 0.125)["az"], 40 * (0.25 ** 2 * 2.5))
self.assertEqual(camera_at(self.keys, 0.5)["az"], 40)
def test_one_key_holds_still(self):
self.assertEqual(camera_at(self.keys[1:2], 0.0)["back"], 1)
def test_keeps_moving_through_a_key(self):
keys = parse_camera({"keys": [{"t": 0}, {"t": 0.5, "az": 20}, {"t": 1, "az": 40}]})["keys"]
speed = (camera_at(keys, 0.51)["az"] - camera_at(keys, 0.49)["az"]) / 0.02
self.assertGreater(speed, 35) # the steady speed is 40 a clip; easing at each key gave about 0
self.assertAlmostEqual(camera_at(keys, 0.5)["az"], 20)
def test_never_runs_past_a_key(self):
keys = parse_camera({"keys": [{"t": 0}, {"t": 0.3, "az": 30}, {"t": 0.5, "az": 30}, {"t": 1, "az": -20}]})["keys"]
values = [camera_at(keys, i / 200)["az"] for i in range(201)]
self.assertLessEqual(max(values), 30 + 1e-9)
self.assertGreaterEqual(min(values), -20 - 1e-9)
self.assertTrue(all(abs(camera_at(keys, t)["az"] - 30) < 1e-9 for t in (0.3, 0.4, 0.5))) # two equal keys hold still
def test_matches_the_face(self):
keys = parse_camera({"keys": [{"t": 0}, {"t": 0.3, "az": 12}, {"t": 0.6, "az": 25, "el": 5},
{"t": 1, "az": 40, "el": 5, "back": 0.4}]})["keys"]
self.assertAlmostEqual(camera_at(keys, 0.45)["az"], 18.547068965517244) # tests/camera_move.test.mjs
self.assertAlmostEqual(camera_at(keys, 0.8)["az"], 34.51724137931035)
def test_frame_moments_run_from_start_to_end(self):
self.assertEqual(frame_moments(1), [0.0])
self.assertEqual(frame_moments(3), [0.0, 0.5, 1.0])
class PoseTests(unittest.TestCase):
pivot = [0.3, -0.2, 2.0]
def test_rotation_is_a_rotation(self):
R = rotation(33, -17)
for r in range(3):
for c in range(3):
dot = sum(R[k][r] * R[k][c] for k in range(3))
self.assertAlmostEqual(dot, 1.0 if r == c else 0.0)
def test_turning_right_moves_the_camera_right(self):
_, C = pose({"az": 30, "el": 0, "back": 0, "px": 0, "py": 0}, [0, 0, 2])
self.assertGreater(C[0], 0)
_, C = pose({"az": 0, "el": 20, "back": 0, "px": 0, "py": 0}, [0, 0, 2])
self.assertLess(C[1], 0) # y is down: rising is negative y
def test_home_is_the_original_camera(self):
R, C = pose({"az": 0, "el": 0, "back": 0, "px": 0, "py": 0}, self.pivot)
self.assertEqual([[round(x, 9) for x in row] for row in R], [[1, 0, 0], [0, 1, 0], [0, 0, 1]])
for x in C:
self.assertAlmostEqual(x, 0.0)
def test_the_pivot_stays_where_it_was_on_screen(self):
d = math.sqrt(sum(x * x for x in self.pivot))
for cam in ({"az": 40, "el": 15, "back": 0.5}, {"az": -70, "el": -25, "back": -0.4}):
cam = {**cam, "px": 0, "py": 0}
R, C = pose(cam, self.pivot)
seen = _apply_t(R, [self.pivot[i] - C[i] for i in range(3)])
# Where the original camera saw it, pushed back along the view by back * d.
self.assertAlmostEqual(seen[0], self.pivot[0])
self.assertAlmostEqual(seen[1], self.pivot[1])
self.assertAlmostEqual(seen[2], self.pivot[2] + cam["back"] * d)
def test_pan_slides_the_camera_in_its_own_frame(self):
d = math.sqrt(sum(x * x for x in self.pivot))
R, C0 = pose({"az": 25, "el": 5, "back": 0, "px": 0, "py": 0}, self.pivot)
_, C1 = pose({"az": 25, "el": 5, "back": 0, "px": 0.5, "py": 0}, self.pivot)
moved = _apply_t(R, [C1[i] - C0[i] for i in range(3)])
self.assertAlmostEqual(moved[0], 0.5 * d)
self.assertAlmostEqual(moved[1], 0.0)
self.assertAlmostEqual(moved[2], 0.0)
def test_lora_range(self):
self.assertFalse(past_lora_range({"az": 45, "el": 30}))
self.assertTrue(past_lora_range({"az": -46, "el": 0}))
self.assertTrue(past_lora_range({"az": 0, "el": -21}))
self.assertEqual(range_report(CameraAtTests.keys), "")
report = range_report([{"t": 0.5, "az": 60, "el": 0}])
self.assertIn("past the LoRA's range", report)
report.encode("ascii")
class SceneTests(unittest.TestCase):
def test_frames_for_the_face(self):
self.assertEqual(scene_frame_indices(0), [])
self.assertEqual(scene_frame_indices(5), [0, 1, 2, 3, 4])
picks = scene_frame_indices(121)
self.assertEqual((picks[0], picks[-1], len(picks)), (0, 120, 61))
self.assertEqual(picks[:3], [0, 2, 4])
def test_scene_size_keeps_the_shape(self):
self.assertEqual(scene_size(832, 480), (416, 240))
self.assertEqual(scene_size(300, 200), (300, 200))
@unittest.skipIf(np is None, "needs numpy")
def test_depth_packing_round_trips(self):
rng = np.random.default_rng(1)
z = rng.uniform(0.4, 60.0, (24, 32))
valid = rng.uniform(size=z.shape) > 0.2
packed = pack_depth(z, valid, float(z[valid].min()), float(z[valid].max()))
self.assertEqual(packed.dtype, np.uint8)
back, ok = unpack_depth(packed, float(z[valid].min()), float(z[valid].max()), 999.0)
self.assertTrue((ok == valid).all())
self.assertLess(float(np.abs(back[valid] / z[valid] - 1).max()), 1e-4)
self.assertTrue((back[~valid] == 999.0).all())
def _wall_and_box(frames=3, H=48, W=64):
"""A wall 4 m away with a box 2 m away in the middle, as MoGe would hand it over."""
f = 60.0
K = torch.tensor([[f, 0, W / 2], [0, f, H / 2], [0, 0, 1.0]])
vv, uu = torch.meshgrid(torch.arange(H) + 0.5, torch.arange(W) + 0.5, indexing="ij")
z = torch.full((H, W), 4.0)
z[H // 3: 2 * H // 3, W // 3: 2 * W // 3] = 2.0
pts = torch.stack([(uu - W / 2) / f * z, (vv - H / 2) / f * z, z], -1)
image = torch.zeros(H, W, 3)
image[..., 0] = uu / W
image[..., 1] = vv / H
image[..., 2] = 0.2
return {
"points": pts[None].expand(frames, H, W, 3).clone(),
"intrinsics": K[None].expand(frames, 3, 3).clone(),
"image": image[None].expand(frames, H, W, 3).clone(),
"mask": torch.ones(frames, H, W, dtype=torch.bool),
}
@unittest.skipIf(torch is None, "needs torch")
class WarpTests(unittest.TestCase):
def setUp(self):
from nodes._camera_move_helpers import auto_pivot, prepare_geometry, resolve_pivot, warp_frame
self.prepare, self.auto, self.resolve, self.warp = prepare_geometry, auto_pivot, resolve_pivot, warp_frame
self.scene = prepare_geometry(_wall_and_box())
def test_home_gives_back_the_clip(self):
home = {"az": 0, "el": 0, "back": 0, "px": 0, "py": 0}
frame, hole = self.warp(self.scene, 0, home, self.auto(self.scene), torch.device("cpu"))
self.assertEqual(int(hole.sum()), 0)
self.assertLess(float((frame - self.scene["image"][0]).abs().max()), 1e-4)
def test_a_turn_opens_pink_holes_beside_the_box(self):
cam = {"az": 25, "el": 0, "back": 0, "px": 0, "py": 0}
frame, hole = self.warp(self.scene, 0, cam, self.auto(self.scene), torch.device("cpu"))
self.assertGreater(int(hole.sum()), 10)
pink = frame[hole]
self.assertTrue(torch.allclose(pink, torch.tensor([1.0, 0.0, 1.0]).expand_as(pink)))
def test_auto_pivot_is_the_box(self):
self.assertAlmostEqual(self.auto(self.scene)[2], 2.0, places=4)
def test_a_tapped_pivot_is_the_spot_tapped(self):
on_wall = self.resolve(self.scene, {"u": 0.05, "v": 0.05, "t": 0.0})
self.assertAlmostEqual(on_wall[2], 4.0, places=4)
self.assertLess(on_wall[0], 0)
self.assertLess(on_wall[1], 0)
def test_one_steady_lens_and_depth_for_the_clip(self):
geometry = _wall_and_box(frames=12)
wobble = torch.tensor([1.0, 1.06, 0.95, 1.04, 0.97, 1.05, 0.94, 1.03, 0.96, 1.05, 0.95, 1.0])
geometry["points"] = geometry["points"] * wobble[:, None, None, None]
geometry["intrinsics"] = geometry["intrinsics"].clone()
geometry["intrinsics"][:, 0, 0] *= wobble
scene = self.prepare(geometry)
self.assertTrue(torch.equal(scene["K"][0], scene["K"][7]))
meds = torch.stack([scene["z"][i][scene["valid"][i]].median() for i in range(12)])
jumps = (meds[1:] / meds[:-1] - 1).abs()
self.assertLess(float(jumps.max()), 0.02) # the clip wobbles up to 11% frame to frame
def test_calms_still_depth_and_leaves_a_mover_alone(self):
from nodes._camera_move_helpers import steady_depth
frames, H, W = 9, 8, 8
z = torch.full((frames, H, W), 4.0)
z[:, :, :4] *= torch.tensor([1.0, 1.08, 0.93, 1.05, 0.95, 1.07, 0.94, 1.04, 1.0])[:, None, None] # wall flicker
image = torch.full((frames, H, W, 3), 0.5)
image[4, :, 6:] = torch.tensor([0.9, 0.2, 0.1]) # someone passes on the right in frame 4
z[4, :, 6:] = 2.0
out = steady_depth(z, torch.ones(frames, H, W, dtype=torch.bool), image)
self.assertLess(float((out[4, :, :4] / out[3, :, :4] - 1).abs().max()), 0.03) # was 12% between frames
self.assertTrue(torch.allclose(out[4, :, 6:], torch.tensor(2.0)))
def test_needs_the_lens(self):
with self.assertRaises(ValueError):
self.prepare({"image": torch.zeros(1, 4, 4, 3), "depth": torch.ones(1, 4, 4)})
@unittest.skipIf(torch is None, "needs torch")
class NodeTests(unittest.TestCase):
def test_run_returns_guide_holes_and_the_scene(self):
temp = tempfile.mkdtemp()
stub = types.ModuleType("folder_paths")
stub.get_temp_directory = lambda: temp
def get_save_image_path(prefix, folder, w, h):
return folder, prefix, 1, "", prefix
stub.get_save_image_path = get_save_image_path
sys.modules["folder_paths"] = stub
import nodes.node_camera_move as node_module
from nodes.node_camera_move import UI_KEY, AusBossCameraMove
node_module._device = lambda: torch.device("cpu")
out = AusBossCameraMove().run(_wall_and_box(frames=4), default_camera_text())
guide, holes = out["result"]
self.assertEqual(tuple(guide.shape), (4, 48, 64, 3))
self.assertEqual(tuple(holes.shape), (4, 48, 64))
self.assertEqual(float(holes[0].sum()), 0.0) # the first key is your own shot
self.assertGreater(float(holes[-1].sum()), 0.0)
info = out["ui"][UI_KEY][0]
self.assertEqual((info["w"], info["h"], info["count"]), (64, 48, 4))
self.assertEqual(len(info["frames"]), 4)
for frame in info["frames"]:
self.assertTrue((Path(temp) / frame["color"]["filename"]).is_file())
self.assertTrue((Path(temp) / frame["depth"]["filename"]).is_file())
json.dumps(info)
if __name__ == "__main__":
unittest.main()