From a0c11c1dd84fe75475f52c08c96e0633ec126830 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 13:21:07 +0000 Subject: [PATCH] Upscale runs as a FunUI background job too (no extra ComfyUI prompt that makes the cache drop the workflow's models) Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Tw9fjW75aYB8UyVRZo6fCr --- CHANGELOG.md | 3 +- app/ui/features/interpolate/job.js | 20 ++------- app/ui/features/upscale/job.js | 4 +- app/ui/features/upscale/job.test.js | 18 ++++---- app/ui/shell/api.js | 30 ++----------- app/ui/shell/renders.js | 23 +++++----- modules/output/render/__init__.py | 15 +++++++ modules/output/render/tests/test_upscale.py | 14 ++++++ modules/output/render/upscale.py | 47 ++++++++++++--------- 9 files changed, 89 insertions(+), 85 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9a439b08..9e4b2f9b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ ### Added +- **Upscale no longer makes the next generation reload the models.** It ran as an extra ComfyUI prompt without the workflow's loaders, and ComfyUI's cache lets go of loaders a prompt does not use; it now runs as a background job of FunUI, beside ComfyUI's queue (Interrupt in ComfyUI no longer stops it). - **Interpolate** (module of its own, add-on **NVIDIA RTX Frame Generation**): adds in-between frames to a finished render with NVIDIA's RTX frame generation (2 to 4 times the frames; the length and the sound stay, 24 fps becomes 48 fps). Settings > Editor > Interpolate: Never, a button beside the rating, or always when a render finishes. Quality Fast, Balanced or Best, and 10-bit processing. It runs as a background job of FunUI, not as a ComfyUI prompt, so it does not make ComfyUI drop the workflow's loaded models (an extra prompt without loaders does exactly that, and the next generation reloads them); the NVIDIA model also stays loaded between presses. Works beside Upscale in either order and never runs twice on the same file. Needs an RTX card and nvidia-vfx 0.2 or newer: switching the add-on on moves an older nvidia-vfx up (nothing else is touched), and Super Resolution never moves a newer one back. ComfyUI's own RIFE/FILM interpolation nodes are a different thing and are not used. Works on a rental GPU (a few seconds for a clip). - **Project settings > Resolution is the** (H3, LTX image to video): what Width and Height mean once the scene picture sets the shape. Short side (the smaller number is the clip's shorter side), Long side (the larger is its longer side) or Pixel count (the old rule: about Width x Height pixels, which turned 1024x1024 into 1312x768 for a wide picture). New projects start on Short side; projects saved earlier keep Pixel count, so nothing changes size by itself. The H3 preset no longer needs the KJNodes ImageTransformKJ node for this (saved H3 pipelines are switched over on load). - **VAE Loader: Decode-only VAE (Optional)**: a second VAE used only for the final picture decode while the first still encodes (references, start pictures). For decoders without an encoder: MiniMax H3's Sol-H3-VAE (`sol_h3_vae.safetensors`, whole clip at once, about 2.5x faster decode, needs a ComfyUI newer than 0.43.1) and the X2 Detail VAE. @@ -132,7 +133,7 @@ A rewrite from scratch, not an update of 4.x. Same idea (a studio for video insi - **Experimental**: continuity, identity transfer, negative erase, camera move, context windows, block influence, block repeat, phrase probe, shadow negative, video detail, temporal style, massive-activation steering, video-to-audio coupling, **LoRA training**. - **Prompt enhancer**: a language model rewrites each finished prompt. Has its own encoder pick (Composer ▸ Enhance ▸ Encoder), because H3's own encoder cannot write prose. Verified with Gemma 4. - **Auto Montage**: builds a new cut at the end of the timeline from your rendered clips; originals stay untouched, and the pieces are never regenerated or rated. -- **Upscale** a finished render with an upscale model (`.safetensors`) or NVIDIA RTX Super Resolution. +- **Upscale** (a background job of FunUI like Interpolate, so it no longer makes ComfyUI drop the workflow's loaded models) a finished render with an upscale model (`.safetensors`) or NVIDIA RTX Super Resolution. - **LoRA training** (add-on, Settings > Modules): a Training window with four steps. Data (optional): drop files, pick from your Bin, or take your rated results (Liked, Disliked, Bad composition, Bad image, All). Tags: a caption editor beside every picture, a tag cloud, bulk edits and automatic tagging with downloadable tagger models. Train: all the settings, a live loss graph, Pause and Resume, and previews: every N steps training waits while your own pipeline makes a picture or clip with the LoRA so far, then carries on. Results: past runs. It trains everything in the set. H3 (picture and sound together) uses the module's own training loop with ComfyUI carrying the batches; other models use ComfyUI's trainer, with the loaders in your pipeline. **Experimental, and UNTESTED on real models or GPUs**: proven only on tiny stand-in models on CPU. - **Add-ons** (spaCy for automatic camera and focus, nvidia-vfx, onnxruntime for LoRA training) install on the next start after you switch them on. diff --git a/app/ui/features/interpolate/job.js b/app/ui/features/interpolate/job.js index 5f35f3ca..878860c2 100644 --- a/app/ui/features/interpolate/job.js +++ b/app/ui/features/interpolate/job.js @@ -1,4 +1,5 @@ -// One interpolation job from start to result. Pure of the page; the swap and the step ledger are shared with Upscale (shell/renders.js). +// One interpolation job from start to result. Pure of the page. +import { waitJob } from "../../shell/renders.js"; export const RTX = "NVIDIA RTX Frame Generation"; // the same string as modules/output/interpolate/framegen.py NAME export const DEFAULTS = { mode: "MEDIUM", multiplier: 2, tenbit: false }; @@ -7,19 +8,6 @@ const MODE = { LOW: "Fast", MEDIUM: "Balanced", HIGH: "Best" }; export const modeLabel = (m) => MODE[m] || m; /** -> the interpolated media, or throws a sentence. `sleep` is injectable so a test need not wait. */ -export async function interpolate(api, media, opts, sleep = (ms) => new Promise((r) => setTimeout(r, ms))) { - const { job_id: id } = await api.interpolate(media, opts); - let lost = 0; - for (;;) { - await sleep(1000); - let r; - try { r = await api.interpolateJob(id); } catch (err) { - if (err.status === 404) throw new Error("ComfyUI restarted while it was working"); - if (++lost < 30) continue; // a dropped reply is not a failed job - throw new Error("lost contact with ComfyUI"); - } - lost = 0; - if (r.state === "error") throw new Error(r.detail || "failed inside ComfyUI"); - if (r.state === "done") return r.media; - } +export async function interpolate(api, media, opts, sleep) { + return waitJob(api.interpolateJob, (await api.interpolate(media, opts)).job_id, sleep); } diff --git a/app/ui/features/upscale/job.js b/app/ui/features/upscale/job.js index d7060ed6..b729f1f1 100644 --- a/app/ui/features/upscale/job.js +++ b/app/ui/features/upscale/job.js @@ -1,5 +1,5 @@ // One upscale job from queue to result. Pure of the page. -import { poll } from "../../shell/renders.js"; +import { waitJob } from "../../shell/renders.js"; export { keyOf, swap } from "../../shell/renders.js"; export const UPSCALED = "funui_upscaled"; export const RTX = "NVIDIA RTX Super Resolution"; // sent as the model name for that engine; the same string as modules/output/render/rtx.py NAME @@ -18,5 +18,5 @@ export const keepsSize = (q) => q.startsWith("DENOISE_") || q.startsWith("DEBLUR /** -> the upscaled media, or throws a sentence. `sleep` is injectable so a test need not wait. */ export async function upscale(api, media, model, sleep, rtx) { - return poll(api, await api.queueUpscale(media, model, rtx), sleep); + return waitJob(api.upscaleJob, (await api.upscale(media, model, rtx)).job_id, sleep); } diff --git a/app/ui/features/upscale/job.test.js b/app/ui/features/upscale/job.test.js index 63280f56..2b5f476c 100644 --- a/app/ui/features/upscale/job.test.js +++ b/app/ui/features/upscale/job.test.js @@ -11,19 +11,19 @@ test("swap replaces the file under every render and ghost that plays it, and not assert.equal(p.scene_renders.c.media.filename, "y.mp4"); }); -test("upscale waits while queued, then returns the video; an error is thrown in words", async () => { - const seq = [null, null, { videos: [m("up.mp4", "funui_upscaled")] }]; - const api = { queueUpscale: async () => "id", upscaleResult: async () => seq.shift() }; +test("upscale waits while it runs, then returns the video; an error is thrown in words", async () => { + const seq = [{ state: "running" }, { state: "running" }, { state: "done", media: m("up.mp4", "funui_upscaled") }]; + const api = { upscale: async () => ({ job_id: "id" }), upscaleJob: async () => seq.shift() }; assert.equal((await upscale(api, m("x.mp4"), "mdl", async () => {})).filename, "up.mp4"); - const bad = { queueUpscale: async () => "id", upscaleResult: async () => ({ error: "out of memory" }) }; + const bad = { upscale: async () => ({ job_id: "id" }), upscaleJob: async () => ({ state: "error", detail: "out of memory" }) }; await assert.rejects(upscale(bad, m("x.mp4"), "mdl", async () => {}), /out of memory/); }); -test("a job that vanishes is reported after a few looks, a dropped reply is retried longer", async () => { - const gone = { queueUpscale: async () => "id", upscaleResult: async () => ({ gone: true }) }; - await assert.rejects(upscale(gone, m("x.mp4"), "mdl", async () => {}), /left ComfyUI's queue/); +test("a restarted ComfyUI is named, a dropped reply is retried longer", async () => { + const gone = { upscale: async () => ({ job_id: "id" }), upscaleJob: async () => { throw Object.assign(new Error("x"), { status: 404 }); } }; + await assert.rejects(upscale(gone, m("x.mp4"), "mdl", async () => {}), /restarted/); let n = 0; - const flaky = { queueUpscale: async () => "id", upscaleResult: async () => (++n < 5 ? { retry: true } : { videos: [m("u.mp4")] }) }; + const flaky = { upscale: async () => ({ job_id: "id" }), upscaleJob: async () => { if (++n < 5) throw new Error("net"); return { state: "done", media: m("u.mp4") }; } }; assert.equal((await upscale(flaky, m("x.mp4"), "mdl", async () => {})).filename, "u.mp4"); }); @@ -44,7 +44,7 @@ test("quality levels read as words, and only denoise and deblur keep the video's test("the RTX options reach the node as one JSON input, and a model upscale sends none", async () => { const seen = []; - const api = { queueUpscale: async (...a) => (seen.push(a), "id"), upscaleResult: async () => ({ videos: [m("u.mp4")] }) }; + const api = { upscale: async (...a) => (seen.push(a), { job_id: "id" }), upscaleJob: async () => ({ state: "done", media: m("u.mp4") }) }; await upscale(api, m("x.mp4"), "NVIDIA RTX Super Resolution", async () => {}, { quality: "ULTRA" }); await upscale(api, m("x.mp4"), "mdl", async () => {}); assert.deepEqual(seen.map((a) => a[2]), [{ quality: "ULTRA" }, undefined]); diff --git a/app/ui/shell/api.js b/app/ui/shell/api.js index d43faecf..384005c9 100644 --- a/app/ui/shell/api.js +++ b/app/ui/shell/api.js @@ -132,17 +132,9 @@ export const api = { return body.prompt_id; }, upscaleModels: () => call("GET", "/api/m/render/upscale_models"), - /** The upscale is an ordinary ComfyUI job: queued on /prompt, read back from /history. -> the prompt id. */ - async queueUpscale(media, model, rtx) { - const graph = { 1: { class_type: "FunUIUpscaleVideo", inputs: { filename: media.filename, subfolder: media.subfolder || "", type: media.type === "temp" ? "temp" : "output", upscale_model: model, ...(rtx ? { rtx_options: JSON.stringify(rtx) } : {}) } } }; - const res = await fetch("/prompt", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ prompt: graph }) }); - const body = await res.json().catch(() => ({})); - if (!res.ok || !body.prompt_id) { - const nodes = Object.values(body.node_errors || {}).flatMap((n) => (n.errors || []).map((x) => x.message)); - throw new Error([(body.error || {}).message, ...nodes].filter(Boolean).join("; ") || `ComfyUI refused the job (HTTP ${res.status})`); - } - return body.prompt_id; - }, + /** The upscale is a background job of FunUI, not a ComfyUI prompt (that would make ComfyUI drop the workflow's loaded models). -> {job_id}. */ + upscale: (media, model, rtx) => call("POST", "/api/m/render/upscale", { media: { filename: media.filename, subfolder: media.subfolder || "", type: media.type === "temp" ? "temp" : "output" }, model, ...(rtx ? { rtx_options: rtx } : {}) }), + upscaleJob: (id) => call("GET", `/api/m/render/upscale/${encodeURIComponent(id)}`), frameGeneration: () => call("GET", "/api/m/interpolate/frame_generation"), /** Interpolation is a background job of FunUI, not a ComfyUI prompt (that would make ComfyUI drop the workflow's loaded models). -> {job_id}. */ interpolate: (media, options) => call("POST", "/api/m/interpolate/run", { media: { filename: media.filename, subfolder: media.subfolder || "", type: media.type === "temp" ? "temp" : "output" }, options }), @@ -154,22 +146,6 @@ export const api = { return [...(q.queue_running || []), ...(q.queue_pending || [])].some((r) => (r[3] || {}).funui_project_id === projectId); } catch { return false; } }, - /** null while it is queued or running, else {videos} | {error} | {gone} | {retry} (a dropped reply is not a failed job). */ - async upscaleResult(promptId) { - let entry = null; - try { const res = await fetch(`/history/${encodeURIComponent(promptId)}`); entry = res.ok ? (await res.json())[promptId] : null; } catch { return { retry: true }; } - if (!entry) { - try { - const q = await (await fetch("/queue")).json(); - return [...(q.queue_running || []), ...(q.queue_pending || [])].some((r) => r[1] === promptId) ? null : { gone: true }; - } catch { return { retry: true }; } - } - const st = entry.status || {}; - if (st.status_str === "error") { const m = (st.messages || []).find((x) => x[0] === "execution_error"); return { error: (m && m[1] && m[1].exception_message) || "failed inside ComfyUI" }; } - if (!st.completed) return null; - const videos = Object.values(entry.outputs || {}).flatMap((o) => o.videos || []); - return videos.length ? { videos } : { error: "no video came out, check the ComfyUI terminal" }; - }, renderLibrary: () => call("GET", "/api/m/render/library"), lastFrame: (pid, body) => call("POST", `/api/m/render/projects/${encodeURIComponent(pid)}/last-frame`, body), suggestionStats: () => call("GET", "/api/shortcuts/suggestion_stats"), diff --git a/app/ui/shell/renders.js b/app/ui/shell/renders.js index 9c7c5136..fa0eba5e 100644 --- a/app/ui/shell/renders.js +++ b/app/ui/shell/renders.js @@ -1,18 +1,21 @@ -// What the steps that rewrite a finished render (upscale, interpolate) share: naming a file, waiting for its job, swapping it under every +// What the steps that rewrite a finished render (upscale, interpolate) share: naming a file, waiting for its background job, swapping it under every // scene that plays it, and remembering which steps a file has been through so two "always" modes can never feed each other forever. Pure of the page. export const keyOf = (m) => `${m.type || "output"}/${m.subfolder || ""}/${m.filename}`; -/** -> the finished job's first video, or throws a sentence. `sleep` is injectable so a test need not wait. */ -export async function poll(api, id, sleep = (ms) => new Promise((r) => setTimeout(r, ms))) { +/** Wait for a FunUI background job. `read(id)` -> {state: running | done | error, media, detail}. -> the media, or throws a sentence. `sleep` is injectable so a test need not wait. */ +export async function waitJob(read, id, sleep = (ms) => new Promise((r) => setTimeout(r, ms))) { let lost = 0; for (;;) { - await sleep(2000); - const r = await api.upscaleResult(id); - if (!r) { lost = 0; continue; } - if (r.retry) { if (++lost < 30) continue; throw new Error("lost contact with ComfyUI"); } - if (r.gone) { if (++lost < 3) continue; throw new Error("the job left ComfyUI's queue without finishing (interrupted or ComfyUI restarted)"); } - if (r.error) throw new Error(r.error); - return r.videos[0]; + await sleep(1000); + let r; + try { r = await read(id); } catch (err) { + if (err.status === 404) throw new Error("ComfyUI restarted while it was working"); + if (++lost < 30) continue; // a dropped reply is not a failed job + throw new Error("lost contact with ComfyUI"); + } + lost = 0; + if (r.state === "error") throw new Error(r.detail || "failed inside ComfyUI"); + if (r.state === "done") return r.media; } } diff --git a/modules/output/render/__init__.py b/modules/output/render/__init__.py index 247829fd..46295673 100644 --- a/modules/output/render/__init__.py +++ b/modules/output/render/__init__.py @@ -83,6 +83,21 @@ def routes(table, base, web): return web.json_response({"models": upscale.models(), "rtx": {"name": rtx.NAME, "qualities": list(rtx.QUALITIES)} if addons.enabled("nvidia_vsr") else None, "rtx_offered": any(a["id"] == "nvidia_vsr" for a in addons.listing())}) # offered = this machine could switch it on in Settings ▸ Modules + @table.post(base + "/upscale") + async def _upscale(req): + body = await body_of(req) + media, model = body.get("media"), body.get("model") + if not isinstance(media, dict) or not isinstance(media.get("filename"), str) or not media["filename"] or not isinstance(model, str) or not model: + return bad("Send {media: {filename, subfolder, type}, model, rtx_options}.") + job = jobs.new() + asyncio.create_task(jobs.run(job, upscale.run, media, model, body.get("rtx_options") or "{}", error=(RuntimeError, ValueError, files.ClipError))) + return web.json_response({"job_id": job}) + + @table.get(base + "/upscale/{job}") + async def _upscale_job(req): + job = jobs.get(req.match_info["job"]) + return web.json_response(job) if job else bad("Upscale job not found (ComfyUI restarted?).", 404) + @table.get(base + "/library") async def _library(_req): return web.json_response(library.payload()) diff --git a/modules/output/render/tests/test_upscale.py b/modules/output/render/tests/test_upscale.py index 3e68c309..7cc58e6a 100644 --- a/modules/output/render/tests/test_upscale.py +++ b/modules/output/render/tests/test_upscale.py @@ -86,3 +86,17 @@ def test_rtx_refuses_when_switched_off_even_if_installed(monkeypatch): with pytest.raises(RuntimeError, match="switched off"): rtx.make(rtx.options("{}")) + + +@needs_ffmpeg +def test_run_writes_the_upscaled_file_beside_the_original_without_a_comfy_prompt(comfy, monkeypatch): + import sys + import types + make_clip(str(comfy.output / "a.mp4"), seconds=0.5, size="64x48", rate=10, audio=True) + monkeypatch.setitem(sys.modules, "folder_paths", types.SimpleNamespace(get_output_directory=lambda: str(comfy.output))) + monkeypatch.setattr(rtx, "make", lambda o: (lambda x: x.repeat_interleave(2, 1).repeat_interleave(2, 2))) + got = upscale.run({"filename": "a.mp4", "subfolder": "", "type": "output"}, rtx.NAME, {"quality": "HIGH"}) + assert got == {"media": {"filename": "a_rtx_high.mp4", "subfolder": "funui_upscaled", "type": "output"}} + assert upscale.probe(str(comfy.output / "funui_upscaled" / "a_rtx_high.mp4"))[:2] == (128, 96) + with pytest.raises(RuntimeError, match="not in the output folder"): + upscale.run({"filename": "gone.mp4"}, rtx.NAME) diff --git a/modules/output/render/upscale.py b/modules/output/render/upscale.py index 0502ddc0..989d79b0 100644 --- a/modules/output/render/upscale.py +++ b/modules/output/render/upscale.py @@ -26,6 +26,31 @@ def models() -> list: return [] +def run(media, model, rtx_options="{}", interrupted=lambda: None) -> dict: + """Blocking. Upscale the output/temp file `media` ({filename, subfolder, type}) with an upscale model or NVIDIA RTX Super Resolution. + -> {"media": the new file}. The page runs this as a background job of its own: as a ComfyUI prompt it would make ComfyUI's cache + let go of the workflow's loaders and models (a prompt without them), and the next generation would load them again.""" + import folder_paths + + src = files.comfy_path(media.get("filename"), media.get("subfolder") or "", media.get("type") or "output") + if not src or not os.path.isfile(src): + raise RuntimeError(f"{media.get('filename')} is not in the {media.get('type') or 'output'} folder (anymore).") + if model == rtx.NAME: + opts = rtx.options(rtx_options) + fn, tag = rtx.make(opts), "rtx_" + opts["quality"].lower() + else: + from comfy_extras.nodes_upscale_model import ImageUpscaleWithModel, UpscaleModelLoader + loaded = UpscaleModelLoader.execute(model).result[0] + fn, tag = (lambda x: ImageUpscaleWithModel.execute(loaded, x).result[0]), model + out_dir = os.path.join(folder_paths.get_output_directory(), SUBFOLDER) + os.makedirs(out_dir, exist_ok=True) + name = free_name(out_dir, media["filename"], tag) + frames = upscale_file(src, os.path.join(out_dir, name), fn, interrupted) + from ..._core import log + log.info("FunUI Upscale", f"{media['filename']} with {tag}: {frames} frames -> {SUBFOLDER}/{name}") + return {"media": {"filename": name, "subfolder": SUBFOLDER, "type": "output"}} + + try: from comfy_api.latest import io except Exception: # noqa: BLE001 -- tests without ComfyUI @@ -56,23 +81,5 @@ if io is not None: @classmethod def execute(cls, filename, subfolder, type, upscale_model, rtx_options="{}") -> io.NodeOutput: import comfy.model_management as mm - import folder_paths - - src = files.comfy_path(filename, subfolder, type) - if not src or not os.path.isfile(src): - raise RuntimeError(f"{filename} is not in the {type} folder (anymore).") - if upscale_model == rtx.NAME: - opts = rtx.options(rtx_options) - fn, tag = rtx.make(opts), "rtx_" + opts["quality"].lower() - else: - from comfy_extras.nodes_upscale_model import ImageUpscaleWithModel, UpscaleModelLoader - model = UpscaleModelLoader.execute(upscale_model).result[0] - fn, tag = (lambda x: ImageUpscaleWithModel.execute(model, x).result[0]), upscale_model - out_dir = os.path.join(folder_paths.get_output_directory(), SUBFOLDER) - os.makedirs(out_dir, exist_ok=True) - name = free_name(out_dir, filename, tag) - frames = upscale_file(src, os.path.join(out_dir, name), fn, mm.throw_exception_if_processing_interrupted) - from ..._core import log - log.info("FunUI Upscale", f"{filename} with {tag}: {frames} frames -> {SUBFOLDER}/{name}") - return io.NodeOutput(ui={"videos": [{"filename": name, "subfolder": SUBFOLDER, - "type": "output", "format": "video/h264-mp4"}]}) + done = run({"filename": filename, "subfolder": subfolder, "type": type}, upscale_model, rtx_options, mm.throw_exception_if_processing_interrupted) + return io.NodeOutput(ui={"videos": [{**done["media"], "format": "video/h264-mp4"}]})