Upscale runs as a FunUI background job too (no extra ComfyUI prompt that makes the cache drop the workflow's models)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tw9fjW75aYB8UyVRZo6fCr

CHANGELOG.md

@@ -4,6 +4,7 @@
 
 ### Added
 
+- **Upscale no longer makes the next generation reload the models.** It ran as an extra ComfyUI prompt without the workflow's loaders, and ComfyUI's cache lets go of loaders a prompt does not use; it now runs as a background job of FunUI, beside ComfyUI's queue (Interrupt in ComfyUI no longer stops it).
 - **Interpolate** (module of its own, add-on **NVIDIA RTX Frame Generation**): adds in-between frames to a finished render with NVIDIA's RTX frame generation (2 to 4 times the frames; the length and the sound stay, 24 fps becomes 48 fps). Settings > Editor > Interpolate: Never, a button beside the rating, or always when a render finishes. Quality Fast, Balanced or Best, and 10-bit processing. It runs as a background job of FunUI, not as a ComfyUI prompt, so it does not make ComfyUI drop the workflow's loaded models (an extra prompt without loaders does exactly that, and the next generation reloads them); the NVIDIA model also stays loaded between presses. Works beside Upscale in either order and never runs twice on the same file. Needs an RTX card and nvidia-vfx 0.2 or newer: switching the add-on on moves an older nvidia-vfx up (nothing else is touched), and Super Resolution never moves a newer one back. ComfyUI's own RIFE/FILM interpolation nodes are a different thing and are not used. Works on a rental GPU (a few seconds for a clip).
 - **Project settings > Resolution is the** (H3, LTX image to video): what Width and Height mean once the scene picture sets the shape. Short side (the smaller number is the clip's shorter side), Long side (the larger is its longer side) or Pixel count (the old rule: about Width x Height pixels, which turned 1024x1024 into 1312x768 for a wide picture). New projects start on Short side; projects saved earlier keep Pixel count, so nothing changes size by itself. The H3 preset no longer needs the KJNodes ImageTransformKJ node for this (saved H3 pipelines are switched over on load).
 - **VAE Loader: Decode-only VAE (Optional)**: a second VAE used only for the final picture decode while the first still encodes (references, start pictures). For decoders without an encoder: MiniMax H3's Sol-H3-VAE (`sol_h3_vae.safetensors`, whole clip at once, about 2.5x faster decode, needs a ComfyUI newer than 0.43.1) and the X2 Detail VAE.
@@ -132,7 +133,7 @@ A rewrite from scratch, not an update of 4.x. Same idea (a studio for video insi
 - **Experimental**: continuity, identity transfer, negative erase, camera move, context windows, block influence, block repeat, phrase probe, shadow negative, video detail, temporal style, massive-activation steering, video-to-audio coupling, **LoRA training**.
 - **Prompt enhancer**: a language model rewrites each finished prompt. Has its own encoder pick (Composer ▸ Enhance ▸ Encoder), because H3's own encoder cannot write prose. Verified with Gemma 4.
 - **Auto Montage**: builds a new cut at the end of the timeline from your rendered clips; originals stay untouched, and the pieces are never regenerated or rated.
-- **Upscale** a finished render with an upscale model (`.safetensors`) or NVIDIA RTX Super Resolution.
+- **Upscale** (a background job of FunUI like Interpolate, so it no longer makes ComfyUI drop the workflow's loaded models) a finished render with an upscale model (`.safetensors`) or NVIDIA RTX Super Resolution.
 - **LoRA training** (add-on, Settings > Modules): a Training window with four steps. Data (optional): drop files, pick from your Bin, or take your rated results (Liked, Disliked, Bad composition, Bad image, All). Tags: a caption editor beside every picture, a tag cloud, bulk edits and automatic tagging with downloadable tagger models. Train: all the settings, a live loss graph, Pause and Resume, and previews: every N steps training waits while your own pipeline makes a picture or clip with the LoRA so far, then carries on. Results: past runs. It trains everything in the set. H3 (picture and sound together) uses the module's own training loop with ComfyUI carrying the batches; other models use ComfyUI's trainer, with the loaders in your pipeline. **Experimental, and UNTESTED on real models or GPUs**: proven only on tiny stand-in models on CPU.
 - **Add-ons** (spaCy for automatic camera and focus, nvidia-vfx, onnxruntime for LoRA training) install on the next start after you switch them on.
 

app/ui/features/interpolate/job.js

@@ -1,4 +1,5 @@
-// One interpolation job from start to result. Pure of the page; the swap and the step ledger are shared with Upscale (shell/renders.js).
+// One interpolation job from start to result. Pure of the page.
+import { waitJob } from "../../shell/renders.js";
 
 export const RTX = "NVIDIA RTX Frame Generation";           // the same string as modules/output/interpolate/framegen.py NAME
 export const DEFAULTS = { mode: "MEDIUM", multiplier: 2, tenbit: false };
@@ -7,19 +8,6 @@ const MODE = { LOW: "Fast", MEDIUM: "Balanced", HIGH: "Best" };
 export const modeLabel = (m) => MODE[m] || m;
 
 /** -> the interpolated media, or throws a sentence. `sleep` is injectable so a test need not wait. */
-export async function interpolate(api, media, opts, sleep = (ms) => new Promise((r) => setTimeout(r, ms))) {
-  const { job_id: id } = await api.interpolate(media, opts);
-  let lost = 0;
-  for (;;) {
-    await sleep(1000);
-    let r;
-    try { r = await api.interpolateJob(id); } catch (err) {
-      if (err.status === 404) throw new Error("ComfyUI restarted while it was working");
-      if (++lost < 30) continue;                                    // a dropped reply is not a failed job
-      throw new Error("lost contact with ComfyUI");
-    }
-    lost = 0;
-    if (r.state === "error") throw new Error(r.detail || "failed inside ComfyUI");
-    if (r.state === "done") return r.media;
-  }
+export async function interpolate(api, media, opts, sleep) {
+  return waitJob(api.interpolateJob, (await api.interpolate(media, opts)).job_id, sleep);
 }

app/ui/features/upscale/job.js

@@ -1,5 +1,5 @@
 // One upscale job from queue to result. Pure of the page.
-import { poll } from "../../shell/renders.js";
+import { waitJob } from "../../shell/renders.js";
 export { keyOf, swap } from "../../shell/renders.js";
 export const UPSCALED = "funui_upscaled";
 export const RTX = "NVIDIA RTX Super Resolution";           // sent as the model name for that engine; the same string as modules/output/render/rtx.py NAME
@@ -18,5 +18,5 @@ export const keepsSize = (q) => q.startsWith("DENOISE_") || q.startsWith("DEBLUR
 
 /** -> the upscaled media, or throws a sentence. `sleep` is injectable so a test need not wait. */
 export async function upscale(api, media, model, sleep, rtx) {
-  return poll(api, await api.queueUpscale(media, model, rtx), sleep);
+  return waitJob(api.upscaleJob, (await api.upscale(media, model, rtx)).job_id, sleep);
 }

app/ui/features/upscale/job.test.js

@@ -11,19 +11,19 @@ test("swap replaces the file under every render and ghost that plays it, and not
   assert.equal(p.scene_renders.c.media.filename, "y.mp4");
 });
 
-test("upscale waits while queued, then returns the video; an error is thrown in words", async () => {
-  const seq = [null, null, { videos: [m("up.mp4", "funui_upscaled")] }];
-  const api = { queueUpscale: async () => "id", upscaleResult: async () => seq.shift() };
+test("upscale waits while it runs, then returns the video; an error is thrown in words", async () => {
+  const seq = [{ state: "running" }, { state: "running" }, { state: "done", media: m("up.mp4", "funui_upscaled") }];
+  const api = { upscale: async () => ({ job_id: "id" }), upscaleJob: async () => seq.shift() };
   assert.equal((await upscale(api, m("x.mp4"), "mdl", async () => {})).filename, "up.mp4");
-  const bad = { queueUpscale: async () => "id", upscaleResult: async () => ({ error: "out of memory" }) };
+  const bad = { upscale: async () => ({ job_id: "id" }), upscaleJob: async () => ({ state: "error", detail: "out of memory" }) };
   await assert.rejects(upscale(bad, m("x.mp4"), "mdl", async () => {}), /out of memory/);
 });
 
-test("a job that vanishes is reported after a few looks, a dropped reply is retried longer", async () => {
-  const gone = { queueUpscale: async () => "id", upscaleResult: async () => ({ gone: true }) };
-  await assert.rejects(upscale(gone, m("x.mp4"), "mdl", async () => {}), /left ComfyUI's queue/);
+test("a restarted ComfyUI is named, a dropped reply is retried longer", async () => {
+  const gone = { upscale: async () => ({ job_id: "id" }), upscaleJob: async () => { throw Object.assign(new Error("x"), { status: 404 }); } };
+  await assert.rejects(upscale(gone, m("x.mp4"), "mdl", async () => {}), /restarted/);
   let n = 0;
-  const flaky = { queueUpscale: async () => "id", upscaleResult: async () => (++n < 5 ? { retry: true } : { videos: [m("u.mp4")] }) };
+  const flaky = { upscale: async () => ({ job_id: "id" }), upscaleJob: async () => { if (++n < 5) throw new Error("net"); return { state: "done", media: m("u.mp4") }; } };
   assert.equal((await upscale(flaky, m("x.mp4"), "mdl", async () => {})).filename, "u.mp4");
 });
 
@@ -44,7 +44,7 @@ test("quality levels read as words, and only denoise and deblur keep the video's
 
 test("the RTX options reach the node as one JSON input, and a model upscale sends none", async () => {
   const seen = [];
-  const api = { queueUpscale: async (...a) => (seen.push(a), "id"), upscaleResult: async () => ({ videos: [m("u.mp4")] }) };
+  const api = { upscale: async (...a) => (seen.push(a), { job_id: "id" }), upscaleJob: async () => ({ state: "done", media: m("u.mp4") }) };
   await upscale(api, m("x.mp4"), "NVIDIA RTX Super Resolution", async () => {}, { quality: "ULTRA" });
   await upscale(api, m("x.mp4"), "mdl", async () => {});
   assert.deepEqual(seen.map((a) => a[2]), [{ quality: "ULTRA" }, undefined]);

app/ui/shell/api.js

@@ -132,17 +132,9 @@ export const api = {
     return body.prompt_id;
   },
   upscaleModels: () => call("GET", "/api/m/render/upscale_models"),
-  /** The upscale is an ordinary ComfyUI job: queued on /prompt, read back from /history. -> the prompt id. */
-  async queueUpscale(media, model, rtx) {
-    const graph = { 1: { class_type: "FunUIUpscaleVideo", inputs: { filename: media.filename, subfolder: media.subfolder || "", type: media.type === "temp" ? "temp" : "output", upscale_model: model, ...(rtx ? { rtx_options: JSON.stringify(rtx) } : {}) } } };
-    const res = await fetch("/prompt", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ prompt: graph }) });
-    const body = await res.json().catch(() => ({}));
-    if (!res.ok || !body.prompt_id) {
-      const nodes = Object.values(body.node_errors || {}).flatMap((n) => (n.errors || []).map((x) => x.message));
-      throw new Error([(body.error || {}).message, ...nodes].filter(Boolean).join("; ") || `ComfyUI refused the job (HTTP ${res.status})`);
-    }
-    return body.prompt_id;
-  },
+  /** The upscale is a background job of FunUI, not a ComfyUI prompt (that would make ComfyUI drop the workflow's loaded models). -> {job_id}. */
+  upscale: (media, model, rtx) => call("POST", "/api/m/render/upscale", { media: { filename: media.filename, subfolder: media.subfolder || "", type: media.type === "temp" ? "temp" : "output" }, model, ...(rtx ? { rtx_options: rtx } : {}) }),
+  upscaleJob: (id) => call("GET", `/api/m/render/upscale/${encodeURIComponent(id)}`),
   frameGeneration: () => call("GET", "/api/m/interpolate/frame_generation"),
   /** Interpolation is a background job of FunUI, not a ComfyUI prompt (that would make ComfyUI drop the workflow's loaded models). -> {job_id}. */
   interpolate: (media, options) => call("POST", "/api/m/interpolate/run", { media: { filename: media.filename, subfolder: media.subfolder || "", type: media.type === "temp" ? "temp" : "output" }, options }),
@@ -154,22 +146,6 @@ export const api = {
       return [...(q.queue_running || []), ...(q.queue_pending || [])].some((r) => (r[3] || {}).funui_project_id === projectId);
     } catch { return false; }
   },
-  /** null while it is queued or running, else {videos} | {error} | {gone} | {retry} (a dropped reply is not a failed job). */
-  async upscaleResult(promptId) {
-    let entry = null;
-    try { const res = await fetch(`/history/${encodeURIComponent(promptId)}`); entry = res.ok ? (await res.json())[promptId] : null; } catch { return { retry: true }; }
-    if (!entry) {
-      try {
-        const q = await (await fetch("/queue")).json();
-        return [...(q.queue_running || []), ...(q.queue_pending || [])].some((r) => r[1] === promptId) ? null : { gone: true };
-      } catch { return { retry: true }; }
-    }
-    const st = entry.status || {};
-    if (st.status_str === "error") { const m = (st.messages || []).find((x) => x[0] === "execution_error"); return { error: (m && m[1] && m[1].exception_message) || "failed inside ComfyUI" }; }
-    if (!st.completed) return null;
-    const videos = Object.values(entry.outputs || {}).flatMap((o) => o.videos || []);
-    return videos.length ? { videos } : { error: "no video came out, check the ComfyUI terminal" };
-  },
   renderLibrary: () => call("GET", "/api/m/render/library"),
   lastFrame: (pid, body) => call("POST", `/api/m/render/projects/${encodeURIComponent(pid)}/last-frame`, body),
   suggestionStats: () => call("GET", "/api/shortcuts/suggestion_stats"),

app/ui/shell/renders.js

@@ -1,18 +1,21 @@
-// What the steps that rewrite a finished render (upscale, interpolate) share: naming a file, waiting for its job, swapping it under every
+// What the steps that rewrite a finished render (upscale, interpolate) share: naming a file, waiting for its background job, swapping it under every
 // scene that plays it, and remembering which steps a file has been through so two "always" modes can never feed each other forever. Pure of the page.
 export const keyOf = (m) => `${m.type || "output"}/${m.subfolder || ""}/${m.filename}`;
 
-/** -> the finished job's first video, or throws a sentence. `sleep` is injectable so a test need not wait. */
-export async function poll(api, id, sleep = (ms) => new Promise((r) => setTimeout(r, ms))) {
+/** Wait for a FunUI background job. `read(id)` -> {state: running | done | error, media, detail}. -> the media, or throws a sentence. `sleep` is injectable so a test need not wait. */
+export async function waitJob(read, id, sleep = (ms) => new Promise((r) => setTimeout(r, ms))) {
   let lost = 0;
   for (;;) {
-    await sleep(2000);
-    const r = await api.upscaleResult(id);
-    if (!r) { lost = 0; continue; }
-    if (r.retry) { if (++lost < 30) continue; throw new Error("lost contact with ComfyUI"); }
-    if (r.gone) { if (++lost < 3) continue; throw new Error("the job left ComfyUI's queue without finishing (interrupted or ComfyUI restarted)"); }
-    if (r.error) throw new Error(r.error);
-    return r.videos[0];
+    await sleep(1000);
+    let r;
+    try { r = await read(id); } catch (err) {
+      if (err.status === 404) throw new Error("ComfyUI restarted while it was working");
+      if (++lost < 30) continue;                                    // a dropped reply is not a failed job
+      throw new Error("lost contact with ComfyUI");
+    }
+    lost = 0;
+    if (r.state === "error") throw new Error(r.detail || "failed inside ComfyUI");
+    if (r.state === "done") return r.media;
   }
 }
 

modules/output/render/__init__.py

@@ -83,6 +83,21 @@ def routes(table, base, web):
         return web.json_response({"models": upscale.models(), "rtx": {"name": rtx.NAME, "qualities": list(rtx.QUALITIES)} if addons.enabled("nvidia_vsr") else None,
                                   "rtx_offered": any(a["id"] == "nvidia_vsr" for a in addons.listing())})   # offered = this machine could switch it on in Settings ▸ Modules
 
+    @table.post(base + "/upscale")
+    async def _upscale(req):
+        body = await body_of(req)
+        media, model = body.get("media"), body.get("model")
+        if not isinstance(media, dict) or not isinstance(media.get("filename"), str) or not media["filename"] or not isinstance(model, str) or not model:
+            return bad("Send {media: {filename, subfolder, type}, model, rtx_options}.")
+        job = jobs.new()
+        asyncio.create_task(jobs.run(job, upscale.run, media, model, body.get("rtx_options") or "{}", error=(RuntimeError, ValueError, files.ClipError)))
+        return web.json_response({"job_id": job})
+
+    @table.get(base + "/upscale/{job}")
+    async def _upscale_job(req):
+        job = jobs.get(req.match_info["job"])
+        return web.json_response(job) if job else bad("Upscale job not found (ComfyUI restarted?).", 404)
+
     @table.get(base + "/library")
     async def _library(_req):
         return web.json_response(library.payload())

modules/output/render/tests/test_upscale.py

@@ -86,3 +86,17 @@ def test_rtx_refuses_when_switched_off_even_if_installed(monkeypatch):
     with pytest.raises(RuntimeError, match="switched off"):
         rtx.make(rtx.options("{}"))
 
+
+
+@needs_ffmpeg
+def test_run_writes_the_upscaled_file_beside_the_original_without_a_comfy_prompt(comfy, monkeypatch):
+    import sys
+    import types
+    make_clip(str(comfy.output / "a.mp4"), seconds=0.5, size="64x48", rate=10, audio=True)
+    monkeypatch.setitem(sys.modules, "folder_paths", types.SimpleNamespace(get_output_directory=lambda: str(comfy.output)))
+    monkeypatch.setattr(rtx, "make", lambda o: (lambda x: x.repeat_interleave(2, 1).repeat_interleave(2, 2)))
+    got = upscale.run({"filename": "a.mp4", "subfolder": "", "type": "output"}, rtx.NAME, {"quality": "HIGH"})
+    assert got == {"media": {"filename": "a_rtx_high.mp4", "subfolder": "funui_upscaled", "type": "output"}}
+    assert upscale.probe(str(comfy.output / "funui_upscaled" / "a_rtx_high.mp4"))[:2] == (128, 96)
+    with pytest.raises(RuntimeError, match="not in the output folder"):
+        upscale.run({"filename": "gone.mp4"}, rtx.NAME)

modules/output/render/upscale.py

@@ -26,6 +26,31 @@ def models() -> list:
         return []
 
 
+def run(media, model, rtx_options="{}", interrupted=lambda: None) -> dict:
+    """Blocking. Upscale the output/temp file `media` ({filename, subfolder, type}) with an upscale model or NVIDIA RTX Super Resolution.
+    -> {"media": the new file}. The page runs this as a background job of its own: as a ComfyUI prompt it would make ComfyUI's cache
+    let go of the workflow's loaders and models (a prompt without them), and the next generation would load them again."""
+    import folder_paths
+
+    src = files.comfy_path(media.get("filename"), media.get("subfolder") or "", media.get("type") or "output")
+    if not src or not os.path.isfile(src):
+        raise RuntimeError(f"{media.get('filename')} is not in the {media.get('type') or 'output'} folder (anymore).")
+    if model == rtx.NAME:
+        opts = rtx.options(rtx_options)
+        fn, tag = rtx.make(opts), "rtx_" + opts["quality"].lower()
+    else:
+        from comfy_extras.nodes_upscale_model import ImageUpscaleWithModel, UpscaleModelLoader
+        loaded = UpscaleModelLoader.execute(model).result[0]
+        fn, tag = (lambda x: ImageUpscaleWithModel.execute(loaded, x).result[0]), model
+    out_dir = os.path.join(folder_paths.get_output_directory(), SUBFOLDER)
+    os.makedirs(out_dir, exist_ok=True)
+    name = free_name(out_dir, media["filename"], tag)
+    frames = upscale_file(src, os.path.join(out_dir, name), fn, interrupted)
+    from ..._core import log
+    log.info("FunUI Upscale", f"{media['filename']} with {tag}: {frames} frames -> {SUBFOLDER}/{name}")
+    return {"media": {"filename": name, "subfolder": SUBFOLDER, "type": "output"}}
+
+
 try:
     from comfy_api.latest import io
 except Exception:                                      # noqa: BLE001 -- tests without ComfyUI
@@ -56,23 +81,5 @@ if io is not None:
         @classmethod
         def execute(cls, filename, subfolder, type, upscale_model, rtx_options="{}") -> io.NodeOutput:
             import comfy.model_management as mm
-            import folder_paths
-
-            src = files.comfy_path(filename, subfolder, type)
-            if not src or not os.path.isfile(src):
-                raise RuntimeError(f"{filename} is not in the {type} folder (anymore).")
-            if upscale_model == rtx.NAME:
-                opts = rtx.options(rtx_options)
-                fn, tag = rtx.make(opts), "rtx_" + opts["quality"].lower()
-            else:
-                from comfy_extras.nodes_upscale_model import ImageUpscaleWithModel, UpscaleModelLoader
-                model = UpscaleModelLoader.execute(upscale_model).result[0]
-                fn, tag = (lambda x: ImageUpscaleWithModel.execute(model, x).result[0]), upscale_model
-            out_dir = os.path.join(folder_paths.get_output_directory(), SUBFOLDER)
-            os.makedirs(out_dir, exist_ok=True)
-            name = free_name(out_dir, filename, tag)
-            frames = upscale_file(src, os.path.join(out_dir, name), fn, mm.throw_exception_if_processing_interrupted)
-            from ..._core import log
-            log.info("FunUI Upscale", f"{filename} with {tag}: {frames} frames -> {SUBFOLDER}/{name}")
-            return io.NodeOutput(ui={"videos": [{"filename": name, "subfolder": SUBFOLDER,
-                                                 "type": "output", "format": "video/h264-mp4"}]})
+            done = run({"filename": filename, "subfolder": subfolder, "type": type}, upscale_model, rtx_options, mm.throw_exception_if_processing_interrupted)
+            return io.NodeOutput(ui={"videos": [{**done["media"], "format": "video/h264-mp4"}]})