Two stages: remove Keep the latent's spread (worked badly on a rental)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tw9fjVWCMpMsh4XkZmnKnh
This commit is contained in:
Claude
+1
-1
@@ -4,7 +4,7 @@
|
||||
|
||||
### Added
|
||||
|
||||
- **Two stages: Latent upscaler as an option** (Settings > Engine > Sampling, "How the hand-over is enlarged"). Bicubic stays the default; plain choices Mitchell, Catmull-Rom, Lanczos, Bislerp, Bilinear, Trilinear (same as Bilinear here, time is never resampled), Area and Nearest-exact are there too (Lanczos is done on the numbers, not through 8-bit pictures as ComfyUI's would, which a latent cannot survive). "Keep the latent's spread" (any method) puts each channel back to its mean and spread from before enlarging. Or pick Latent upscaler and a file from models/latent_upscale_models (H3's convolution upscaler) to enlarge the first stage's clean guess with the learned network instead. It runs at the ratio that reaches the full size (it works in even steps, then a bicubic trim fits the exact grid), once per run, on a small latent; the file is loaded once and kept. A missing or unreadable file falls back to bicubic with a note. Untested on a GPU.
|
||||
- **Two stages: Latent upscaler as an option** (Settings > Engine > Sampling, "How the hand-over is enlarged"). Bicubic stays the default; plain choices Mitchell, Catmull-Rom, Lanczos, Bislerp, Bilinear, Trilinear (same as Bilinear here, time is never resampled), Area and Nearest-exact are there too (Lanczos is done on the numbers, not through 8-bit pictures as ComfyUI's would, which a latent cannot survive). Or pick Latent upscaler and a file from models/latent_upscale_models (H3's convolution upscaler) to enlarge the first stage's clean guess with the learned network instead. It runs at the ratio that reaches the full size (it works in even steps, then a bicubic trim fits the exact grid), once per run, on a small latent; the file is loaded once and kept. A missing or unreadable file falls back to bicubic with a note. Untested on a GPU.
|
||||
- **Upscale no longer makes the next generation reload the models.** It ran as an extra ComfyUI prompt without the workflow's loaders, and ComfyUI's cache lets go of loaders a prompt does not use; it now runs as a background job of FunUI, beside ComfyUI's queue (Interrupt in ComfyUI no longer stops it).
|
||||
- **Interpolate** (module of its own, add-on **NVIDIA RTX Frame Generation**): adds in-between frames to a finished render with NVIDIA's RTX frame generation (2 to 4 times the frames; the length and the sound stay, 24 fps becomes 48 fps). Settings > Editor > Interpolate: Never, a button beside the rating, or always when a render finishes. Quality Fast, Balanced or Best, and 10-bit processing. It runs as a background job of FunUI, not as a ComfyUI prompt, so it does not make ComfyUI drop the workflow's loaded models (an extra prompt without loaders does exactly that, and the next generation reloads them); the NVIDIA model also stays loaded between presses. Works beside Upscale in either order and never runs twice on the same file. Needs an RTX card and nvidia-vfx 0.2 or newer: switching the add-on on moves an older nvidia-vfx up (nothing else is touched), and Super Resolution never moves a newer one back. ComfyUI's own RIFE/FILM interpolation nodes are a different thing and are not used. Works on a rental GPU (a few seconds for a clip).
|
||||
- **Project settings > Resolution is the** (H3, LTX image to video): what Width and Height mean once the scene picture sets the shape. Short side (the smaller number is the clip's shorter side), Long side (the larger is its longer side) or Pixel count (the old rule: about Width x Height pixels, which turned 1024x1024 into 1312x768 for a wide picture). New projects start on Short side; projects saved earlier keep Pixel count, so nothing changes size by itself. The H3 preset no longer needs the KJNodes ImageTransformKJ node for this (saved H3 pipelines are switched over on load).
|
||||
|
||||
@@ -71,13 +71,6 @@ SETTINGS = {
|
||||
"once per run. The first stage size is kept, the upscaler is run at the ratio that reaches the full size.",
|
||||
"when": {"enabled": True},
|
||||
},
|
||||
"match": {
|
||||
"type": "bool", "default": False,
|
||||
"label": "Keep the latent's spread",
|
||||
"hint": "Enlarging smooths the picture and shrinks each channel's spread. This puts every channel back to the "
|
||||
"mean and spread it had before enlarging, whichever method is chosen.",
|
||||
"when": {"enabled": True},
|
||||
},
|
||||
"upscaler": {
|
||||
"type": "enum", "default": (_files() or [""])[0],
|
||||
"options": _options(),
|
||||
@@ -94,10 +87,9 @@ def install(patcher, values, key):
|
||||
size, steps = float(values.get("size", 0.7)), int(values.get("steps", 2))
|
||||
resize = values.get("upscaler") if values.get("resize") == "upscaler" and values.get("upscaler") else None
|
||||
method = values.get("resize") if values.get("resize") in METHODS else "bicubic"
|
||||
patcher.add_wrapper_with_key(WrappersMP.OUTER_SAMPLE, key, make_wrapper(size, steps, resize, method, bool(values.get("match"))))
|
||||
patcher.add_wrapper_with_key(WrappersMP.OUTER_SAMPLE, key, make_wrapper(size, steps, resize, method))
|
||||
patcher.add_wrapper_with_key(WrappersMP.SAMPLER_SAMPLE, f"{key}.schedule", keep_whole_schedule)
|
||||
return f"first {steps} step(s) at {size:.0%} of the size" + (f", enlarged by {resize}" if resize else f", enlarged by {method}" if method != "bicubic" else "") \
|
||||
+ (", spread kept" if values.get("match") else "")
|
||||
return f"first {steps} step(s) at {size:.0%} of the size" + (f", enlarged by {resize}" if resize else f", enlarged by {method}" if method != "bicubic" else "")
|
||||
|
||||
|
||||
PROVIDES = {"modifier": install, "setting_options": lambda: {"upscaler": _options()}}
|
||||
|
||||
@@ -129,43 +129,31 @@ def resize_video(video, h, w, method="bicubic"):
|
||||
return x.reshape(b, f, c, h, w).permute(0, 2, 1, 3, 4).to(video.dtype)
|
||||
|
||||
|
||||
def match_spread(out, ref):
|
||||
"""Enlarging smooths, so each channel's spread shrinks; put every channel back to the mean and spread it had
|
||||
before (over all frames and pixels), so stage 2 starts from a latent that looks like a real one."""
|
||||
o, r = out.float(), ref.float()
|
||||
dims = (2, 3, 4)
|
||||
mo, so = o.mean(dims, keepdim=True), o.std(dims, keepdim=True)
|
||||
mr, sr = r.mean(dims, keepdim=True), r.std(dims, keepdim=True)
|
||||
return ((o - mo) * (sr / so.clamp_min(1e-6)) + mr).to(out.dtype)
|
||||
|
||||
|
||||
def enlarge(video, h, w, upscaler=None, method="bicubic", match=False):
|
||||
def enlarge(video, h, w, upscaler=None, method="bicubic"):
|
||||
"""The hand-over: the first stage's clean guess at the full grid. With an upscaler, its learned resize at the
|
||||
ratio that gets nearest (it works in even steps), then a bicubic trim to the exact grid; without, `method`."""
|
||||
from ...latent.second_pass.nodes import run_upscaler
|
||||
if upscaler is None:
|
||||
out = resize_video(video, h, w, method)
|
||||
else:
|
||||
ratio = (h / video.shape[-2] + w / video.shape[-1]) / 2.0
|
||||
if ratio <= 1.0:
|
||||
return video
|
||||
out = resize_video(run_upscaler(upscaler, video, ratio), h, w)
|
||||
return match_spread(out, video) if match and out is not video else out
|
||||
return resize_video(video, h, w, method)
|
||||
ratio = (h / video.shape[-2] + w / video.shape[-1]) / 2.0
|
||||
if ratio <= 1.0:
|
||||
return video
|
||||
return resize_video(run_upscaler(upscaler, video, ratio), h, w)
|
||||
|
||||
|
||||
def make_wrapper(size, steps, resize=None, method="bicubic", match=False):
|
||||
def make_wrapper(size, steps, resize=None, method="bicubic"):
|
||||
def outer(executor, *args, **kwargs):
|
||||
# Every OUTER_SAMPLE wrapper behind this one belongs around the WHOLE run, not around each stage (Noise
|
||||
# merging's per-run state, Taste, STAS' report ...). They are moved to wrap the staged run as one call.
|
||||
from comfy.patcher_extension import WrapperExecutor
|
||||
rest = executor.wrappers[executor.idx + 1:]
|
||||
staged = lambda *a, **k: _staged(executor.original, executor.class_obj, size, steps, resize, method, match, *a, **k)
|
||||
staged = lambda *a, **k: _staged(executor.original, executor.class_obj, size, steps, resize, method, *a, **k)
|
||||
return WrapperExecutor.new_class_executor(staged, executor.class_obj, rest, 0).execute(*args, **kwargs)
|
||||
|
||||
return outer
|
||||
|
||||
|
||||
def _staged(original, guider, size, steps, resize, method, match, *args, **kwargs):
|
||||
def _staged(original, guider, size, steps, resize, method, *args, **kwargs):
|
||||
run = original
|
||||
import comfy.nested_tensor
|
||||
import comfy.utils
|
||||
@@ -229,7 +217,7 @@ def _staged(original, guider, size, steps, resize, method, match, *args, **kwarg
|
||||
done = guider.model_patcher.model.process_latent_out(comfy.nested_tensor.NestedTensor((video, audio)).cpu())
|
||||
video, audio = done.unbind()
|
||||
handed, _ = comfy.utils.pack_latents(
|
||||
(enlarge(video, h, w, upscaler, method, match).to(latent_image.device, latent_image.dtype), audio.to(latent_image.device, latent_image.dtype)))
|
||||
(enlarge(video, h, w, upscaler, method).to(latent_image.device, latent_image.dtype), audio.to(latent_image.device, latent_image.dtype)))
|
||||
|
||||
guider.conds = raw_conds # the first stage encoded the originals for its own size
|
||||
shifted = None
|
||||
|
||||
@@ -183,22 +183,3 @@ def test_flat_stays_flat_for_the_cubics_too():
|
||||
for method in ("mitchell", "catmull-rom"):
|
||||
out = stages.resize_video(flat, 7, 9, method)
|
||||
assert torch.allclose(out, flat.new_full(out.shape, 2.5), atol=1e-5)
|
||||
|
||||
|
||||
def test_keeping_the_spread_restores_each_channels_mean_and_std():
|
||||
from modules.sampling.two_stage import stages
|
||||
torch.manual_seed(0)
|
||||
video = torch.randn(1, 24, 2, 8, 8) * torch.linspace(0.5, 3, 24).view(1, 24, 1, 1, 1) + 1.0
|
||||
smooth = stages.enlarge(video, 16, 16, method="area")
|
||||
kept = stages.enlarge(video, 16, 16, method="area", match=True)
|
||||
dims = (2, 3, 4)
|
||||
assert torch.allclose(kept.std(dims), video.std(dims), rtol=1e-4)
|
||||
assert torch.allclose(kept.mean(dims), video.mean(dims), atol=1e-4)
|
||||
assert kept.dtype == video.dtype and smooth.std(dims).mean() <= video.std(dims).mean() + 1e-3
|
||||
|
||||
|
||||
def test_keeping_the_spread_runs_through_a_staged_run_and_in_bf16():
|
||||
from modules.sampling.two_stage import stages
|
||||
video = torch.randn(1, 24, 2, 4, 4).to(torch.bfloat16)
|
||||
out = stages.enlarge(video, 8, 8, method="mitchell", match=True)
|
||||
assert out.dtype == torch.bfloat16 and torch.isfinite(out.float()).all()
|
||||
|
||||
Reference in New Issue
Block a user