feat: block repeat + the detail measurement that can falsify it
h3_block_repeat runs the named block(s) on their own output before
passing it on — a second pass that never leaves latent space. To first
order this doubles that block's contribution; the nonlinear part is a
genuine second refinement of its own result. ~2% of a step per repeat.
detail_probe scores whether that (or anything else) actually sharpened
the picture, by comparing a run against the previous one on three
numbers:
detail high-frequency energy — did fine structure increase
structure low-frequency survival — same shot, or a different one
edge-aligned did the EXTRA detail land on existing edges, or spread
The third is the one that earns its place: noise is high-frequency too,
so `detail` alone cannot tell sharpening from grain — and grain is
exactly what an out-of-distribution stream would produce. Verified on
video-latent-shaped data: unsharp-mask reads detail x3.04 / edge +0.885
("sharper"), uniform noise reads detail x1.12 / edge -0.248 ("grain").
Both raised detail; only edge-alignment separated them.
Measured in latent space deliberately — the tensor is already in hand at
the call site DynaShift banks from, so no VAE decode and no new
plumbing. A real approximation, valid for the A/B it is for (same seed,
identical decoder both sides) and documented as ratio-only.
Reuses block_influence's collection switch rather than adding a second
research toggle, and pairs CONSECUTIVE runs, which is the workflow
anyway: generate with it off, generate with it on, read the row.
Also scores second_pass sharpen, quality_sharpness and segmented
detailing — every mechanism here that claims detail and is currently
judged by eye.
This commit is contained in:
digital-garbage
+187
@@ -0,0 +1,187 @@
|
||||
"""Does an intervention add DETAIL, or just noise, or just a different picture?
|
||||
|
||||
Every mechanism here that claims to sharpen -- second_pass_op "sharpen", quality_sharpness,
|
||||
segmented detailing, and now block repeat -- is currently judged by eye. This scores the
|
||||
claim instead, by comparing a run against the one before it on three numbers:
|
||||
|
||||
* **detail** -- high-frequency energy (Laplacian). Did fine structure increase?
|
||||
* **structure**-- how much of the low-frequency picture survived. Same shot, or a new one?
|
||||
* **edge-aligned** -- did the EXTRA high frequency land on edges that already existed, or
|
||||
spread evenly? Noise and ringing are high-frequency too, so `detail` alone cannot tell
|
||||
sharpening from grain; this is the number that can.
|
||||
|
||||
Read together they separate the three outcomes: detail up + edge-aligned + structure kept =
|
||||
genuinely refined. Detail up + not edge-aligned = grain. Structure down = a different
|
||||
generation, and "it looks different" is not "it looks better".
|
||||
|
||||
MEASURED IN LATENT SPACE, deliberately: the video latent is already in hand at the call site
|
||||
(the same tensor DynaShift banks), so there is no VAE decode, no pixel buffer and no new
|
||||
plumbing. That is a real approximation -- the VAE is nonlinear, so latent high-frequency
|
||||
energy is not pixel high-frequency energy. It holds for the comparison this is FOR: an A/B
|
||||
on the same seed where both sides pass through the identical decoder, so the DIRECTION of a
|
||||
change is trustworthy even where its magnitude is not. Do not quote these as absolute image
|
||||
metrics; they are only ever meaningful as a ratio between two runs.
|
||||
|
||||
Pairing is CONSECUTIVE runs, which is the workflow anyway: generate with the toggle off,
|
||||
generate again with it on, read the comparison. One previous latent is kept per key; a run
|
||||
whose latent shape differs from the stored one is not compared (different resolution or
|
||||
length is not an A/B).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import time
|
||||
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
|
||||
MAX_ROWS = 20 # comparisons kept per key; oldest roll off
|
||||
|
||||
_LAPLACIAN = torch.tensor([[0., 1., 0.], [1., -4., 1.], [0., 1., 0.]])
|
||||
_BLUR = torch.ones(1, 1, 5, 5) / 25.0
|
||||
|
||||
|
||||
def _log():
|
||||
try:
|
||||
from . import funpack_log as fl
|
||||
except ImportError:
|
||||
import funpack_log as fl
|
||||
return fl
|
||||
|
||||
|
||||
def collection_enabled():
|
||||
"""Rides the block-influence switch -- both are research measurement, opted into from
|
||||
the same place, and a second toggle for the same decision is a worse UI, not a safer one."""
|
||||
try:
|
||||
from . import block_influence as bi
|
||||
except ImportError:
|
||||
import block_influence as bi
|
||||
return bi.collection_enabled()
|
||||
|
||||
|
||||
def state_path(refinement_key):
|
||||
try:
|
||||
from .conditioning import refinement_state_path
|
||||
except ImportError:
|
||||
from conditioning import refinement_state_path
|
||||
return refinement_state_path(refinement_key, "detail_probe", prefix="refine_v2",
|
||||
extension="pt")
|
||||
|
||||
|
||||
def _load(refinement_key):
|
||||
path = state_path(refinement_key)
|
||||
if not os.path.exists(path):
|
||||
return {"rows": [], "previous": None}
|
||||
try:
|
||||
data = torch.load(path, map_location="cpu", weights_only=True)
|
||||
except Exception: # noqa: BLE001
|
||||
return {"rows": [], "previous": None}
|
||||
if not isinstance(data, dict):
|
||||
return {"rows": [], "previous": None}
|
||||
data.setdefault("rows", [])
|
||||
data.setdefault("previous", None)
|
||||
return data
|
||||
|
||||
|
||||
def _save(refinement_key, data):
|
||||
path = state_path(refinement_key)
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
tmp = path + ".tmp"
|
||||
torch.save(data, tmp)
|
||||
os.replace(tmp, path)
|
||||
|
||||
|
||||
def _planes(latent):
|
||||
"""Any [..., H, W] video latent -> [N, 1, H, W] float32 planes. Leading dims are pooled
|
||||
rather than interpreted: this needs spatial structure, not a channel/time convention, and
|
||||
guessing which axis is time is how the LTXAV/H3 axis disagreement bites."""
|
||||
x = latent.detach().float()
|
||||
if x.dim() < 2 or x.shape[-1] < 3 or x.shape[-2] < 3:
|
||||
return None
|
||||
return x.reshape(-1, 1, x.shape[-2], x.shape[-1])
|
||||
|
||||
|
||||
def _hf(planes):
|
||||
"""Per-plane high-frequency magnitude map."""
|
||||
k = _LAPLACIAN.view(1, 1, 3, 3).to(planes)
|
||||
return F.conv2d(planes, k, padding=1).abs()
|
||||
|
||||
|
||||
def _lf(planes):
|
||||
k = _BLUR.to(planes)
|
||||
return F.conv2d(planes, k, padding=2)
|
||||
|
||||
|
||||
def _corr(a, b):
|
||||
a = a.flatten() - a.mean()
|
||||
b = b.flatten() - b.mean()
|
||||
d = a.norm() * b.norm()
|
||||
return float((a @ b / d).item()) if float(d) > 1e-12 else 0.0
|
||||
|
||||
|
||||
def compare(before, after):
|
||||
"""-> {"detail": ratio, "structure": 0..1, "edge_aligned": -1..1} or None.
|
||||
|
||||
detail = mean HF energy after / before. >1 means finer structure appeared.
|
||||
structure = 1 - ||LF(after) - LF(before)|| / ||LF(before)||, clamped to 0. Near 1 means
|
||||
the same picture underneath; a low value means the content itself moved.
|
||||
edge_aligned = correlation between the HF INCREASE and the HF that was already there.
|
||||
High means the extra detail landed on existing edges (sharpening); near 0
|
||||
means it spread evenly over flat regions too (grain)."""
|
||||
pa, pb = _planes(before), _planes(after)
|
||||
if pa is None or pb is None or pa.shape != pb.shape:
|
||||
return None
|
||||
hf_a, hf_b = _hf(pa), _hf(pb)
|
||||
base = float(hf_a.mean())
|
||||
if not (base > 1e-12):
|
||||
return None
|
||||
lf_a, lf_b = _lf(pa), _lf(pb)
|
||||
denom = float(lf_a.norm())
|
||||
structure = 1.0 - float((lf_b - lf_a).norm()) / denom if denom > 1e-12 else 0.0
|
||||
return {
|
||||
"detail": float(hf_b.mean()) / base,
|
||||
"structure": max(0.0, min(1.0, structure)),
|
||||
"edge_aligned": _corr(hf_b - hf_a, hf_a),
|
||||
}
|
||||
|
||||
|
||||
def record(refinement_key, video_latent, label=""):
|
||||
"""Score this run against the previous one on the same key, then become the new previous.
|
||||
-> the comparison dict (with "label_before"/"label_after"), or None when there was nothing
|
||||
to compare against yet."""
|
||||
if not refinement_key or not isinstance(video_latent, torch.Tensor):
|
||||
return None
|
||||
try:
|
||||
lat = video_latent.detach()
|
||||
if lat.dim() >= 4 and lat.shape[0] == 1:
|
||||
lat = lat.squeeze(0)
|
||||
lat = lat.to(torch.float16).cpu()
|
||||
data = _load(refinement_key)
|
||||
prev = data.get("previous")
|
||||
row = None
|
||||
if isinstance(prev, dict) and isinstance(prev.get("latent"), torch.Tensor):
|
||||
row = compare(prev["latent"], lat)
|
||||
if row is not None:
|
||||
row["label_before"] = str(prev.get("label") or "")
|
||||
row["label_after"] = str(label or "")
|
||||
row["stamp"] = time.strftime("%Y-%m-%d_%H-%M-%S")
|
||||
data["rows"] = (data["rows"] + [row])[-MAX_ROWS:]
|
||||
data["previous"] = {"latent": lat, "label": str(label or "")}
|
||||
_save(refinement_key, data)
|
||||
return row
|
||||
except Exception as e: # noqa: BLE001
|
||||
_log().failed("H3 detail probe", "record", e, "this run is not scored")
|
||||
return None
|
||||
|
||||
|
||||
def rows(refinement_key):
|
||||
return _load(refinement_key).get("rows") or []
|
||||
|
||||
|
||||
def clear_all(refinement_key):
|
||||
if not refinement_key:
|
||||
return
|
||||
try:
|
||||
os.remove(state_path(refinement_key))
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
@@ -265,6 +265,8 @@ _EDITOR_DEFAULTS = {
|
||||
"embed_guidance_source": "relative",
|
||||
"embed_guidance_strength": 0.02,
|
||||
"frame_overlap": 16,
|
||||
"h3_block_repeat": "",
|
||||
"h3_block_repeat_times": 1,
|
||||
"h3_phrase_emphasis": False,
|
||||
"h3_phrase_variability": 0.0,
|
||||
"h3_repr_steering": False,
|
||||
|
||||
@@ -235,6 +235,9 @@
|
||||
link.remove();
|
||||
URL.revokeObjectURL(url);
|
||||
},
|
||||
detailProbeStatus: (key) => j("GET", API("/detail_probe") + `?key=${encodeURIComponent(key || "default")}`),
|
||||
detailProbeClear: (key) => j("POST", API("/detail_probe/clear"), { key: key || "default" }),
|
||||
|
||||
// Served by ComfyUI's own same-origin /view endpoint — no editor route needed.
|
||||
tempFileUrl: (f) => "/view?" + new URLSearchParams({
|
||||
filename: f.filename, subfolder: f.subfolder || "", type: "temp", t: String(f.mtime || Date.now()),
|
||||
|
||||
@@ -305,6 +305,11 @@
|
||||
hint: "How blurry those frames get while they're blurred. Higher = looser guidance and more freedom to move." },
|
||||
{ name: "alg_guide_blur_sigma_threshold", label: "Guide blur sigma threshold", kind: "float", default: 0.975, min: 0.5, max: 0.999, step: 0.005, dependsOn: "alg_blur_guides",
|
||||
hint: "How long they stay blurred before switching to sharp. Higher = a shorter blurred window." },
|
||||
{ name: "h3_block_repeat", label: "Repeat block(s) (H3, experimental)", kind: "text", default: "",
|
||||
hint: "Runs the named block(s) on their own output before passing it on — a second pass that never leaves latent space. Blank = off. A block, a range, or a list: 40 | 38-42 | 10,40,44.",
|
||||
detail: "Because a block's contribution is small, running it twice roughly doubles what that block does, with the nonlinear part being a genuine second refinement of its own result. Costs one extra block forward per repeat (~2% of a step each) — no VAE round trip, no re-noising, no extra sampler step. Unvalidated: the blocks AFTER a repeated one were trained on its normal output, so a twice-processed stream is mildly out of distribution for them. Turn on measurement recording in Refinement & Taste and generate the same seed with this off, then on, to see whether it sharpens or just adds grain." },
|
||||
{ name: "h3_block_repeat_times", label: "Extra passes per repeated block", kind: "int", default: 1, min: 1, max: 4,
|
||||
hint: "1 = the block runs twice in total. Higher pushes further from what the following blocks were trained to receive." },
|
||||
{ name: "bounded_attention_enabled", label: "Bounded attention (experimental, untested)", kind: "bool", default: false,
|
||||
hint: "Stops two people in one frame swapping each other's features, by letting each half of the frame see only its own sentence. Nearly free, and does nothing unless the scene prompt has two sentences describing two subjects. Built but never run — the left/right split is fixed, so it suits two subjects side by side and nothing else." },
|
||||
{ name: "dynashift", label: "DynaShift (steer off bad gens)", kind: "bool", default: false,
|
||||
@@ -510,7 +515,7 @@
|
||||
chain_timing: ["frame_overlap", "transition_duration", "use_same_seed", "cut_opening_frames"],
|
||||
chain_guidance: ["cfg", "embed_guidance", "embed_guidance_source", "embed_guidance_strength", "score_slider", "score_slider_strength", "taste_nearest_prompt", "output_guidance", "output_guidance_strength", "trajectory_guidance", "trajectory_guidance_strength", "dynashift", "dynashift_strength", "dynashift_threshold", "h3_repr_steering", "h3_repr_steering_strength", "h3_repr_steering_block"],
|
||||
chain_decode: ["decode_noise_scale", "decode_timestep", "decode_tile_size"],
|
||||
chain_experimental: ["context_windows", "context_window_length", "context_window_overlap", "context_window_schedule", "context_window_fuse", "context_window_freenoise", "context_window_retain_first", "h3_video_detail", "joyai_memory", "joyai_memory_size", "joyai_fix_frames", "joyai_frame_select", "joyai_memory_strength", "joyai_audio_memory", "v2a_grad_scale", "alg_blur_guides", "alg_guide_blur_strength", "alg_guide_blur_sigma_threshold", "bounded_attention_enabled", "identity_transfer_enabled", "source_id", "phase_scale", "id_strength", "arcface_mode", "debug_log"],
|
||||
chain_experimental: ["context_windows", "context_window_length", "context_window_overlap", "context_window_schedule", "context_window_fuse", "context_window_freenoise", "context_window_retain_first", "h3_video_detail", "joyai_memory", "joyai_memory_size", "joyai_fix_frames", "joyai_frame_select", "joyai_memory_strength", "joyai_audio_memory", "v2a_grad_scale", "alg_blur_guides", "alg_guide_blur_strength", "alg_guide_blur_sigma_threshold", "bounded_attention_enabled", "h3_block_repeat", "h3_block_repeat_times", "identity_transfer_enabled", "source_id", "phase_scale", "id_strength", "arcface_mode", "debug_log"],
|
||||
};
|
||||
|
||||
function countChainView(p, id, st) {
|
||||
|
||||
@@ -401,6 +401,65 @@
|
||||
// only on explicit request (the "Read" button below).
|
||||
let reinsState = null, reinsSweep = null, reinsSweepBusy = false, reinsError = "";
|
||||
let biState = null, biError = "";
|
||||
let dpRows = null, dpError = "";
|
||||
|
||||
// Reads a comparison the way the user would judge it by eye, but from three numbers that
|
||||
// can disagree with each other -- that disagreement is the whole point. Detail up on
|
||||
// existing edges with the picture intact is sharpening; detail up spread evenly is grain;
|
||||
// structure down means it is simply a different generation.
|
||||
function dpVerdict(r) {
|
||||
if (r.structure < 0.85) return "different picture — not an A/B of the same shot";
|
||||
const gain = (r.detail - 1) * 100;
|
||||
if (Math.abs(gain) < 2) return "no real change in detail";
|
||||
if (gain < 0) return `detail DOWN ${Math.abs(gain).toFixed(0)}% — softer, not sharper`;
|
||||
return r.edge_aligned > 0.35
|
||||
? `sharper — +${gain.toFixed(0)}% detail, landing on existing edges`
|
||||
: `+${gain.toFixed(0)}% detail but spread evenly — reads as grain, not sharpening`;
|
||||
}
|
||||
|
||||
function detailProbeRows(key) {
|
||||
const box = el("div", "sw-stack");
|
||||
const paint = () => {
|
||||
clear(box);
|
||||
const rows = el("div", "sw-rows");
|
||||
const n = dpRows ? dpRows.length : 0;
|
||||
rows.append(actionRow("Start fresh",
|
||||
"Throws away every recorded comparison for this key.",
|
||||
"Clear…", async () => {
|
||||
dpError = "";
|
||||
try { const r = await window.MovieEditorAPI.detailProbeClear(key); dpRows = r.rows; }
|
||||
catch (e) { dpError = String(e.message || e); }
|
||||
paint();
|
||||
}, { disabled: !n, danger: true }));
|
||||
box.append(rows);
|
||||
if (dpError) box.append(el("div", "sw-hint", dpError));
|
||||
if (!n) {
|
||||
box.append(el("div", "sw-hint",
|
||||
"Nothing compared yet. With recording on, generate the same seed twice — once "
|
||||
+ "with the change off, once with it on — and each pair is scored here."));
|
||||
return;
|
||||
}
|
||||
const tbl = el("div", "sw-rows");
|
||||
// Newest first: the run you just made is the one you are asking about.
|
||||
dpRows.slice().reverse().forEach((r) => {
|
||||
tbl.append(infoRow(
|
||||
`${r.label_before || "?"} → ${r.label_after || "?"}`,
|
||||
dpVerdict(r),
|
||||
r.structure >= 0.85 && r.detail > 1.02 && r.edge_aligned > 0.35));
|
||||
tbl.append(el("div", "sw-hint",
|
||||
`detail ×${r.detail.toFixed(3)} · structure kept ${r.structure.toFixed(3)} · `
|
||||
+ `edge-aligned ${r.edge_aligned >= 0 ? "+" : ""}${r.edge_aligned.toFixed(3)}`));
|
||||
});
|
||||
box.append(tbl);
|
||||
};
|
||||
paint();
|
||||
if (window.MovieEditorAPI) {
|
||||
window.MovieEditorAPI.detailProbeStatus(key)
|
||||
.then((r) => { dpRows = r.rows || []; paint(); })
|
||||
.catch(() => {});
|
||||
}
|
||||
return box;
|
||||
}
|
||||
|
||||
// The whole point of the probe in one number. Flatness is the spread of the per-block
|
||||
// profile as a fraction of its own mean, so it is comparable across models and runs: near
|
||||
@@ -792,6 +851,10 @@
|
||||
wrap.append(el("div", "sw-rows-label", "Block influence (measurement only)"));
|
||||
wrap.append(blockInfluenceRows(st.project?.refinement_key || "default"));
|
||||
|
||||
wrap.append(el("div", "sw-rows-label",
|
||||
"Detail check — did a change sharpen the picture, or just alter it?"));
|
||||
wrap.append(detailProbeRows(st.project?.refinement_key || "default"));
|
||||
|
||||
wrap.append(el("div", "sw-rows-label", "Danger zone"));
|
||||
const danger = el("div", "sw-rows");
|
||||
danger.append(actionRow("Delete refinement key",
|
||||
|
||||
@@ -2254,6 +2254,33 @@ if web is not None and PromptServer is not None:
|
||||
_bi, state = _block_influence_state(key)
|
||||
return web.json_response(state)
|
||||
|
||||
# Detail probe: rides block_influence's collection switch (same research opt-in), so it
|
||||
# only needs a readout and a clear.
|
||||
def _detail_module():
|
||||
try:
|
||||
import detail_probe as dp
|
||||
except ImportError:
|
||||
from .. import detail_probe as dp # type: ignore
|
||||
return dp
|
||||
|
||||
@routes.get(UI_PREFIX + "/api/detail_probe")
|
||||
async def _detail_status(req):
|
||||
key = str(req.rel_url.query.get("key") or "default").strip() or "default"
|
||||
try:
|
||||
return web.json_response({"key": key, "rows": _detail_module().rows(key)})
|
||||
except Exception as e:
|
||||
return web.json_response({"key": key, "rows": [], "error": str(e)})
|
||||
|
||||
@routes.post(UI_PREFIX + "/api/detail_probe/clear")
|
||||
async def _detail_clear(req):
|
||||
try:
|
||||
body = await req.json()
|
||||
except Exception:
|
||||
body = {}
|
||||
key = str(body.get("key") or "default").strip() or "default"
|
||||
_detail_module().clear_all(key)
|
||||
return web.json_response({"key": key, "rows": []})
|
||||
|
||||
@routes.get(UI_PREFIX + "/api/temp")
|
||||
async def _temp_list(_req):
|
||||
return web.json_response({"files": _list_temp_media()})
|
||||
|
||||
+110
-1
@@ -2970,6 +2970,14 @@ class FunPackLTXAVSceneChainSampler:
|
||||
"default": "25",
|
||||
"tooltip": "TEST-ONLY: which block actually gets steered (the rest of the sweep's candidate blocks are still captured read-only, same as always). direction() is recomputed for whichever block you pick here -- this does not change what block_sweep ranks, only which block's learned direction gets injected into this run.",
|
||||
}),
|
||||
"h3_block_repeat": ("STRING", {
|
||||
"default": "",
|
||||
"tooltip": "EXPERIMENTAL, unvalidated (H3). Runs the named block(s) on their OWN output before passing the result on -- a second pass that never leaves latent space (no VAE round trip, no re-noising, no extra sampler step). Because a block's contribution is small, running it twice roughly DOUBLES what that block does, with the nonlinear part being a genuine second refinement of its own result. Blank = off. Accepts a block, a range, or a list: 40 | 38-42 | 10,40,44. Costs one extra block forward per repeat (~2% of a step each). The blocks AFTER a repeated one were trained on its normal output, so a twice-processed stream is mildly out of distribution for them -- expect this to be visible, and not always as an improvement.",
|
||||
}),
|
||||
"h3_block_repeat_times": ("INT", {
|
||||
"default": 1, "min": 1, "max": 4,
|
||||
"tooltip": "How many EXTRA passes each named block gets. 1 = the block runs twice in total. Higher pushes further from what the following blocks were trained to receive.",
|
||||
}),
|
||||
# A connection socket, never a widget — safe at the end, and it must stay after
|
||||
# every widget above (see the widgets_values note at the top of this block).
|
||||
"second_pass_sigmas": ("SIGMAS", {
|
||||
@@ -3679,6 +3687,7 @@ class FunPackLTXAVSceneChainSampler:
|
||||
bounded_attention_enabled=False,
|
||||
h3_repr_steering=False, h3_repr_steering_strength=0.05,
|
||||
h3_repr_steering_block="25",
|
||||
h3_block_repeat="", h3_block_repeat_times=1,
|
||||
refinement_key="",):
|
||||
if sampler is None:
|
||||
raise ValueError("sampler input is required.")
|
||||
@@ -3741,7 +3750,12 @@ class FunPackLTXAVSceneChainSampler:
|
||||
# key, no rating and no Studio state, so it works on a graph with the Refiner absent
|
||||
# entirely. 1.0 on all three does not even clone the model.
|
||||
model = self._install_h3_final_layer(model, positive)
|
||||
# Measurement first, steering second: whatever is installed later chains this probe
|
||||
# Block repeat is innermost: it changes what the block DOES, so the probe outside it
|
||||
# reports the block's total contribution including the repeat.
|
||||
_repeat_blocks = self._parse_block_spec(h3_block_repeat)
|
||||
if _repeat_blocks:
|
||||
model = self._install_block_repeat(model, _repeat_blocks, h3_block_repeat_times)
|
||||
# Measurement next, steering last: whatever is installed later chains this probe
|
||||
# as its inner call, so the delta it records is the block's OWN, not one that already
|
||||
# contains another mechanism's injection.
|
||||
_influence_capture = [{}, {}] # [0] relative magnitudes, [1] novelty cosines
|
||||
@@ -6765,6 +6779,87 @@ class FunPackLTXAVSceneChainSampler:
|
||||
"not applied and not capturing this run")
|
||||
return model
|
||||
|
||||
@staticmethod
|
||||
def _parse_block_spec(spec, n_blocks=50):
|
||||
""""40" | "38-42" | "10,40,44" | "" -> a set of block indices. Anything unparseable
|
||||
in a comma group is skipped rather than failing the run: this is a text field on an
|
||||
experimental knob, and a typo should cost you that entry, not the generation."""
|
||||
out = set()
|
||||
for part in str(spec or "").replace(" ", "").split(","):
|
||||
if not part:
|
||||
continue
|
||||
try:
|
||||
if "-" in part[1:]:
|
||||
a, b = part.split("-", 1)
|
||||
lo, hi = int(a), int(b)
|
||||
if lo > hi:
|
||||
lo, hi = hi, lo
|
||||
out.update(range(max(0, lo), min(int(n_blocks) - 1, hi) + 1))
|
||||
else:
|
||||
i = int(part)
|
||||
if 0 <= i < int(n_blocks):
|
||||
out.add(i)
|
||||
except ValueError:
|
||||
continue
|
||||
return out
|
||||
|
||||
def _install_block_repeat(self, model, blocks, times=1):
|
||||
"""EXPERIMENTAL (H3): run a block on its OWN output before passing it on.
|
||||
|
||||
`x' = x + f(x)` then `x'' = x' + f(x')`. Because the delta is small, f(x') is close to
|
||||
f(x), so to first order this doubles that block's contribution -- it sits between a
|
||||
2x per-block gain and a genuine second refinement pass, and the difference between
|
||||
those two IS the nonlinear part, which is the point of doing it this way rather than
|
||||
just scaling the output.
|
||||
|
||||
A second pass that never leaves latent space: no VAE round trip, no re-noising, no
|
||||
extra sampler step. Cost is one extra block forward per repeat -- ~2% of a step per
|
||||
repeated block on a 50-block stack.
|
||||
|
||||
UNVALIDATED, and there is a known reason to expect trouble: the blocks AFTER this one
|
||||
were trained on this block's output as it normally is, and a twice-processed stream is
|
||||
mildly out of distribution for them. Depth-upscaled models that duplicate layers this
|
||||
way (SOLAR-style) need continued pretraining to recover quality; nothing is retrained
|
||||
here. Whether that reads as finer detail or as artifacts is exactly what the toggle is
|
||||
for. Installed INNERMOST (before the influence probe) so the probe reports the block's
|
||||
total contribution including the repeat, not just its first pass."""
|
||||
try:
|
||||
if not blocks or int(times) < 1:
|
||||
return model
|
||||
patched = model.clone()
|
||||
to = patched.model_options.get("transformer_options", {}).copy()
|
||||
patches_replace = dict(to.get("patches_replace", {}))
|
||||
dit_patches = dict(patches_replace.get("dit", {}))
|
||||
_times = int(times)
|
||||
|
||||
def _make_repeat(block):
|
||||
inner = dit_patches.get(("double_block", block))
|
||||
|
||||
def _repeat(args, extra):
|
||||
run = (lambda a: extra["original_block"](a)["img"]) if inner is None else \
|
||||
(lambda a: inner(a, extra)["img"])
|
||||
out = run(args)
|
||||
for _ in range(_times):
|
||||
# A fresh args dict each pass: the block reads "img" from it, and
|
||||
# mutating the caller's dict would leak the intermediate state back
|
||||
# to whatever else reads args after this hook returns.
|
||||
out = run({**args, "img": out})
|
||||
return {"img": out}
|
||||
return _repeat
|
||||
|
||||
for _b in sorted(blocks):
|
||||
dit_patches[("double_block", _b)] = _make_repeat(_b)
|
||||
patches_replace["dit"] = dit_patches
|
||||
to["patches_replace"] = patches_replace
|
||||
patched.model_options["transformer_options"] = to
|
||||
print(f"[FunPackSceneChain] H3 block repeat: block(s) "
|
||||
f"{', '.join(str(b) for b in sorted(blocks))} run {_times + 1}x "
|
||||
f"(+{_times} extra pass each).")
|
||||
return patched
|
||||
except Exception as _e: # noqa: BLE001
|
||||
_log.failed("FunPackSceneChain", "H3 block repeat", _e, "blocks run once as usual")
|
||||
return model
|
||||
|
||||
def _install_block_influence(self, model, capture_holder, n_blocks=50, max_rows=512):
|
||||
"""MEASUREMENT ONLY (H3). See block_influence.py.
|
||||
|
||||
@@ -7118,6 +7213,7 @@ class FunPackLTXAVSceneChainSampler:
|
||||
dynashift=False, dynashift_strength=0.3, dynashift_threshold=0.6,
|
||||
h3_repr_steering=False, h3_repr_steering_strength=0.05,
|
||||
h3_repr_steering_block="25",
|
||||
h3_block_repeat="", h3_block_repeat_times=1,
|
||||
alg_guide_blur_strength=2.0, alg_guide_blur_sigma_threshold=0.975,
|
||||
alg_anchor=False, alg_anchor_strength=2.0, alg_anchor_sigma_threshold=0.975,
|
||||
identity_transfer_enabled=False, identity_projector="None", source_id=2.0,
|
||||
@@ -7974,6 +8070,8 @@ class FunPackLTXAVSceneChainSampler:
|
||||
h3_repr_steering=h3_repr_steering,
|
||||
h3_repr_steering_strength=h3_repr_steering_strength,
|
||||
h3_repr_steering_block=h3_repr_steering_block,
|
||||
h3_block_repeat=h3_block_repeat,
|
||||
h3_block_repeat_times=h3_block_repeat_times,
|
||||
refinement_key=refinement_key_input,
|
||||
)
|
||||
# cut_opening_frames: let the real, untouched i2v anchor condition the scene
|
||||
@@ -8545,6 +8643,17 @@ class FunPackLTXAVSceneChainSampler:
|
||||
_cond0 if isinstance(_cond0, torch.Tensor) else None)
|
||||
except Exception as _e:
|
||||
print(f"[FunPackSceneChain] dynashift pending save failed: {_e}")
|
||||
# Detail probe: scores THIS run's fine structure against the previous run on
|
||||
# the same key. Same latent, no decode. Labelled with what actually differs
|
||||
# between A and B runs, so the comparison row says which way round it was.
|
||||
try:
|
||||
from . import detail_probe as _dp
|
||||
except ImportError:
|
||||
import detail_probe as _dp
|
||||
if _dp.collection_enabled():
|
||||
_dp.record(refinement_key_input, _snap,
|
||||
label=(f"repeat {h3_block_repeat}x{int(h3_block_repeat_times) + 1}"
|
||||
if self._parse_block_spec(h3_block_repeat) else "no repeat"))
|
||||
|
||||
# Trajectory probe: same run/rating pairing as the two above — the rating that scores
|
||||
# THIS run appends these per-bucket descriptors to the measurement log.
|
||||
|
||||
@@ -0,0 +1,263 @@
|
||||
"""Detail probe: the three numbers that separate "sharper" from "grainier" from "different",
|
||||
and the block-spec parser feeding the repeat toggle they exist to judge."""
|
||||
import sys
|
||||
import types
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
for _name in (
|
||||
"comfy", "comfy.k_diffusion", "comfy.k_diffusion.sampling",
|
||||
"comfy.model_sampling", "comfy.nested_tensor", "comfy.sample",
|
||||
"comfy.samplers", "comfy.utils",
|
||||
):
|
||||
sys.modules.setdefault(_name, types.ModuleType(_name))
|
||||
sys.modules["comfy.nested_tensor"].NestedTensor = object
|
||||
|
||||
import detail_probe as dp # noqa: E402
|
||||
import samplers # noqa: E402
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolate(monkeypatch, tmp_path):
|
||||
monkeypatch.setattr(dp, "state_path", lambda key: str(tmp_path / f"{key}.pt"))
|
||||
|
||||
|
||||
def _edged(h=32, w=32, seed=0):
|
||||
"""A plane with real structure: a few hard edges, flat elsewhere."""
|
||||
g = torch.Generator().manual_seed(seed)
|
||||
x = torch.zeros(1, 1, h, w)
|
||||
x[:, :, :, w // 3:] = 1.0 # vertical edge
|
||||
x[:, :, h // 2:, :] += 0.5 # horizontal edge
|
||||
x += torch.rand(1, 1, h, w, generator=g) * 0.01
|
||||
return x
|
||||
|
||||
|
||||
# --- the three numbers ----------------------------------------------------------
|
||||
|
||||
def test_identical_inputs_are_a_perfect_null():
|
||||
a = _edged()
|
||||
r = dp.compare(a, a.clone())
|
||||
assert r["detail"] == pytest.approx(1.0)
|
||||
assert r["structure"] == pytest.approx(1.0)
|
||||
|
||||
|
||||
def test_sharpening_existing_edges_reads_as_edge_aligned_detail():
|
||||
"""An unsharp-mask style boost: more high frequency, concentrated where edges already
|
||||
were, low frequencies untouched. This is the outcome the probe exists to recognise."""
|
||||
a = _edged()
|
||||
hf = dp._hf(a)
|
||||
b = a + 0.6 * torch.nn.functional.conv2d(
|
||||
a, dp._LAPLACIAN.view(1, 1, 3, 3), padding=1) * -1.0
|
||||
r = dp.compare(a, b)
|
||||
assert r["detail"] > 1.1 # finer structure appeared
|
||||
assert r["structure"] > 0.9 # same picture underneath
|
||||
assert r["edge_aligned"] > 0.5 # and it landed on the edges
|
||||
assert float(hf.mean()) > 0 # sanity: the baseline had edges at all
|
||||
|
||||
|
||||
def test_uniform_noise_is_detail_without_edge_alignment():
|
||||
"""Grain raises `detail` exactly like sharpening does -- `edge_aligned` is the only
|
||||
number that separates them, which is the reason it exists."""
|
||||
a = _edged()
|
||||
g = torch.Generator().manual_seed(1)
|
||||
b = a + torch.randn(a.shape, generator=g) * 0.05
|
||||
r = dp.compare(a, b)
|
||||
assert r["detail"] > 1.1 # looks like "more detail"
|
||||
assert r["edge_aligned"] < 0.35 # but it is spread everywhere
|
||||
|
||||
|
||||
def test_a_different_picture_shows_up_as_lost_structure():
|
||||
a = _edged(seed=0)
|
||||
b = _edged(seed=1) * 0.3 + 0.7 # same kind of content, different low frequencies
|
||||
r = dp.compare(a, b)
|
||||
assert r["structure"] < 0.85
|
||||
|
||||
|
||||
def test_blurring_reads_as_detail_lost():
|
||||
a = _edged()
|
||||
b = dp._lf(a)
|
||||
assert dp.compare(a, b)["detail"] < 1.0
|
||||
|
||||
|
||||
def test_shape_mismatch_is_not_compared():
|
||||
assert dp.compare(_edged(32, 32), _edged(16, 16)) is None
|
||||
|
||||
|
||||
def test_a_flat_baseline_is_refused_rather_than_dividing_by_zero():
|
||||
flat = torch.zeros(1, 1, 16, 16)
|
||||
assert dp.compare(flat, flat + 1.0) is None
|
||||
|
||||
|
||||
def test_planes_pool_leading_dims_whatever_they_mean():
|
||||
"""C/T ordering differs between model families; this needs spatial structure only, so it
|
||||
must not care which leading axis is which."""
|
||||
x = torch.randn(4, 7, 16, 16)
|
||||
assert dp._planes(x).shape == (28, 1, 16, 16)
|
||||
assert dp._planes(torch.randn(2, 3, 5, 16, 16)).shape == (30, 1, 16, 16)
|
||||
|
||||
|
||||
def test_too_small_to_filter_is_none():
|
||||
assert dp._planes(torch.randn(2, 2)) is None
|
||||
|
||||
|
||||
# --- pairing consecutive runs ---------------------------------------------------
|
||||
|
||||
def test_the_first_run_has_nothing_to_compare_against():
|
||||
assert dp.record("k", _edged(), label="no repeat") is None
|
||||
assert dp.rows("k") == []
|
||||
|
||||
|
||||
def test_the_second_run_is_scored_against_the_first():
|
||||
dp.record("k", _edged(), label="no repeat")
|
||||
row = dp.record("k", _edged(), label="repeat 40")
|
||||
assert row is not None
|
||||
assert row["label_before"] == "no repeat" and row["label_after"] == "repeat 40"
|
||||
assert len(dp.rows("k")) == 1
|
||||
|
||||
|
||||
def test_each_run_becomes_the_baseline_for_the_next():
|
||||
for i in range(3):
|
||||
dp.record("k", _edged(seed=i), label=f"run{i}")
|
||||
rows = dp.rows("k")
|
||||
assert [r["label_before"] for r in rows] == ["run0", "run1"]
|
||||
assert [r["label_after"] for r in rows] == ["run1", "run2"]
|
||||
|
||||
|
||||
def test_a_resolution_change_is_not_an_ab():
|
||||
dp.record("k", _edged(32, 32), label="a")
|
||||
assert dp.record("k", _edged(16, 16), label="b") is None
|
||||
assert dp.rows("k") == []
|
||||
# ...but it still becomes the baseline, so the NEXT matching run pairs with it.
|
||||
assert dp.record("k", _edged(16, 16), label="c") is not None
|
||||
|
||||
|
||||
def test_rows_are_capped():
|
||||
for i in range(dp.MAX_ROWS + 5):
|
||||
dp.record("k", _edged(seed=i % 3), label=f"r{i}")
|
||||
assert len(dp.rows("k")) == dp.MAX_ROWS
|
||||
|
||||
|
||||
def test_clear_removes_everything():
|
||||
dp.record("k", _edged(), label="a")
|
||||
dp.record("k", _edged(), label="b")
|
||||
dp.clear_all("k")
|
||||
assert dp.rows("k") == []
|
||||
|
||||
|
||||
def test_a_non_tensor_is_a_safe_no_op():
|
||||
assert dp.record("k", None) is None
|
||||
assert dp.record("", _edged()) is None
|
||||
|
||||
|
||||
def test_corrupt_state_degrades_to_empty(tmp_path, monkeypatch):
|
||||
path = tmp_path / "junk.pt"
|
||||
path.write_bytes(b"not a torch file")
|
||||
monkeypatch.setattr(dp, "state_path", lambda key: str(path))
|
||||
assert dp.rows("k") == []
|
||||
|
||||
|
||||
# --- the block-spec parser the repeat toggle reads ------------------------------
|
||||
|
||||
@pytest.mark.parametrize("spec,expected", [
|
||||
("", set()),
|
||||
("40", {40}),
|
||||
("38-42", {38, 39, 40, 41, 42}),
|
||||
("10,40,44", {10, 40, 44}),
|
||||
("10, 40 , 44", {10, 40, 44}), # whitespace
|
||||
("42-38", {38, 39, 40, 41, 42}), # reversed range
|
||||
("48-99", {48, 49}), # clamped to the stack
|
||||
("-5", set()), # negative: no leading-dash range
|
||||
("99", set()), # out of range
|
||||
("40,junk,41", {40, 41}), # a typo costs that entry, not the run
|
||||
("40-", set()),
|
||||
])
|
||||
def test_block_spec_parsing(spec, expected):
|
||||
assert samplers.FunPackLTXAVSceneChainSampler._parse_block_spec(spec, 50) == expected
|
||||
|
||||
|
||||
# --- the repeat hook actually repeats -------------------------------------------
|
||||
|
||||
class _Model:
|
||||
def __init__(self):
|
||||
self.model_options = {}
|
||||
|
||||
def clone(self):
|
||||
m = _Model()
|
||||
m.model_options = {k: dict(v) if isinstance(v, dict) else v
|
||||
for k, v in self.model_options.items()}
|
||||
return m
|
||||
|
||||
|
||||
def _repeat_hook(blocks, times):
|
||||
s = samplers.FunPackLTXAVSceneChainSampler()
|
||||
patched = s._install_block_repeat(_Model(), blocks, times)
|
||||
return patched.model_options["transformer_options"]["patches_replace"]["dit"]
|
||||
|
||||
|
||||
def test_the_block_runs_once_more_per_extra_pass():
|
||||
calls = []
|
||||
|
||||
def original(args):
|
||||
calls.append(args["img"].clone())
|
||||
return {"img": args["img"] + 1.0}
|
||||
|
||||
dit = _repeat_hook({0}, 1)
|
||||
out = dit[("double_block", 0)]({"img": torch.zeros(2, 2)}, {"original_block": original})
|
||||
assert len(calls) == 2 # ran twice
|
||||
assert torch.equal(out["img"], torch.full((2, 2), 2.0))
|
||||
assert torch.equal(calls[1], torch.ones(2, 2)) # 2nd pass saw the 1st pass's output
|
||||
|
||||
|
||||
def test_extra_passes_scale():
|
||||
def original(args):
|
||||
return {"img": args["img"] + 1.0}
|
||||
|
||||
dit = _repeat_hook({0}, 3)
|
||||
out = dit[("double_block", 0)]({"img": torch.zeros(1, 1)}, {"original_block": original})
|
||||
assert float(out["img"]) == 4.0 # 1 normal pass + 3 extra
|
||||
|
||||
|
||||
def test_only_the_named_blocks_are_patched():
|
||||
dit = _repeat_hook({0, 3}, 1)
|
||||
assert set(b for _tag, b in dit) == {0, 3}
|
||||
|
||||
|
||||
def test_no_blocks_is_an_untouched_model():
|
||||
s = samplers.FunPackLTXAVSceneChainSampler()
|
||||
m = _Model()
|
||||
assert s._install_block_repeat(m, set(), 1) is m
|
||||
|
||||
|
||||
def test_the_callers_args_dict_is_not_mutated():
|
||||
"""The block reads "img" out of args; leaking the intermediate back to the caller would
|
||||
hand whatever reads args after this hook a half-processed stream."""
|
||||
def original(args):
|
||||
return {"img": args["img"] + 1.0}
|
||||
|
||||
dit = _repeat_hook({0}, 2)
|
||||
args = {"img": torch.zeros(1, 1), "t_emb": "sentinel"}
|
||||
dit[("double_block", 0)](args, {"original_block": original})
|
||||
assert float(args["img"]) == 0.0
|
||||
assert args["t_emb"] == "sentinel"
|
||||
|
||||
|
||||
def test_an_existing_patch_at_the_same_block_is_repeated_not_dropped():
|
||||
calls = []
|
||||
|
||||
def inner(args, extra):
|
||||
calls.append(True)
|
||||
return {"img": args["img"] + 1.0}
|
||||
|
||||
s = samplers.FunPackLTXAVSceneChainSampler()
|
||||
m = _Model()
|
||||
m.model_options["transformer_options"] = {
|
||||
"patches_replace": {"dit": {("double_block", 0): inner}}}
|
||||
patched = s._install_block_repeat(m, {0}, 1)
|
||||
hook = patched.model_options["transformer_options"]["patches_replace"]["dit"][
|
||||
("double_block", 0)]
|
||||
out = hook({"img": torch.zeros(1, 1)}, {"original_block": lambda a: {"img": a["img"]}})
|
||||
assert len(calls) == 2 # the existing patch is what gets repeated
|
||||
assert float(out["img"]) == 2.0
|
||||
Reference in New Issue
Block a user