Release 2.1.0
This commit is contained in:
digital-garbage
+17
-3
@@ -1,15 +1,21 @@
|
||||
# Changelog
|
||||
|
||||
## [2.0.0] - 2026-04-23
|
||||
## [2.1.0] - 2026-04-23
|
||||
|
||||
### Added
|
||||
|
||||
Added `FunPack Apply LoRA Weights` and `FunPack LoRA Loader`, a prompt-exact LoRA weight workflow designed to work with `FunPack Gemma Embedding Refiner`.
|
||||
Added `FunPack Apply LoRA Weights` and `FunPack LoRA Loader`, a prompt-exact LoRA weight workflow designed to work with `FunPack Video Refiner`.
|
||||
|
||||
Added `FunPack Save Refinement Latent`, which stores latent tensor bundles by refinement key for optional latent refinement in `FunPack Video Refiner`.
|
||||
|
||||
Added hidden LTX per-block LoRA redistribution for supported `ltx2` model stacks. The UI still exposes normal LoRA weights, while the loader derives per-block strengths from the LoRA patch magnitudes when the model and LoRA layout support it.
|
||||
|
||||
The new workflow uses base LoRA weights on the first run for a prompt, then lets the refiner save prompt-specific suggested LoRA weights into its existing JSON state for later runs.
|
||||
|
||||
### Changed
|
||||
|
||||
Promoted the `FunPack Video Refiner` development workflow to `main` and renamed the visible refiner title from `FunPack Gemma Embedding Refiner` to `FunPack Video Refiner` while keeping the old node key as a compatibility alias.
|
||||
|
||||
Split the old single `funpack.py` implementation into focused modules:
|
||||
|
||||
- `conditioning.py`
|
||||
@@ -19,7 +25,15 @@ Split the old single `funpack.py` implementation into focused modules:
|
||||
|
||||
`funpack.py` remains as a compatibility re-export for older imports.
|
||||
|
||||
Updated `FunPack Gemma Embedding Refiner` so it can accept a FunPack LoRA stack and save next-run model LoRA weight suggestions based on prompt concepts, LoRA type hints, and user ratings.
|
||||
Updated `FunPack Video Refiner` so it can accept a FunPack LoRA stack and save next-run model LoRA weight suggestions based on prompt concepts, LoRA type hints, and user ratings.
|
||||
|
||||
Updated `FunPack Video Refiner` with optional latent input/output refinement. If no matching saved latent exists and both latent input and output are connected, the input latent is saved as the first reference and passed through unchanged.
|
||||
|
||||
Updated prompt analysis so quoted speech and backslash-wrapped phrases can be protected as whole prompt units.
|
||||
|
||||
### Documentation
|
||||
|
||||
Documented unintended and edge-case usage for the new refiner workflow, including disconnected latent paths, saved-latent-only runs, wrong LTX audio/AV latent connections, exact-prompt LoRA lookup behavior, base-weight mismatch behavior, zero-weight LoRA skipping, and unsupported per-block fallback behavior.
|
||||
|
||||
## [1.3.3] - 2026-04-22
|
||||
|
||||
|
||||
@@ -2,32 +2,30 @@
|
||||
|
||||
A set of ComfyUI nodes for experimenting with video generation workflows based on WAN, HunyuanVideo, LTX, and similar models.
|
||||
|
||||
## What's New in 2.0.0
|
||||
## What's New in 2.1.0
|
||||
|
||||
FunPack is now split into focused Python modules for easier editing and maintenance:
|
||||
Version 2.1.0 promotes the latest refiner workflow from `dev` to `main`.
|
||||
|
||||
- `conditioning.py`
|
||||
- `samplers.py`
|
||||
- `image_processing.py`
|
||||
- `model_management.py`
|
||||
`FunPack Gemma Embedding Refiner` is now displayed as `FunPack Video Refiner`. The old node key remains available as a compatibility alias, so existing workflows that reference `FunPackGemmaEmbeddingRefiner` can continue to load.
|
||||
|
||||
Version 2.0.0 also adds a prompt-exact LoRA weight workflow that works together with `FunPack Gemma Embedding Refiner`.
|
||||
The Video Refiner now supports:
|
||||
|
||||
Use this order:
|
||||
- concept-aware feedback and clearer split `status` / `training_info` outputs
|
||||
- protected prompt phrases with either quoted speech or backslash-wrapped phrases
|
||||
- optional sigma refinement
|
||||
- optional video-latent refinement through `FunPack Save Refinement Latent`
|
||||
- prompt-exact LoRA weight suggestions through `FunPack Apply LoRA Weights` and `FunPack LoRA Loader`
|
||||
- hidden per-block LoRA redistribution for supported LTX model stacks
|
||||
|
||||
`FunPack Apply LoRA Weights` -> `FunPack LoRA Loader` -> `FunPack Gemma Embedding Refiner`
|
||||
For the LoRA/refiner workflow, use this order:
|
||||
|
||||
`FunPack Apply LoRA Weights` defines selected LoRAs, LoRA type, and model base weights. If the refiner has saved suggested weights for the exact prompt, it applies them; otherwise it uses the base weights.
|
||||
`FunPack Apply LoRA Weights` -> `FunPack LoRA Loader` -> `FunPack Video Refiner`
|
||||
|
||||
`FunPack LoRA Loader` loads the prepared LoRA stack into the model. Its CLIP input is optional and is left untouched when omitted.
|
||||
For LTX audio/video latent refinement, split the AV latent first:
|
||||
|
||||
`FunPack Gemma Embedding Refiner` does the prompt/concept/rating work and saves next-run LoRA weight suggestions into its existing refinement JSON. It can improve the balance and stability of concepts the model can already express, but it cannot teach a model missing concepts or capabilities from scratch.
|
||||
`LTXVSeparateAVLatent` video output -> `FunPack Save Refinement Latent` / `FunPack Video Refiner` -> `LTXVConcatAVLatent` video input
|
||||
|
||||
## WIP on `dev`
|
||||
|
||||
The `dev` branch contains unstable work-in-progress changes, including the Video Refiner rename, latent refinement experiments, and related LTX video-latent helpers.
|
||||
|
||||
Switch to `dev` only if you intentionally want to test unfinished features. For stable daily use, stay on `main`.
|
||||
Do not send combined AV latents or audio latents into the FunPack latent refinement path. Those structures are intentionally rejected so the workflow does not silently refine the wrong tensor.
|
||||
|
||||
## Installation
|
||||
|
||||
@@ -42,7 +40,7 @@ FunPack is available on Comfy Registry and can be installed in any of these ways
|
||||
|
||||
## Dependencies
|
||||
|
||||
FunPack now includes a [`requirements.txt`](requirements.txt) file for the Python packages it expects outside the standard library.
|
||||
FunPack includes a [`requirements.txt`](requirements.txt) file for the Python packages it expects outside the standard library.
|
||||
|
||||
Install them with:
|
||||
|
||||
@@ -77,7 +75,11 @@ If you install `hpsv3`, use `--no-build-isolation`. Otherwise the install may ap
|
||||
|
||||
Per-node documentation is available in the [`docs`](docs) folder.
|
||||
|
||||
The LoRA/refiner helper workflow is documented in [`docs/FunPackLoraWorkflow.md`](docs/FunPackLoraWorkflow.md).
|
||||
Start with:
|
||||
|
||||
- [`docs/FunPackGemmaEmbeddingRefiner.md`](docs/FunPackGemmaEmbeddingRefiner.md) for `FunPack Video Refiner`
|
||||
- [`docs/FunPackSaveRefinementLatent.md`](docs/FunPackSaveRefinementLatent.md) for latent references
|
||||
- [`docs/FunPackLoraWorkflow.md`](docs/FunPackLoraWorkflow.md) for the LoRA/refiner helper workflow
|
||||
|
||||
Version history is available in [CHANGELOG.md](CHANGELOG.md).
|
||||
|
||||
|
||||
+7
-1
@@ -5,7 +5,9 @@ from .conditioning import (
|
||||
FunPackLorebookEnhancer,
|
||||
FunPackPromptCombiner,
|
||||
FunPackPromptEnhancer,
|
||||
FunPackSaveRefinementLatent,
|
||||
FunPackStoryWriter,
|
||||
FunPackVideoRefiner,
|
||||
)
|
||||
from .image_processing import (
|
||||
FunPackContinueVideo,
|
||||
@@ -28,6 +30,8 @@ NODE_CLASS_MAPPINGS = {
|
||||
"FunPackContinueVideo": FunPackContinueVideo,
|
||||
"FunPackLorebookEnhancer": FunPackLorebookEnhancer,
|
||||
"FunPackGemmaEmbeddingRefiner": FunPackGemmaEmbeddingRefiner,
|
||||
"FunPackVideoRefiner": FunPackVideoRefiner,
|
||||
"FunPackSaveRefinementLatent": FunPackSaveRefinementLatent,
|
||||
"FunPackHybridEuler2SSampler": FunPackHybridEuler2SSampler,
|
||||
"FunPackApplyLoraWeights": FunPackApplyLoraWeights,
|
||||
"FunPackLoraLoader": FunPackLoraLoader,
|
||||
@@ -42,7 +46,9 @@ NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"FunPackVideoStitch": "FunPack Video Stitch",
|
||||
"FunPackContinueVideo": "FunPack Continue Video",
|
||||
"FunPackLorebookEnhancer": "FunPack Lorebook Enhancer",
|
||||
"FunPackGemmaEmbeddingRefiner": "FunPack Gemma Embedding Refiner (Self-Refinement)",
|
||||
"FunPackGemmaEmbeddingRefiner": "FunPack Video Refiner (Compatibility)",
|
||||
"FunPackVideoRefiner": "FunPack Video Refiner",
|
||||
"FunPackSaveRefinementLatent": "FunPack Save Refinement Latent",
|
||||
"FunPackHybridEuler2SSampler": "FunPack Hybrid Euler 2S Sampler",
|
||||
"FunPackApplyLoraWeights": "FunPack Apply LoRA Weights",
|
||||
"FunPackLoraLoader": "FunPack LoRA Loader",
|
||||
|
||||
+574
-50
@@ -19,11 +19,11 @@ from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
|
||||
import folder_paths
|
||||
|
||||
LORA_REFINER_TYPE_PROFILES = {
|
||||
"general": {"step": 0.025, "max_offset": 0.20, "bad_max_offset": 0.45},
|
||||
"concept": {"step": 0.040, "max_offset": 0.35, "bad_max_offset": 0.70},
|
||||
"style": {"step": 0.032, "max_offset": 0.28, "bad_max_offset": 0.58},
|
||||
"quality": {"step": 0.022, "max_offset": 0.18, "bad_max_offset": 0.38},
|
||||
"character": {"step": 0.024, "max_offset": 0.20, "bad_max_offset": 0.42},
|
||||
"general": {"step": 0.025, "max_offset": 0.20, "min_offset": -0.35, "bad_max_offset": 0.45, "bad_min_offset": -1.35, "culprit_bias": 0.28},
|
||||
"concept": {"step": 0.046, "max_offset": 0.35, "min_offset": -0.45, "bad_max_offset": 0.75, "bad_min_offset": -2.10, "culprit_bias": 0.16},
|
||||
"style": {"step": 0.032, "max_offset": 0.28, "min_offset": -0.38, "bad_max_offset": 0.58, "bad_min_offset": -1.55, "culprit_bias": 0.20},
|
||||
"quality": {"step": 0.022, "max_offset": 0.18, "min_offset": -0.30, "bad_max_offset": 0.38, "bad_min_offset": -1.20, "culprit_bias": 0.18},
|
||||
"character": {"step": 0.024, "max_offset": 0.20, "min_offset": -0.32, "bad_max_offset": 0.42, "bad_min_offset": -1.30, "culprit_bias": 0.10},
|
||||
}
|
||||
|
||||
|
||||
@@ -31,6 +31,9 @@ def _clamp(value, low, high):
|
||||
return max(low, min(high, value))
|
||||
|
||||
|
||||
PROTECTED_PHRASE_RE = re.compile(r'"([^"]+)"|“([^”]+)”|\\([^\\]+)\\')
|
||||
|
||||
|
||||
def tensor_to_serializable(t: torch.Tensor) -> dict:
|
||||
if not isinstance(t, torch.Tensor):
|
||||
raise TypeError(f"Expected torch.Tensor, got {type(t)}")
|
||||
@@ -151,7 +154,88 @@ def render_refinement_loss_graph(refinement_key, scheduler_mode, mode, total_ite
|
||||
|
||||
return _to_image_tensor(image.convert("RGB"))
|
||||
|
||||
class FunPackGemmaEmbeddingRefiner:
|
||||
|
||||
def refinement_state_path(refinement_key, mode, prefix="refine", extension="json"):
|
||||
base_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
refinements_dir = os.path.join(base_dir, "refinements")
|
||||
safe_key = md5(f"{(mode or 'ltx2').lower()}::{refinement_key}".encode("utf-8")).hexdigest()
|
||||
return os.path.join(refinements_dir, f"{prefix}_{safe_key}.{extension}")
|
||||
|
||||
|
||||
def clone_latent(latent):
|
||||
if not isinstance(latent, dict):
|
||||
return None
|
||||
|
||||
cloned = {}
|
||||
for key, value in latent.items():
|
||||
cloned[key] = value.detach().clone() if isinstance(value, torch.Tensor) else value
|
||||
return cloned
|
||||
|
||||
|
||||
def latent_samples(latent):
|
||||
if not isinstance(latent, dict):
|
||||
return None
|
||||
samples = latent.get("samples")
|
||||
return samples if isinstance(samples, torch.Tensor) else None
|
||||
|
||||
|
||||
def latent_sample_type_name(latent):
|
||||
if not isinstance(latent, dict):
|
||||
return "missing"
|
||||
samples = latent.get("samples")
|
||||
if samples is None:
|
||||
return "missing samples"
|
||||
shape = tuple(samples.shape) if isinstance(samples, torch.Tensor) else "unknown"
|
||||
return f"{type(samples).__module__}.{type(samples).__name__}, shape={shape}"
|
||||
|
||||
|
||||
def latent_is_plain_video_tensor(latent):
|
||||
samples = latent_samples(latent)
|
||||
if samples is None:
|
||||
return False
|
||||
if samples.dim() == 5:
|
||||
return True
|
||||
if samples.dim() == 4 and latent.get("type") != "audio":
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def cpu_tensor_bundle(latent):
|
||||
if not isinstance(latent, dict):
|
||||
return {}
|
||||
|
||||
bundle = {}
|
||||
for key, value in latent.items():
|
||||
if isinstance(value, torch.Tensor):
|
||||
bundle[key] = value.detach().cpu().clone()
|
||||
return bundle
|
||||
|
||||
|
||||
def latent_from_tensor_bundle(bundle):
|
||||
if not isinstance(bundle, dict):
|
||||
return None
|
||||
|
||||
latent = {}
|
||||
for key, value in bundle.items():
|
||||
if key == "_meta":
|
||||
continue
|
||||
latent[key] = value.detach().clone() if isinstance(value, torch.Tensor) else value
|
||||
return latent if latent_samples(latent) is not None else None
|
||||
|
||||
|
||||
class FunPackVideoRefiner:
|
||||
LATENT_OUTPUT_INDEX = 6
|
||||
NO_LATENT_REFERENCE_ERROR = "No available latent to operate. Please connect reference latent to input of Video Refiner."
|
||||
WRONG_LATENT_ERROR = (
|
||||
"Video Refiner latent input must be a plain video LATENT with tensor samples. "
|
||||
"Audio latents and LTX audio/video combined NestedTensor latents are not supported here. "
|
||||
"Connect only the video latent to Video Refiner, then feed the refined video latent into "
|
||||
"LTXVConcatAVLatent as video_latent."
|
||||
)
|
||||
SAVED_LATENT_ONLY_STATUS = (
|
||||
"Running refinement on saved latent. Changing reference latent shape and size will cause no effect to generation."
|
||||
)
|
||||
|
||||
_tokenizers = {}
|
||||
_tokenizer_sources = {
|
||||
"ltx2": [
|
||||
@@ -224,7 +308,7 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
cls._tokenizers[mode] = AutoTokenizer.from_pretrained(model_id, **kwargs)
|
||||
return cls._tokenizers[mode]
|
||||
except Exception as e:
|
||||
print(f"[FunPackGemmaEmbeddingRefiner] Tokenizer load failed for mode '{mode}' from '{model_id}': {e}")
|
||||
print(f"[FunPackVideoRefiner] Tokenizer load failed for mode '{mode}' from '{model_id}': {e}")
|
||||
|
||||
return None
|
||||
|
||||
@@ -308,6 +392,28 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
def _iter_prompt_segments(self, prompt: str):
|
||||
if not prompt:
|
||||
return
|
||||
|
||||
last_end = 0
|
||||
for match in PROTECTED_PHRASE_RE.finditer(prompt):
|
||||
start, end = match.span()
|
||||
if start > last_end:
|
||||
yield ("text", prompt[last_end:start])
|
||||
protected_text = (match.group(1) or match.group(2) or match.group(3) or "").strip()
|
||||
if protected_text:
|
||||
yield ("protected", protected_text)
|
||||
last_end = end
|
||||
|
||||
if last_end < len(prompt):
|
||||
yield ("text", prompt[last_end:])
|
||||
|
||||
def _mask_quoted_text(self, prompt: str):
|
||||
if not prompt:
|
||||
return ""
|
||||
return PROTECTED_PHRASE_RE.sub(lambda match: " " * len(match.group(0)), prompt)
|
||||
|
||||
def _build_word_groups(self, prompt: str, tokenizer, seq_len: int, token_mask=None):
|
||||
if not tokenizer or not prompt or seq_len <= 0:
|
||||
return []
|
||||
@@ -326,7 +432,33 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
|
||||
word_groups = []
|
||||
grouped_seen = set()
|
||||
raw_words = [w.strip() for w in prompt.split() if w.strip()]
|
||||
for segment_type, segment_text in self._iter_prompt_segments(prompt):
|
||||
if segment_type != "protected":
|
||||
continue
|
||||
|
||||
clean_phrase = segment_text.strip()
|
||||
lower = clean_phrase.lower()
|
||||
if lower in grouped_seen or len(clean_phrase) < 3 or not any(c.isalpha() for c in clean_phrase):
|
||||
continue
|
||||
|
||||
grouped_seen.add(lower)
|
||||
phrase_token_list = self._tokenize_ids(
|
||||
tokenizer,
|
||||
clean_phrase,
|
||||
add_special_tokens=False
|
||||
)
|
||||
if not phrase_token_list:
|
||||
continue
|
||||
|
||||
for start in range(max(0, len(full_token_ids) - len(phrase_token_list) + 1)):
|
||||
end = start + len(phrase_token_list)
|
||||
if token_mask_list is not None and not all(token_mask_list[start:min(effective_seq_len, end)]):
|
||||
continue
|
||||
if full_token_ids[start:end] == phrase_token_list and start < effective_seq_len:
|
||||
word_groups.append((start, min(effective_seq_len, end), clean_phrase, phrase_token_list))
|
||||
break
|
||||
|
||||
raw_words = [w.strip() for w in self._mask_quoted_text(prompt).split() if w.strip()]
|
||||
for word in raw_words:
|
||||
clean_word = word
|
||||
lower = clean_word.lower()
|
||||
@@ -355,7 +487,10 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
if not found:
|
||||
continue
|
||||
|
||||
return [group for group in word_groups if group[1] > group[0]]
|
||||
return sorted(
|
||||
[group for group in word_groups if group[1] > group[0]],
|
||||
key=lambda group: (group[0], group[1], group[2].lower()),
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(s):
|
||||
@@ -403,11 +538,18 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
"lora_stack": ("FUNPACK_LORA_STACK", {
|
||||
"tooltip": "Optional stack from FunPack LoRA Loader. The refiner uses it to save prompt-specific suggested LoRA weights."
|
||||
}),
|
||||
"latent": ("LATENT", {
|
||||
"tooltip": "Optional latent to refine. If no saved latent exists for this key, it passes through unchanged."
|
||||
}),
|
||||
},
|
||||
"hidden": {
|
||||
"prompt": "PROMPT",
|
||||
"unique_id": "UNIQUE_ID",
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("CONDITIONING", "STRING", "STRING", "STRING", "IMAGE", "SIGMAS")
|
||||
RETURN_NAMES = ("modified_positive", "status", "feedback_question", "training_info", "loss_graph", "refined_sigmas")
|
||||
RETURN_TYPES = ("CONDITIONING", "STRING", "STRING", "STRING", "IMAGE", "SIGMAS", "LATENT")
|
||||
RETURN_NAMES = ("modified_positive", "status", "feedback_question", "training_info", "loss_graph", "refined_sigmas", "refined_latent")
|
||||
FUNCTION = "refine"
|
||||
CATEGORY = "FunPack/Refinement"
|
||||
|
||||
@@ -718,23 +860,42 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
"""
|
||||
if not prompt:
|
||||
return []
|
||||
phrases = [p.strip() for p in re.split(r'[,;]', prompt) if p.strip()]
|
||||
result = []
|
||||
for phrase in phrases:
|
||||
words = [w.strip().lower() for w in phrase.split() if self._is_valuable_token(w.strip())]
|
||||
if words:
|
||||
result.append(words)
|
||||
for segment_type, segment_text in self._iter_prompt_segments(prompt):
|
||||
if segment_type == "protected":
|
||||
protected_phrase = segment_text.strip().lower()
|
||||
if len(protected_phrase) >= 3 and any(c.isalpha() for c in protected_phrase):
|
||||
result.append([protected_phrase])
|
||||
continue
|
||||
|
||||
phrases = [p.strip() for p in re.split(r'[,;]', segment_text) if p.strip()]
|
||||
for phrase in phrases:
|
||||
words = [w.strip().lower() for w in phrase.split() if self._is_valuable_token(w.strip())]
|
||||
if words:
|
||||
result.append(words)
|
||||
return result
|
||||
|
||||
def _build_prompt_fallback_concept(self, prompt: str, concept_clusters: dict):
|
||||
if not prompt:
|
||||
return None
|
||||
|
||||
fallback_words = [
|
||||
w.strip().lower()
|
||||
for w in re.split(r'[\s,;]+', prompt)
|
||||
if self._is_valuable_token(w.strip())
|
||||
][:8]
|
||||
fallback_words = []
|
||||
for segment_type, segment_text in self._iter_prompt_segments(prompt):
|
||||
if segment_type == "protected":
|
||||
protected_phrase = segment_text.strip().lower()
|
||||
if len(protected_phrase) >= 3 and any(c.isalpha() for c in protected_phrase):
|
||||
fallback_words.append(protected_phrase)
|
||||
continue
|
||||
|
||||
fallback_words.extend(
|
||||
w.strip().lower()
|
||||
for w in re.split(r'[\s,;]+', segment_text)
|
||||
if self._is_valuable_token(w.strip())
|
||||
)
|
||||
if len(fallback_words) >= 8:
|
||||
break
|
||||
|
||||
fallback_words = fallback_words[:8]
|
||||
if not fallback_words:
|
||||
return None
|
||||
|
||||
@@ -1233,6 +1394,221 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
)
|
||||
return tuned_sigmas, sigma_status
|
||||
|
||||
# =========================================================================
|
||||
# LATENT REFINEMENT
|
||||
# =========================================================================
|
||||
|
||||
def _resize_tensor_like(self, tensor: torch.Tensor, reference: torch.Tensor):
|
||||
if not isinstance(tensor, torch.Tensor) or not isinstance(reference, torch.Tensor):
|
||||
return None
|
||||
|
||||
out = tensor.to(device=reference.device, dtype=reference.dtype)
|
||||
if list(out.shape) == list(reference.shape):
|
||||
return out
|
||||
if out.dim() != reference.dim() or out.dim() not in {4, 5}:
|
||||
return None
|
||||
|
||||
batch = reference.shape[0]
|
||||
if out.shape[0] < batch:
|
||||
reps = [1] * out.dim()
|
||||
reps[0] = math.ceil(batch / max(1, out.shape[0]))
|
||||
out = out.repeat(*reps)
|
||||
out = out[:batch]
|
||||
|
||||
channels = reference.shape[1]
|
||||
if out.shape[1] < channels:
|
||||
pad_shape = list(out.shape)
|
||||
pad_shape[1] = channels - out.shape[1]
|
||||
out = torch.cat([out, torch.zeros(pad_shape, device=out.device, dtype=out.dtype)], dim=1)
|
||||
out = out[:, :channels]
|
||||
|
||||
if out.dim() == 4:
|
||||
return F.interpolate(out, size=reference.shape[-2:], mode="bilinear", align_corners=False)
|
||||
return F.interpolate(out, size=reference.shape[-3:], mode="trilinear", align_corners=False)
|
||||
|
||||
def _load_saved_latent(self, refinement_key, mode):
|
||||
path = refinement_state_path(refinement_key, mode, prefix="latent", extension="pt")
|
||||
if not os.path.exists(path):
|
||||
return None, "Latent: no saved latent for this key."
|
||||
|
||||
try:
|
||||
data = torch.load(path, map_location="cpu", weights_only=False)
|
||||
except Exception as e:
|
||||
return None, f"Latent: failed to load saved latent ({e})."
|
||||
|
||||
if not isinstance(data, dict) or not isinstance(data.get("samples"), torch.Tensor):
|
||||
return None, "Latent: saved latent file is invalid."
|
||||
|
||||
return data, "Latent: saved latent loaded."
|
||||
|
||||
def _value_links_to_output(self, value, source_id, output_index):
|
||||
if isinstance(value, (list, tuple)):
|
||||
if len(value) >= 2 and str(value[0]) == source_id:
|
||||
try:
|
||||
if int(value[1]) == output_index:
|
||||
return True
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
return any(self._value_links_to_output(item, source_id, output_index) for item in value)
|
||||
|
||||
if isinstance(value, dict):
|
||||
return any(self._value_links_to_output(item, source_id, output_index) for item in value.values())
|
||||
|
||||
return False
|
||||
|
||||
def _is_output_connected(self, prompt, unique_id, output_index):
|
||||
if prompt is None or unique_id is None:
|
||||
return False
|
||||
|
||||
source_id = str(unique_id)
|
||||
prompt_nodes = prompt.get("output", prompt) if isinstance(prompt, dict) else {}
|
||||
if not isinstance(prompt_nodes, dict):
|
||||
return False
|
||||
|
||||
for node in prompt_nodes.values():
|
||||
if not isinstance(node, dict):
|
||||
continue
|
||||
inputs = node.get("inputs", {})
|
||||
if isinstance(inputs, dict) and self._value_links_to_output(inputs, source_id, output_index):
|
||||
return True
|
||||
|
||||
workflow = prompt.get("workflow") if isinstance(prompt, dict) else None
|
||||
links = workflow.get("links", []) if isinstance(workflow, dict) else []
|
||||
for link in links:
|
||||
if isinstance(link, (list, tuple)) and len(link) >= 3:
|
||||
try:
|
||||
if str(link[1]) == source_id and int(link[2]) == output_index:
|
||||
return True
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
elif isinstance(link, dict):
|
||||
try:
|
||||
if str(link.get("origin_id")) == source_id and int(link.get("origin_slot")) == output_index:
|
||||
return True
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
|
||||
return False
|
||||
|
||||
def _latent_refinement_disabled(self, latent):
|
||||
return clone_latent(latent), "Latent: disabled because latent refinement input/output path is not fully connected."
|
||||
|
||||
def _raise_wrong_latent(self, latent):
|
||||
sample_type = latent_sample_type_name(latent)
|
||||
latent_type = latent.get("type", "unspecified") if isinstance(latent, dict) else "missing"
|
||||
raise ValueError(f"{self.WRONG_LATENT_ERROR} Received samples={sample_type}, type={latent_type}.")
|
||||
|
||||
def _save_latent_reference(self, latent, refinement_key, mode):
|
||||
samples = latent_samples(latent)
|
||||
if samples is None:
|
||||
return False
|
||||
|
||||
path = refinement_state_path(refinement_key, mode, prefix="latent", extension="pt")
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
bundle = cpu_tensor_bundle(latent)
|
||||
bundle["_meta"] = {
|
||||
"refinement_key": refinement_key,
|
||||
"mode": (mode or "ltx2").lower(),
|
||||
"samples_shape": list(samples.shape),
|
||||
}
|
||||
torch.save(bundle, path)
|
||||
return True
|
||||
|
||||
def _delete_latent_reference(self, refinement_key, mode):
|
||||
path = refinement_state_path(refinement_key, mode, prefix="latent", extension="pt")
|
||||
try:
|
||||
os.remove(path)
|
||||
return True
|
||||
except FileNotFoundError:
|
||||
return False
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
def _refine_latent(self, latent, refinement_key, mode, rating, reward, global_adaptive):
|
||||
if isinstance(latent, dict) and not latent_is_plain_video_tensor(latent):
|
||||
self._raise_wrong_latent(latent)
|
||||
|
||||
current = latent_samples(latent)
|
||||
saved_bundle, load_status = self._load_saved_latent(refinement_key, mode)
|
||||
|
||||
if current is None:
|
||||
if saved_bundle is None:
|
||||
raise ValueError(self.NO_LATENT_REFERENCE_ERROR)
|
||||
|
||||
saved_latent = latent_from_tensor_bundle(saved_bundle)
|
||||
saved_samples = latent_samples(saved_latent)
|
||||
if saved_samples is None:
|
||||
raise ValueError(self.NO_LATENT_REFERENCE_ERROR)
|
||||
|
||||
latent_state = global_adaptive.setdefault("latent_refinement", {})
|
||||
latent_state["last_shape"] = list(saved_samples.shape)
|
||||
latent_state["saved_shape"] = list(saved_samples.shape)
|
||||
latent_state["status"] = self.SAVED_LATENT_ONLY_STATUS
|
||||
return saved_latent, latent_state["status"]
|
||||
|
||||
refined_latent = clone_latent(latent)
|
||||
if saved_bundle is None:
|
||||
self._save_latent_reference(latent, refinement_key, mode)
|
||||
global_adaptive.setdefault("latent_refinement", {})["last_shape"] = list(current.shape)
|
||||
return refined_latent, f"{load_status} Current latent saved as reference."
|
||||
|
||||
raw_saved_samples = saved_bundle.get("samples")
|
||||
if list(raw_saved_samples.shape) != list(current.shape):
|
||||
self._save_latent_reference(latent, refinement_key, mode)
|
||||
latent_state = global_adaptive.setdefault("latent_refinement", {})
|
||||
latent_state.pop("momentum", None)
|
||||
latent_state["last_shape"] = list(current.shape)
|
||||
latent_state["saved_shape"] = list(raw_saved_samples.shape)
|
||||
return refined_latent, "Latent: input shape changed. Reference rewritten, passthrough."
|
||||
|
||||
saved_samples = raw_saved_samples.to(device=current.device, dtype=current.dtype)
|
||||
latent_state = global_adaptive.setdefault("latent_refinement", {})
|
||||
latent_state["last_shape"] = list(current.shape)
|
||||
latent_state["saved_shape"] = list(raw_saved_samples.shape)
|
||||
|
||||
momentum_data = latent_state.get("momentum")
|
||||
if isinstance(momentum_data, dict):
|
||||
try:
|
||||
momentum = serializable_to_tensor(momentum_data).to(device=current.device, dtype=current.dtype)
|
||||
momentum = self._resize_tensor_like(momentum, current)
|
||||
except Exception:
|
||||
momentum = None
|
||||
else:
|
||||
momentum = None
|
||||
if momentum is None:
|
||||
momentum = torch.zeros_like(current)
|
||||
|
||||
nonzero_mask = current.ne(0) & saved_samples.ne(0)
|
||||
if not bool(nonzero_mask.any()):
|
||||
refined_latent["samples"] = current
|
||||
latent_state["status"] = "Latent: only zero-valued positions available, passthrough."
|
||||
return refined_latent, latent_state["status"]
|
||||
|
||||
lr = 0.035 if rating >= 7 else 0.055 if rating <= 4 else 0.018
|
||||
target_delta = (saved_samples - current) * nonzero_mask.to(dtype=current.dtype)
|
||||
update = target_delta * (reward * lr)
|
||||
momentum = 0.82 * momentum + 0.18 * update
|
||||
max_step = max(0.001, min(0.08, float(current.detach().abs().mean().item()) * 0.18))
|
||||
latent_delta = torch.clamp(momentum, min=-max_step, max=max_step)
|
||||
|
||||
refined_samples = current + latent_delta
|
||||
refined_samples = torch.where(nonzero_mask, refined_samples, current)
|
||||
refined_samples = torch.nan_to_num(refined_samples, nan=0.0)
|
||||
range_min = torch.minimum(current, saved_samples)
|
||||
range_max = torch.maximum(current, saved_samples)
|
||||
refined_samples = torch.maximum(torch.minimum(refined_samples, range_max), range_min)
|
||||
refined_latent["samples"] = refined_samples
|
||||
|
||||
latent_state["momentum"] = tensor_to_serializable(momentum.detach().cpu())
|
||||
latent_state["last_rating"] = int(rating)
|
||||
latent_state["last_reward"] = round(float(reward), 4)
|
||||
latent_state["nonzero_ratio"] = round(float(nonzero_mask.float().mean().item()), 6)
|
||||
latent_state["status"] = (
|
||||
f"Latent: adjusted shape {tuple(current.shape)} | nonzero {latent_state['nonzero_ratio']:.1%} | "
|
||||
f"max step {max_step:.5f} | bounded to input/reference range"
|
||||
)
|
||||
return refined_latent, latent_state["status"]
|
||||
|
||||
# =========================================================================
|
||||
# LORA WEIGHT SUGGESTIONS
|
||||
# =========================================================================
|
||||
@@ -1258,8 +1634,10 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
"offset_ratio": 0.0,
|
||||
"stable_offset_ratio": None,
|
||||
"reward_ema": 0.0,
|
||||
"culprit_score": 0.0,
|
||||
"good_streak": 0,
|
||||
"bad_streak": 0,
|
||||
"culprit_hits": 0,
|
||||
"iterations": 0,
|
||||
},
|
||||
)
|
||||
@@ -1268,8 +1646,10 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
state.setdefault("offset_ratio", 0.0)
|
||||
state.setdefault("stable_offset_ratio", None)
|
||||
state.setdefault("reward_ema", 0.0)
|
||||
state.setdefault("culprit_score", 0.0)
|
||||
state.setdefault("good_streak", 0)
|
||||
state.setdefault("bad_streak", 0)
|
||||
state.setdefault("culprit_hits", 0)
|
||||
state.setdefault("iterations", 0)
|
||||
return lora_id, state
|
||||
|
||||
@@ -1342,47 +1722,91 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
|
||||
offset = float(state.get("offset_ratio", 0.0))
|
||||
stable_offset = state.get("stable_offset_ratio")
|
||||
culprit_score = float(state.get("culprit_score", 0.0))
|
||||
culprit_hits = int(state.get("culprit_hits", 0))
|
||||
if stable_offset is not None and rating >= 6:
|
||||
offset = 0.72 * offset + 0.28 * float(stable_offset)
|
||||
|
||||
step = profile["step"] * max(0.15, relation)
|
||||
max_offset = profile["max_offset"]
|
||||
if relation <= 0.05:
|
||||
offset *= 0.94
|
||||
elif rating >= 8:
|
||||
min_offset = profile["min_offset"]
|
||||
effective_relation = max(relation, profile.get("culprit_bias", 0.0))
|
||||
base_model = float(entry.get("base_model_weight", entry.get("model_weight", 1.0)))
|
||||
base_abs = abs(base_model)
|
||||
is_concept_lora = lora_type == "concept"
|
||||
concept_match_strength = 1.0
|
||||
if is_concept_lora:
|
||||
concept_match_strength += 0.30 if matched_labels else 0.0
|
||||
concept_match_strength += max(0.0, relation - 0.30) * 1.35
|
||||
|
||||
if rating >= 8:
|
||||
value_mult = 0.75 + min(1.4, max(0.5, concept_importance)) * 0.25
|
||||
offset += step * (0.45 + max(0.0, reward)) * value_mult
|
||||
culprit_score *= 0.72
|
||||
if state["good_streak"] >= 3:
|
||||
if stable_offset is None:
|
||||
stable_offset = offset
|
||||
else:
|
||||
stable_offset = 0.78 * float(stable_offset) + 0.22 * offset
|
||||
state["stable_offset_ratio"] = _clamp(stable_offset, -max_offset, max_offset)
|
||||
state["stable_offset_ratio"] = _clamp(stable_offset, min_offset, max_offset)
|
||||
offset = state["stable_offset_ratio"]
|
||||
elif rating <= 4:
|
||||
severity = _clamp((5.0 - float(rating)) / 4.0, 0.0, 1.0)
|
||||
culprit_signal = max(0.20, effective_relation) * (0.70 + base_abs * 0.30)
|
||||
if is_concept_lora:
|
||||
culprit_signal *= concept_match_strength
|
||||
culprit_score = _clamp(culprit_score * 0.72 + severity * culprit_signal, 0.0, 2.5)
|
||||
culprit_hits = culprit_hits + 1 if culprit_score >= 0.45 else max(0, culprit_hits - 1)
|
||||
max_offset = profile["bad_max_offset"] if rating <= 2 else max(max_offset, profile["bad_max_offset"] * 0.72)
|
||||
offset -= step * (1.0 + severity * 3.0) * max(0.6, relation)
|
||||
bad_floor_strength = min(1.0, 0.45 + 0.35 * culprit_score + 0.14 * state["bad_streak"])
|
||||
if is_concept_lora:
|
||||
bad_floor_strength = min(1.0, bad_floor_strength + 0.18 * concept_match_strength)
|
||||
min_offset = min(min_offset, profile["bad_min_offset"] * bad_floor_strength)
|
||||
offset -= step * (1.0 + severity * 3.0) * max(0.4, effective_relation) * max(0.8, 0.85 + culprit_score)
|
||||
if state["bad_streak"] >= 2:
|
||||
offset -= step * (0.65 + severity * 1.8) * max(0.25, effective_relation)
|
||||
if is_concept_lora and state["bad_streak"] >= 2:
|
||||
offset -= step * (0.85 + severity * 2.4) * max(0.50, effective_relation) * concept_match_strength
|
||||
if state["bad_streak"] >= 2:
|
||||
state["stable_offset_ratio"] = None
|
||||
if is_concept_lora and matched_labels and state["bad_streak"] >= 2 and culprit_score >= 0.70:
|
||||
offset = min(offset, -1.0)
|
||||
if state["bad_streak"] >= 3 and culprit_score >= 0.85 and abs(1.0 + offset) < 0.08:
|
||||
offset = -1.0
|
||||
else:
|
||||
if stable_offset is not None:
|
||||
culprit_score *= 0.92
|
||||
culprit_hits = max(0, culprit_hits - 1)
|
||||
if relation <= 0.05:
|
||||
offset *= 0.94
|
||||
elif stable_offset is not None:
|
||||
offset = 0.65 * offset + 0.35 * float(stable_offset)
|
||||
else:
|
||||
offset *= 0.90
|
||||
|
||||
offset = _clamp(offset, -max_offset, max_offset)
|
||||
offset = _clamp(offset, min_offset, max_offset)
|
||||
state["offset_ratio"] = offset
|
||||
state["culprit_score"] = culprit_score
|
||||
state["culprit_hits"] = culprit_hits
|
||||
state["iterations"] = int(state.get("iterations", 0)) + 1
|
||||
|
||||
base_model = float(entry.get("base_model_weight", entry.get("model_weight", 1.0)))
|
||||
model_weight = base_model * (1.0 + offset)
|
||||
suspect = culprit_score >= 0.65 or model_weight <= 0.0 or culprit_hits >= 2
|
||||
action = (
|
||||
"invert" if model_weight < 0.0 else
|
||||
"mute" if model_weight == 0.0 else
|
||||
"reduce" if abs(model_weight) < abs(base_model) else
|
||||
"boost"
|
||||
)
|
||||
suggestions[lora_id] = {
|
||||
"name": entry.get("name", ""),
|
||||
"type": lora_type,
|
||||
"model_weight": model_weight,
|
||||
"base_model_weight": base_model,
|
||||
"offset_ratio": offset,
|
||||
"culprit_score": culprit_score,
|
||||
"culprit_hits": culprit_hits,
|
||||
"suspect": suspect,
|
||||
"action": action,
|
||||
"relation": relation,
|
||||
"matched_concepts": matched_labels,
|
||||
"rating": int(rating),
|
||||
@@ -1395,6 +1819,7 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
status_parts.append(
|
||||
f"{entry.get('name', '?')}[{lora_type}] rel={relation:.2f} "
|
||||
f"offset={offset:+.3f} next={model_weight:+.3f} "
|
||||
f"sus={culprit_score:.2f}{' !' if suspect else ''} "
|
||||
f"match={match_text}"
|
||||
)
|
||||
|
||||
@@ -1412,7 +1837,8 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
scheduler_mode: str = "original", positive_prompt: str = "",
|
||||
reset_session: bool = False, unlimited_history: bool = False,
|
||||
seed: int = 0, feedback_enabled: bool = False, feedback_rating: int = 3,
|
||||
sigmas=None, sigma_strength: str = "subtle", lora_stack=None):
|
||||
sigmas=None, sigma_strength: str = "subtle", lora_stack=None, latent=None,
|
||||
prompt=None, unique_id=None):
|
||||
|
||||
mode = (mode or "ltx2").lower()
|
||||
if mode not in self._tokenizer_sources:
|
||||
@@ -1422,12 +1848,10 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
torch.manual_seed(seed)
|
||||
random.seed(seed)
|
||||
|
||||
base_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
refinements_dir = os.path.join(base_dir, "refinements")
|
||||
refinements_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "refinements")
|
||||
os.makedirs(refinements_dir, exist_ok=True)
|
||||
|
||||
safe_key = md5(f"{mode}::{refinement_key}".encode("utf-8")).hexdigest()
|
||||
json_file = os.path.join(refinements_dir, f"refine_{safe_key}.json")
|
||||
json_file = refinement_state_path(refinement_key, mode)
|
||||
fallback_loss_graph = render_refinement_loss_graph(
|
||||
refinement_key=refinement_key,
|
||||
scheduler_mode=scheduler_mode,
|
||||
@@ -1437,9 +1861,12 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
points=[],
|
||||
)
|
||||
fallback_sigmas = sigmas.detach().clone() if isinstance(sigmas, torch.Tensor) else torch.FloatTensor([])
|
||||
latent_output_connected = self._is_output_connected(prompt, unique_id, self.LATENT_OUTPUT_INDEX)
|
||||
fallback_latent = clone_latent(latent)
|
||||
fallback_latent_status = "Latent: not evaluated before conditioning validation."
|
||||
|
||||
if not positive_conditioning or not isinstance(positive_conditioning, list) or len(positive_conditioning) == 0:
|
||||
return (positive_conditioning, "ERROR: Empty positive CONDITIONING input", "", "ERROR: No positive conditioning", fallback_loss_graph, fallback_sigmas)
|
||||
return (positive_conditioning, "ERROR: Empty positive CONDITIONING input", "", "ERROR: No positive conditioning", fallback_loss_graph, fallback_sigmas, fallback_latent)
|
||||
|
||||
item = positive_conditioning[0]
|
||||
if isinstance(item, (list, tuple)) and len(item) >= 2:
|
||||
@@ -1450,7 +1877,7 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
positive_meta = {"pooled_output": None}
|
||||
|
||||
if not isinstance(raw_positive, torch.Tensor):
|
||||
return (positive_conditioning, "ERROR: No positive embedding tensor found", "", "ERROR: Invalid embedding", fallback_loss_graph, fallback_sigmas)
|
||||
return (positive_conditioning, "ERROR: No positive embedding tensor found", "", "ERROR: Invalid embedding", fallback_loss_graph, fallback_sigmas, fallback_latent)
|
||||
|
||||
analysis_prompt = self._normalize_prompt_for_mode(positive_prompt, mode)
|
||||
prompt_key = analysis_prompt if mode == "wan" else positive_prompt
|
||||
@@ -1503,25 +1930,49 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
|
||||
# ====================== RESET / NEW SESSION ======================
|
||||
if reset_session or not os.path.exists(json_file):
|
||||
if reset_session:
|
||||
self._delete_latent_reference(refinement_key, mode)
|
||||
if latent_output_connected:
|
||||
fallback_latent, fallback_latent_status = self._refine_latent(
|
||||
latent,
|
||||
refinement_key,
|
||||
mode,
|
||||
rating,
|
||||
(rating - 5.5) / 4.5,
|
||||
{},
|
||||
)
|
||||
else:
|
||||
fallback_latent, fallback_latent_status = self._latent_refinement_disabled(latent)
|
||||
data = _fresh_data()
|
||||
with open(json_file, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2)
|
||||
return (positive_conditioning, "✓ New session started – Reference saved", "", "✓ New session started. Reference embedding saved.", fallback_loss_graph, fallback_sigmas)
|
||||
return (positive_conditioning, "New session started - Reference saved", "", f"New session started. Reference embedding saved.\n{fallback_latent_status}", fallback_loss_graph, fallback_sigmas, fallback_latent)
|
||||
|
||||
# ====================== SAFE JSON LOAD ======================
|
||||
try:
|
||||
with open(json_file, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
except (json.JSONDecodeError, OSError, ValueError) as e:
|
||||
print(f"[FunPackGemmaEmbeddingRefiner] Corrupt session file, resetting: {e}")
|
||||
print(f"[FunPackVideoRefiner] Corrupt session file, resetting: {e}")
|
||||
try:
|
||||
os.remove(json_file)
|
||||
except OSError:
|
||||
pass
|
||||
if latent_output_connected:
|
||||
fallback_latent, fallback_latent_status = self._refine_latent(
|
||||
latent,
|
||||
refinement_key,
|
||||
mode,
|
||||
rating,
|
||||
(rating - 5.5) / 4.5,
|
||||
{},
|
||||
)
|
||||
else:
|
||||
fallback_latent, fallback_latent_status = self._latent_refinement_disabled(latent)
|
||||
data = _fresh_data()
|
||||
with open(json_file, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2)
|
||||
return (positive_conditioning, "⚠️ Session file was corrupt – Reset and started fresh", "", "⚠️ Session reset due to corrupt file", fallback_loss_graph, fallback_sigmas)
|
||||
return (positive_conditioning, "Session file was corrupt - Reset and started fresh", "", f"Session reset due to corrupt file\n{fallback_latent_status}", fallback_loss_graph, fallback_sigmas, fallback_latent)
|
||||
|
||||
global_adaptive = data["global_adaptive"]
|
||||
# Migrate sessions created before the multi-level concept system was added
|
||||
@@ -1584,7 +2035,7 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
try:
|
||||
old_reference = serializable_to_tensor(active["reference_embeds"])
|
||||
except Exception as e:
|
||||
print(f"[FunPackGemmaEmbeddingRefiner] Failed to load reference embedding: {e}. Resetting for this prompt.")
|
||||
print(f"[FunPackVideoRefiner] Failed to load reference embedding: {e}. Resetting for this prompt.")
|
||||
old_reference = raw_positive.clone()
|
||||
active["reference_embeds"] = tensor_to_serializable(old_reference)
|
||||
active["history"] = []
|
||||
@@ -1595,7 +2046,7 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
|
||||
# Shape mismatch guard
|
||||
if old_reference.shape != cur_positive.shape:
|
||||
print(f"[FunPackGemmaEmbeddingRefiner] Reference shape {old_reference.shape} != current {cur_positive.shape}. Resetting reference.")
|
||||
print(f"[FunPackVideoRefiner] Reference shape {old_reference.shape} != current {cur_positive.shape}. Resetting reference.")
|
||||
old_reference = cur_positive.clone()
|
||||
active["reference_embeds"] = tensor_to_serializable(old_reference)
|
||||
active["history"] = []
|
||||
@@ -1921,6 +2372,17 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
sigma_strength,
|
||||
seed,
|
||||
)
|
||||
if latent_output_connected:
|
||||
refined_latent, latent_status = self._refine_latent(
|
||||
latent,
|
||||
refinement_key,
|
||||
mode,
|
||||
rating,
|
||||
reward,
|
||||
global_adaptive,
|
||||
)
|
||||
else:
|
||||
refined_latent, latent_status = self._latent_refinement_disabled(latent)
|
||||
|
||||
# ====================== INTELLIGENT CONCEPT FEEDBACK SELECTION ======================
|
||||
# Chooses one concept/question pair per run using category-aware scoring.
|
||||
@@ -2043,7 +2505,8 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
f"Concept groups ({len(concept_groups)} total): {group_line}\n"
|
||||
f"Global top words: {current_top}\n"
|
||||
f"{lora_suggestion_status}\n"
|
||||
f"{sigma_status}"
|
||||
f"{sigma_status}\n"
|
||||
f"{latent_status}"
|
||||
)
|
||||
|
||||
if scheduler_mode == "accurate":
|
||||
@@ -2080,15 +2543,40 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
"⚠️ Still exploring" if avg_reward_ema > -0.2 else
|
||||
"🔄 Heavy correction"
|
||||
)
|
||||
pending_feedback = data.get("pending_feedback")
|
||||
if not feedback_enabled:
|
||||
feedback_state = "Feedback off"
|
||||
elif pending_feedback is not None:
|
||||
feedback_state = f"Feedback queued for '{pending_feedback.get('concept_label', 'concept')}'"
|
||||
else:
|
||||
feedback_state = "Feedback ready"
|
||||
|
||||
suggestions_snapshot = active.get("lora_weight_suggestions", {})
|
||||
suspect_count = sum(1 for item in suggestions_snapshot.values() if item.get("suspect"))
|
||||
if isinstance(lora_stack, dict) and lora_stack.get("loras"):
|
||||
lora_state = f"LoRA active ({len(lora_stack.get('loras', []))} loaded"
|
||||
if suspect_count > 0:
|
||||
lora_state += f", {suspect_count} suspect"
|
||||
lora_state += ")"
|
||||
else:
|
||||
lora_state = "LoRA idle"
|
||||
|
||||
sigma_state = "Sigma active" if isinstance(sigmas, torch.Tensor) and sigma_strength != "off" else (
|
||||
"Sigma connected (off)" if isinstance(sigmas, torch.Tensor) else "Sigma idle"
|
||||
)
|
||||
|
||||
latent_state = (
|
||||
"Latent active" if latent_output_connected and refined_latent is not None else
|
||||
"Latent armed" if latent_output_connected else
|
||||
"Latent idle"
|
||||
)
|
||||
|
||||
status = (
|
||||
f"Mode {mode.upper()} | Iter {iter_num} | Total iters {global_total_iterations} | "
|
||||
f"History {len(history)} | Unique prompts {len(prompt_histories)}\n"
|
||||
f"Rating {rating}/10 {trend} | Sim {similarity:.3f} | Reward {reward:+.2f} | "
|
||||
f"EMA {avg_reward_ema:+.2f} | Good ratio {good_ratio:.0%} | Expl {expl:.3f} | {health}\n"
|
||||
f"Dominant: {dominant_line} | Groups: {group_line}\n"
|
||||
f"{lora_suggestion_status}\n"
|
||||
f"{sigma_status}"
|
||||
f"{health} | Mode {mode.upper()} | Rating {rating}/10 {trend} | Iter {iter_num}\n"
|
||||
f"Session: {len(prompt_histories)} prompt(s), {global_total_iterations} total update(s), "
|
||||
f"{len(history)} history item(s) on this prompt\n"
|
||||
f"Focus: {dominant_line} | {feedback_state}\n"
|
||||
f"Systems: {lora_state} | {sigma_state} | {latent_state}"
|
||||
)
|
||||
|
||||
# ====================== FINAL SAVE ======================
|
||||
@@ -2098,7 +2586,43 @@ class FunPackGemmaEmbeddingRefiner:
|
||||
# ====================== RETURN ======================
|
||||
modified_positive = [(new_positive, positive_meta)]
|
||||
|
||||
return (modified_positive, status, feedback_question_output, training_info, loss_graph, refined_sigmas)
|
||||
return (modified_positive, status, feedback_question_output, training_info, loss_graph, refined_sigmas, refined_latent)
|
||||
|
||||
|
||||
FunPackGemmaEmbeddingRefiner = FunPackVideoRefiner
|
||||
|
||||
|
||||
class FunPackSaveRefinementLatent:
|
||||
CATEGORY = "FunPack/Refinement"
|
||||
RETURN_TYPES = ("LATENT", "STRING")
|
||||
RETURN_NAMES = ("latent", "status")
|
||||
FUNCTION = "save_latent"
|
||||
DESCRIPTION = "Saves a latent tensor bundle under a refinement key for FunPack Video Refiner latent refinement."
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"latent": ("LATENT",),
|
||||
"refinement_key": ("STRING", {"default": "my_style_v1", "multiline": False}),
|
||||
"mode": (["ltx2", "wan"], {"default": "ltx2"}),
|
||||
}
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def IS_CHANGED(cls, **kwargs):
|
||||
return float("nan")
|
||||
|
||||
def save_latent(self, latent, refinement_key, mode):
|
||||
if isinstance(latent, dict) and not latent_is_plain_video_tensor(latent):
|
||||
FunPackVideoRefiner()._raise_wrong_latent(latent)
|
||||
|
||||
samples = latent_samples(latent)
|
||||
if samples is None:
|
||||
return (clone_latent(latent), "No latent samples tensor found.")
|
||||
|
||||
FunPackVideoRefiner()._save_latent_reference(latent, refinement_key, mode)
|
||||
return (clone_latent(latent), f"Saved latent for key '{refinement_key}' ({mode}) shape={tuple(samples.shape)}")
|
||||
|
||||
|
||||
class FunPackPromptCombiner:
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# FunPack Gemma Embedding Refiner
|
||||
# FunPack Video Refiner
|
||||
|
||||
This node analyzes the positive conditioning produced by your text encoder, tracks your feedback over time, and nudges the conditioning toward directions that should score higher in future generations.
|
||||
This node analyzes the positive conditioning produced by your text encoder, tracks your feedback over time, and nudges the conditioning toward directions that should score higher in future generations. It also has optional sigma and latent refinement paths for video workflows.
|
||||
|
||||
## Parameters
|
||||
|
||||
@@ -16,6 +16,8 @@ This node analyzes the positive conditioning produced by your text encoder, trac
|
||||
|
||||
**positive_prompt**: Optional prompt text used for token analysis and concept-level feedback.
|
||||
|
||||
If you want the model to preserve speech or spoken lines, wrap that text in `"` or `“ ”`. If you want a non-speech phrase to be treated as one whole phrase, wrap it in backslashes like `\a woman holds his drawings in her hands\`. Protected phrases are treated as one unit during prompt analysis, so dialogue like `"You should go for it, young man."` stays intact and a phrase like `\She calls him in.\` is kept whole instead of being reduced to a few "important" words. If speech or phrase-level text is not wrapped this way, common words inside it can still be treated as low-value prompt words and may be omitted or ignored.
|
||||
|
||||
**sigmas**: Optional input sigma schedule to co-refine alongside the conditioning. When connected, the node preserves the first and last sigma exactly and only adjusts the middle values.
|
||||
|
||||
**sigma_strength**: Preset strength for sigma refinement. `off` disables sigma changes, while `subtle`, `medium`, `strong`, and `max` allow progressively larger movement of the middle sigma values.
|
||||
@@ -30,24 +32,87 @@ This node analyzes the positive conditioning produced by your text encoder, trac
|
||||
|
||||
**feedback_rating**: Answer to the feedback question on its own 1 to 6 scale. This is only useful when concept feedback is enabled.
|
||||
|
||||
**lora_stack**: Optional stack from `FunPack LoRA Loader`. The refiner uses it to save next-run model LoRA weight suggestions.
|
||||
|
||||
**latent**: Optional latent input. It is only used when the `refined_latent` output is also connected. If no saved latent exists yet, the node uses the connected input latent as the current reference.
|
||||
|
||||
## Compatibility Name
|
||||
|
||||
The node is now displayed as `FunPack Video Refiner`. The old `FunPackGemmaEmbeddingRefiner` node key remains available as a compatibility alias for existing workflows and older documentation links.
|
||||
|
||||
## Outputs
|
||||
|
||||
**modified_positive**: Refined conditioning. Connect this output to the next conditioning stage in your workflow.
|
||||
|
||||
**status**: Human-readable training status, including iteration and refinement diagnostics.
|
||||
**status**: Short general node state summary. Use this to quickly see whether the refiner is learning well, what it is focused on, and which subsystems are active.
|
||||
|
||||
**feedback_question**: Optional concept feedback prompt generated by the node when feedback mode is active.
|
||||
|
||||
**training_info**: Additional training details for the current refinement step.
|
||||
**training_info**: Detailed training diagnostics for the current refinement step, including exact metrics, concept breakdown, LoRA suggestion details, and sigma/latent refinement details.
|
||||
|
||||
**loss_graph**: Graph image showing learning loss over total session iterations for the current refinement key.
|
||||
|
||||
**refined_sigmas**: Refined sigma schedule. If no `sigmas` input is connected, this output is empty.
|
||||
|
||||
**refined_latent**: Refined latent output. Connecting this output enables latent refinement logic; leaving it disconnected disables latent refinement even if the latent input is connected.
|
||||
|
||||
## Purpose
|
||||
|
||||
Use this node when you want to iteratively tune prompt conditioning based on your own ratings instead of manually rewriting prompts every time. Over multiple runs, it builds a refinement profile for the selected `refinement_key`.
|
||||
|
||||
## Latent Refinement
|
||||
|
||||
Use `FunPack Save Refinement Latent` to store a reference latent tensor bundle under a `refinement_key` and tokenizer `mode`. When `FunPack Video Refiner` later receives a latent input with the same key and mode, it loads that saved latent and applies a small rating-driven adjustment to the latent output.
|
||||
|
||||
For LTX audio/video workflows, the latent path must use video latents only:
|
||||
|
||||
- Split sampler output with `LTXVSeparateAVLatent`.
|
||||
- Send the video latent output to `FunPack Save Refinement Latent`.
|
||||
- Send the plain video latent to `FunPack Video Refiner` latent input.
|
||||
- Send `FunPack Video Refiner` `refined_latent` to `LTXVConcatAVLatent` `video_latent`.
|
||||
- Send the original audio latent to `LTXVConcatAVLatent` `audio_latent`.
|
||||
|
||||
Do not connect the combined AV latent from `LTXVConcatAVLatent`, the sampler AV output, or the audio latent to this node. Those use LTX nested/audio latent structures and are intentionally rejected with a clear error.
|
||||
|
||||
Some LTX separated video latents keep `type: audio` metadata even after `LTXVSeparateAVLatent`; the node accepts those when their `samples` tensor is the 5D video tensor shape.
|
||||
|
||||
Latent refinement follows the connection state exactly:
|
||||
|
||||
- If the latent input is disconnected and `refined_latent` is disconnected, latent tweaking is disabled.
|
||||
- If the latent input is disconnected, `refined_latent` is connected, and no saved latent exists, the node errors with: `No available latent to operate. Please connect reference latent to input of Video Refiner.`
|
||||
- If the latent input is disconnected, `refined_latent` is connected, and a saved latent exists, the node reports: `Running refinement on saved latent. Changing reference latent shape and size will cause no effect to generation.`
|
||||
- If the latent input is connected and `refined_latent` is disconnected, latent tweaking is disabled.
|
||||
- If both latent input and `refined_latent` are connected but no saved latent exists, the node passes the input latent through unchanged and saves it as the current reference.
|
||||
- If both are connected and the saved latent shape does not match the current input, the node rewrites the reference and passes the current input through unchanged.
|
||||
- If both are connected and the saved latent shape matches the current input, the node tweaks the latent using the connected input as the base and the saved latent as the learned reference. The refined tensor is bounded to the current/reference value range to avoid out-of-range latent values.
|
||||
|
||||
Resetting the session deletes the saved latent reference too.
|
||||
|
||||
Latent refinement is intentionally conservative. It tracks latent shape changes in the session state and leaves zero-valued latent positions unchanged because zero values can be intentional.
|
||||
|
||||
## LoRA Suggestions
|
||||
|
||||
Connect the `lora_stack` output from `FunPack LoRA Loader` when you want the refiner to write next-run `lora_weight_suggestions` into the prompt history. The suggestions are prompt-specific and are applied by `FunPack Apply LoRA Weights` on a later run.
|
||||
|
||||
LoRA suggestion boundaries:
|
||||
|
||||
- If `lora_stack` is not connected, no LoRA suggestions are updated.
|
||||
- If `positive_prompt` is empty, the refiner can still process conditioning, but exact prompt matching for LoRA suggestions is not useful.
|
||||
- Suggestions are scoped to the same `refinement_key`, `mode`, and normalized prompt text.
|
||||
- A LoRA can be reduced to `0.0` or moved negative if repeated bad ratings make it look harmful for that exact prompt.
|
||||
- Changing a LoRA's base weight later prevents stale suggestions for the old base from being applied.
|
||||
|
||||
## Prompt Protection
|
||||
|
||||
Quoted text and backslash-wrapped phrases protect prompt spans during word-group analysis. This is intended for dialogue, named phrases, and short actions that would lose meaning if split into separate "important" words.
|
||||
|
||||
Examples:
|
||||
|
||||
- `"You should go for it, young man."`
|
||||
- `\a woman holds his drawings in her hands\`
|
||||
|
||||
Unwrapped speech or phrase text can still work, but common words inside it may be ignored by the analysis because they look like low-value prompt words on their own.
|
||||
|
||||
## Capability Boundary
|
||||
|
||||
This node cannot make the underlying model generate concepts it does not already know how to represent. If a required subject, action, style, object, anatomy detail, or relationship is missing from the model's learned capabilities, repeating ratings will not teach it from scratch.
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
# FunPack LoRA Weight Workflow
|
||||
|
||||
This workflow lets `FunPack Gemma Embedding Refiner` suggest prompt-specific LoRA weights while keeping LoRA loading deterministic.
|
||||
This workflow lets `FunPack Video Refiner` suggest prompt-specific LoRA weights while keeping LoRA loading deterministic.
|
||||
|
||||
## Node Order
|
||||
|
||||
Use this pattern:
|
||||
|
||||
`FunPack Apply LoRA Weights` -> `FunPack LoRA Loader` -> `FunPack Gemma Embedding Refiner`
|
||||
`FunPack Apply LoRA Weights` -> `FunPack LoRA Loader` -> `FunPack Video Refiner`
|
||||
|
||||
The loader does not learn. The refiner owns prompt parsing, concept matching, rating interpretation, and saving suggested weights.
|
||||
|
||||
@@ -17,8 +17,9 @@ This node defines the selected LoRAs and their base weights.
|
||||
Inputs:
|
||||
|
||||
- **positive_prompt**: Exact prompt text for lookup.
|
||||
- **refinement_key**: Same key used by `FunPack Gemma Embedding Refiner`.
|
||||
- **refinement_key**: Same key used by `FunPack Video Refiner`.
|
||||
- **mode**: Same tokenizer mode as the refiner.
|
||||
- **per_block**: If enabled for `ltx2` stacks, the loader derives hidden transformer block scales from each LoRA's own patch magnitudes. The UI still exposes only the regular base weights.
|
||||
- **lora_N**: LoRA file.
|
||||
- **lora_N_type**: `general`, `concept`, `style`, `quality`, or `character`.
|
||||
- **lora_N_base_weight**: Trainer-recommended model base weight.
|
||||
@@ -27,18 +28,40 @@ Use **+ Add LoRA** to add more LoRA rows as needed.
|
||||
|
||||
On the first run for an exact prompt, no saved suggestion exists, so base weights are used. On later runs, the node reads the refiner JSON and applies saved suggestions for that exact prompt only.
|
||||
|
||||
Important boundaries:
|
||||
|
||||
- Suggestions are keyed by the exact prompt text after the selected mode's prompt normalization. Changing punctuation, wording, or line breaks can intentionally start a different prompt record.
|
||||
- Saved suggestions are ignored when the LoRA name, LoRA type, or saved base model weight no longer matches the current slot. This prevents stale learning from being applied after you change a LoRA setup.
|
||||
- `lora_N_type` is a hint for the refiner, not a loader category. A wrong type will not break loading, but it can make future suggestions less useful.
|
||||
- A saved or base model weight of `0.0` means the loader skips that LoRA for the current run.
|
||||
- Negative suggested weights are possible after repeated poor ratings when the refiner marks a LoRA as harmful for that prompt.
|
||||
- If `refinement_key` or `mode` differs from the refiner, the node correctly falls back to base weights because it is looking in a different state file.
|
||||
|
||||
## FunPack LoRA Loader
|
||||
|
||||
This node only loads the LoRA stack. It applies the model weights prepared by `FunPack Apply LoRA Weights`, then passes the same stack forward so the refiner can learn from it. The `clip` input is optional and uses zero CLIP strength when omitted.
|
||||
|
||||
## Gemma Refiner Integration
|
||||
When `per_block` is enabled on an `ltx2` stack, the loader keeps the user-facing global LoRA weight but redistributes it across detected `transformer_blocks.N` patches automatically. Blocks with stronger LoRA magnitude get a bit more weight, weaker blocks get a bit less, and non-block patches stay at the global weight. If the loader cannot detect a usable LTX transformer block layout, it falls back to the normal global application path.
|
||||
|
||||
Connect the `lora_stack` output from `FunPack LoRA Loader` into the optional `lora_stack` input on `FunPack Gemma Embedding Refiner`.
|
||||
Per-block notes:
|
||||
|
||||
- Per-block mode only applies to `ltx2` stacks on supported LTX image model configs.
|
||||
- Wan and other non-LTX workflows fall back to normal global LoRA loading even if `per_block` is enabled.
|
||||
- LoRAs without at least two detectable transformer blocks fall back to normal global loading.
|
||||
- The per-block scales are derived internally and are intentionally not exposed as separate UI controls.
|
||||
- The loader status reports whether each LoRA used `global`, `per-block`, or a fallback path.
|
||||
|
||||
## Video Refiner Integration
|
||||
|
||||
Connect the `lora_stack` output from `FunPack LoRA Loader` into the optional `lora_stack` input on `FunPack Video Refiner`.
|
||||
|
||||
The refiner compares each LoRA's declared type and filename with the concepts it already extracted from the prompt. After processing the rating, it saves `lora_weight_suggestions` into the same prompt entry in the refiner JSON.
|
||||
|
||||
Examples:
|
||||
|
||||
- A `quality` LoRA matching a valuable quality concept can be boosted after good ratings.
|
||||
- A `concept` LoRA matching a bad-rated prompt can be reduced if it may be over-weighting or distorting that concept.
|
||||
- A `concept` LoRA matching a bad-rated prompt is treated more aggressively than broad LoRAs, so it can be muted or inverted faster if it keeps distorting that concept.
|
||||
- Repeated bad ratings can mark a LoRA as a likely culprit, push its automatic weight down to `0.0`, and even invert it with a negative weight if it keeps ruining output.
|
||||
- Consistently good ratings can stabilize around a saved offset instead of continually drifting.
|
||||
|
||||
The refiner can only suggest weights for the stack it receives from `FunPack LoRA Loader`. If the `lora_stack` connection is omitted, existing conditioning, sigma, and latent refinement still work, but LoRA suggestions are not updated.
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
# FunPack Save Refinement Latent
|
||||
|
||||
This node saves a latent tensor bundle under a refinement key so `FunPack Video Refiner` can use it as an optional latent refinement reference on later runs.
|
||||
|
||||
## Parameters
|
||||
|
||||
**latent**: Plain video latent tensor bundle to save. In LTX audio/video workflows, connect the video output from `LTXVSeparateAVLatent`, not the audio latent or combined AV latent.
|
||||
|
||||
**refinement_key**: Name of the refinement session. Use the same key in `FunPack Video Refiner`.
|
||||
|
||||
**mode**: Tokenizer mode namespace. Use the same mode as `FunPack Video Refiner`.
|
||||
|
||||
## Outputs
|
||||
|
||||
**latent**: Passthrough latent.
|
||||
|
||||
**status**: Save status with the stored tensor shape.
|
||||
|
||||
## Notes
|
||||
|
||||
The saved latent lives in FunPack's local `refinements` folder as a PyTorch tensor file. The refiner only uses it when both the key and mode match and the `refined_latent` output is connected. If the refiner receives a latent and no saved reference exists, it saves the incoming latent automatically. Zero-valued latent positions are treated as intentional and are not refined.
|
||||
|
||||
Audio latents and LTX audio/video combined `NestedTensor` latents are intentionally rejected. Use only the video latent side, then reconnect the refined result to `LTXVConcatAVLatent` as `video_latent`. If a separated LTX video latent still reports `type: audio`, it is accepted when its `samples` tensor has the 5D video latent shape.
|
||||
|
||||
## Unintended Usage Boundaries
|
||||
|
||||
- Do not connect the sampler's combined AV latent directly to this node.
|
||||
- Do not connect the audio output from `LTXVSeparateAVLatent`.
|
||||
- Do not connect the output of `LTXVConcatAVLatent` back into this node.
|
||||
- Do not expect a saved latent to apply across different `refinement_key` or `mode` values.
|
||||
- Do not expect an old saved latent shape to resize a new generation. If the current latent shape changes, `FunPack Video Refiner` rewrites the reference and passes the current latent through unchanged.
|
||||
- Do not use this node as a general latent cache. It stores the reference used by the Video Refiner and may be replaced by reset/session behavior.
|
||||
@@ -0,0 +1,3 @@
|
||||
# FunPack Video Refiner
|
||||
|
||||
See [`FunPackGemmaEmbeddingRefiner.md`](FunPackGemmaEmbeddingRefiner.md) for the full documentation. The old filename is kept as a compatibility documentation path for users and workflows that still know the node by its former name.
|
||||
+11
-1
@@ -7,7 +7,9 @@ try:
|
||||
FunPackLorebookEnhancer,
|
||||
FunPackPromptCombiner,
|
||||
FunPackPromptEnhancer,
|
||||
FunPackSaveRefinementLatent,
|
||||
FunPackStoryWriter,
|
||||
FunPackVideoRefiner,
|
||||
)
|
||||
from .image_processing import (
|
||||
FunPackContinueVideo,
|
||||
@@ -23,7 +25,9 @@ except ImportError:
|
||||
FunPackLorebookEnhancer,
|
||||
FunPackPromptCombiner,
|
||||
FunPackPromptEnhancer,
|
||||
FunPackSaveRefinementLatent,
|
||||
FunPackStoryWriter,
|
||||
FunPackVideoRefiner,
|
||||
)
|
||||
from image_processing import (
|
||||
FunPackContinueVideo,
|
||||
@@ -44,6 +48,8 @@ except ImportError:
|
||||
"FunPackContinueVideo": FunPackContinueVideo,
|
||||
"FunPackLorebookEnhancer": FunPackLorebookEnhancer,
|
||||
"FunPackGemmaEmbeddingRefiner": FunPackGemmaEmbeddingRefiner,
|
||||
"FunPackVideoRefiner": FunPackVideoRefiner,
|
||||
"FunPackSaveRefinementLatent": FunPackSaveRefinementLatent,
|
||||
"FunPackHybridEuler2SSampler": FunPackHybridEuler2SSampler,
|
||||
"FunPackApplyLoraWeights": FunPackApplyLoraWeights,
|
||||
"FunPackLoraLoader": FunPackLoraLoader,
|
||||
@@ -58,7 +64,9 @@ except ImportError:
|
||||
"FunPackVideoStitch": "FunPack Video Stitch",
|
||||
"FunPackContinueVideo": "FunPack Continue Video",
|
||||
"FunPackLorebookEnhancer": "FunPack Lorebook Enhancer",
|
||||
"FunPackGemmaEmbeddingRefiner": "FunPack Gemma Embedding Refiner (Self-Refinement)",
|
||||
"FunPackGemmaEmbeddingRefiner": "FunPack Video Refiner (Compatibility)",
|
||||
"FunPackVideoRefiner": "FunPack Video Refiner",
|
||||
"FunPackSaveRefinementLatent": "FunPack Save Refinement Latent",
|
||||
"FunPackHybridEuler2SSampler": "FunPack Hybrid Euler 2S Sampler",
|
||||
"FunPackApplyLoraWeights": "FunPack Apply LoRA Weights",
|
||||
"FunPackLoraLoader": "FunPack LoRA Loader",
|
||||
@@ -68,6 +76,8 @@ __all__ = [
|
||||
"NODE_CLASS_MAPPINGS",
|
||||
"NODE_DISPLAY_NAME_MAPPINGS",
|
||||
"FunPackGemmaEmbeddingRefiner",
|
||||
"FunPackSaveRefinementLatent",
|
||||
"FunPackVideoRefiner",
|
||||
"FunPackHybridEuler2SSampler",
|
||||
"FunPackPromptCombiner",
|
||||
"FunPackLorebookEnhancer",
|
||||
|
||||
+178
-4
@@ -1,17 +1,24 @@
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
from collections import defaultdict
|
||||
from hashlib import md5
|
||||
|
||||
import comfy.lora
|
||||
import comfy.lora_convert
|
||||
import comfy.sd
|
||||
import comfy.utils
|
||||
import folder_paths
|
||||
import torch
|
||||
from aiohttp import web
|
||||
from server import PromptServer
|
||||
|
||||
|
||||
LORA_TYPES = ["general", "concept", "style", "quality", "character"]
|
||||
LORA_STACK_TYPE = "FUNPACK_LORA_STACK"
|
||||
TRANSFORMER_BLOCK_PATTERN = re.compile(r"(?:^|\.)transformer_blocks\.(\d+)\.")
|
||||
LTX_IMAGE_MODELS = {"ltxv", "ltxav"}
|
||||
|
||||
|
||||
@PromptServer.instance.routes.get("/funpack/loras")
|
||||
@@ -82,6 +89,46 @@ def refiner_state_path(refinement_key, mode):
|
||||
return os.path.join(refinements_dir, f"refine_{safe_key}.json")
|
||||
|
||||
|
||||
def coerce_bool(value):
|
||||
if isinstance(value, str):
|
||||
return value.strip().lower() in {"1", "true", "yes", "on"}
|
||||
return bool(value)
|
||||
|
||||
|
||||
def patch_target_key(patch_key):
|
||||
if isinstance(patch_key, tuple) and patch_key:
|
||||
return patch_key[0]
|
||||
return patch_key
|
||||
|
||||
|
||||
def transformer_block_index(patch_key):
|
||||
target_key = patch_target_key(patch_key)
|
||||
if not isinstance(target_key, str):
|
||||
return None
|
||||
|
||||
match = TRANSFORMER_BLOCK_PATTERN.search(target_key)
|
||||
if not match:
|
||||
return None
|
||||
return int(match.group(1))
|
||||
|
||||
|
||||
def patch_energy(value):
|
||||
if isinstance(value, torch.Tensor):
|
||||
return float(value.abs().mean().item())
|
||||
|
||||
weights = getattr(value, "weights", None)
|
||||
if weights is not None:
|
||||
return patch_energy(weights)
|
||||
|
||||
if isinstance(value, dict):
|
||||
return sum(patch_energy(item) for item in value.values())
|
||||
|
||||
if isinstance(value, (list, tuple)):
|
||||
return sum(patch_energy(item) for item in value)
|
||||
|
||||
return 0.0
|
||||
|
||||
|
||||
class FunPackApplyLoraWeights:
|
||||
"""
|
||||
Builds a LoRA stack from user base weights, then applies prompt-specific
|
||||
@@ -120,6 +167,13 @@ class FunPackApplyLoraWeights:
|
||||
"positive_prompt": ("STRING", {"multiline": True, "default": ""}),
|
||||
"refinement_key": ("STRING", {"default": "my_style_v1", "multiline": False}),
|
||||
"mode": (["ltx2", "wan"], {"default": "ltx2"}),
|
||||
"per_block": (
|
||||
"BOOLEAN",
|
||||
{
|
||||
"default": False,
|
||||
"tooltip": "For LTX-mode stacks, derive hidden transformer block strengths from the LoRA itself and apply them automatically.",
|
||||
},
|
||||
),
|
||||
},
|
||||
"optional": optional,
|
||||
}
|
||||
@@ -217,13 +271,15 @@ class FunPackApplyLoraWeights:
|
||||
|
||||
return abs(float(saved_base) - float(entry["base_model_weight"])) <= 1e-6
|
||||
|
||||
def apply_lora_weights(self, positive_prompt, refinement_key, mode, **kwargs):
|
||||
def apply_lora_weights(self, positive_prompt, refinement_key, mode, per_block=False, **kwargs):
|
||||
mode = (mode or "ltx2").lower()
|
||||
per_block = coerce_bool(per_block)
|
||||
prompt_key = prompt_key_for_mode(positive_prompt, mode)
|
||||
suggestions, source_message = self._load_suggestions(refinement_key, mode, prompt_key)
|
||||
|
||||
loras = []
|
||||
lines = [f"FunPack Apply LoRA Weights | {source_message}"]
|
||||
lines.append(f"Per-block application: {'enabled' if per_block else 'disabled'}")
|
||||
for entry in self._iter_slots(kwargs):
|
||||
suggestion = self._get_suggestion(suggestions, entry)
|
||||
model_weight = float(suggestion.get("model_weight", entry["base_model_weight"]))
|
||||
@@ -240,9 +296,10 @@ class FunPackApplyLoraWeights:
|
||||
)
|
||||
|
||||
stack = {
|
||||
"version": 1,
|
||||
"version": 2,
|
||||
"refinement_key": refinement_key,
|
||||
"mode": mode,
|
||||
"per_block": per_block,
|
||||
"positive_prompt": positive_prompt,
|
||||
"prompt_key": prompt_key,
|
||||
"loras": loras,
|
||||
@@ -293,9 +350,115 @@ class FunPackLoraLoader:
|
||||
|
||||
return lora
|
||||
|
||||
def _model_image_model(self, model):
|
||||
model_wrapper = getattr(model, "model", None)
|
||||
model_config = getattr(model_wrapper, "model_config", None)
|
||||
unet_config = getattr(model_config, "unet_config", None)
|
||||
if isinstance(unet_config, dict):
|
||||
return unet_config.get("image_model")
|
||||
return None
|
||||
|
||||
def _per_block_requested(self, entry, lora_stack):
|
||||
return coerce_bool(entry.get("per_block", lora_stack.get("per_block", False)))
|
||||
|
||||
def _per_block_supported(self, model, lora_stack, entry):
|
||||
if not self._per_block_requested(entry, lora_stack):
|
||||
return False
|
||||
|
||||
if (lora_stack.get("mode") or "ltx2").lower() != "ltx2":
|
||||
return False
|
||||
|
||||
return self._model_image_model(model) in LTX_IMAGE_MODELS
|
||||
|
||||
def _load_model_lora_patches(self, model, lora):
|
||||
key_map = comfy.lora.model_lora_keys_unet(model.model, {})
|
||||
converted_lora = comfy.lora_convert.convert_lora(lora)
|
||||
return comfy.lora.load_lora(converted_lora, key_map)
|
||||
|
||||
def _split_model_patches_by_block(self, loaded):
|
||||
global_patches = {}
|
||||
block_patches = defaultdict(dict)
|
||||
|
||||
for patch_key, patch_value in loaded.items():
|
||||
block_index = transformer_block_index(patch_key)
|
||||
if block_index is None:
|
||||
global_patches[patch_key] = patch_value
|
||||
continue
|
||||
block_patches[block_index][patch_key] = patch_value
|
||||
|
||||
return global_patches, dict(block_patches)
|
||||
|
||||
def _block_scales_from_patches(self, block_patches):
|
||||
block_scores = {}
|
||||
for block_index, patches in block_patches.items():
|
||||
score = 0.0
|
||||
for patch_value in patches.values():
|
||||
score += patch_energy(patch_value)
|
||||
if score > 0.0:
|
||||
block_scores[block_index] = score
|
||||
|
||||
if len(block_scores) < 2:
|
||||
return {}
|
||||
|
||||
mean_score = sum(block_scores.values()) / len(block_scores)
|
||||
if mean_score <= 0.0:
|
||||
return {}
|
||||
|
||||
scales = {}
|
||||
for block_index, score in block_scores.items():
|
||||
ratio = max(0.0, score / mean_score)
|
||||
scales[block_index] = max(0.25, min(1.75, ratio ** 0.5))
|
||||
|
||||
return scales
|
||||
|
||||
def _apply_model_patches(self, model, loaded, model_weight, block_scales=None):
|
||||
new_model = model.clone()
|
||||
applied = set()
|
||||
global_patches, block_patches = self._split_model_patches_by_block(loaded)
|
||||
|
||||
if global_patches:
|
||||
applied.update(new_model.add_patches(global_patches, model_weight))
|
||||
|
||||
if block_scales:
|
||||
for block_index in sorted(block_patches):
|
||||
block_strength = model_weight * block_scales.get(block_index, 1.0)
|
||||
applied.update(new_model.add_patches(block_patches[block_index], block_strength))
|
||||
else:
|
||||
for block_index in sorted(block_patches):
|
||||
applied.update(new_model.add_patches(block_patches[block_index], model_weight))
|
||||
|
||||
for patch_key in loaded:
|
||||
if patch_key not in applied:
|
||||
logging.warning("NOT LOADED %s", patch_key)
|
||||
|
||||
return new_model, len(global_patches), len(block_patches)
|
||||
|
||||
def _load_lora_per_block(self, model, lora, model_weight):
|
||||
loaded = self._load_model_lora_patches(model, lora)
|
||||
_, block_patches = self._split_model_patches_by_block(loaded)
|
||||
block_scales = self._block_scales_from_patches(block_patches)
|
||||
if not block_scales:
|
||||
return None, "per-block fallback=global"
|
||||
|
||||
new_model, non_block_count, _ = self._apply_model_patches(
|
||||
model,
|
||||
loaded,
|
||||
model_weight,
|
||||
block_scales=block_scales,
|
||||
)
|
||||
min_scale = min(block_scales.values())
|
||||
max_scale = max(block_scales.values())
|
||||
status = (
|
||||
f"per-block blocks={len(block_scales)} non_block={non_block_count} "
|
||||
f"range={min_scale:.2f}..{max_scale:.2f}"
|
||||
)
|
||||
return new_model, status
|
||||
|
||||
def load_loras(self, model, lora_stack, clip=None):
|
||||
loras = lora_stack.get("loras", []) if isinstance(lora_stack, dict) else []
|
||||
per_block = coerce_bool(lora_stack.get("per_block", False)) if isinstance(lora_stack, dict) else False
|
||||
lines = [f"FunPack LoRA Loader | loading {len(loras)} LoRA(s)"]
|
||||
lines.append(f"Per-block application: {'enabled' if per_block else 'disabled'}")
|
||||
loaded_count = 0
|
||||
|
||||
for entry in loras:
|
||||
@@ -305,11 +468,22 @@ class FunPackLoraLoader:
|
||||
continue
|
||||
|
||||
lora = self._load_lora_file(entry["name"])
|
||||
model, clip = comfy.sd.load_lora_for_models(model, clip, lora, model_weight, 0.0)
|
||||
apply_status = "global"
|
||||
if self._per_block_supported(model, lora_stack, entry):
|
||||
per_block_model, apply_status = self._load_lora_per_block(model, lora, model_weight)
|
||||
if per_block_model is not None:
|
||||
model = per_block_model
|
||||
else:
|
||||
model, clip = comfy.sd.load_lora_for_models(model, clip, lora, model_weight, 0.0)
|
||||
elif self._per_block_requested(entry, lora_stack):
|
||||
apply_status = "per-block unsupported -> global"
|
||||
model, clip = comfy.sd.load_lora_for_models(model, clip, lora, model_weight, 0.0)
|
||||
else:
|
||||
model, clip = comfy.sd.load_lora_for_models(model, clip, lora, model_weight, 0.0)
|
||||
loaded_count += 1
|
||||
lines.append(
|
||||
f"lora_{entry.get('slot', '?')}: {entry['name']} "
|
||||
f"applied={model_weight:+.3f} source={entry.get('source', 'base')}"
|
||||
f"applied={model_weight:+.3f} source={entry.get('source', 'base')} mode={apply_status}"
|
||||
)
|
||||
|
||||
if loaded_count == 0:
|
||||
|
||||
+1
-1
@@ -2,7 +2,7 @@
|
||||
[project]
|
||||
name = "ComfyUI-FunPack" # Unique identifier for your node. Immutable after creation.
|
||||
description = "A set of nodes for experiments with video generation models, including prompt refinement, adaptive LoRA weight workflows, story tools, video stitching, lorebook enhancement and more."
|
||||
version = "2.0.0" # Custom Node version. Must be semantically versioned.
|
||||
version = "2.1.0" # Custom Node version. Must be semantically versioned.
|
||||
license = { text = "GNU GPL v3" }
|
||||
dependencies = [] # Filled in from requirements.txt
|
||||
|
||||
|
||||
Reference in New Issue
Block a user