diff --git a/README.md b/README.md index 95975520..04df206a 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,8 @@ Important: the Scene Chain sampler is resource heavy. Long scene chains can crea Also improved the Studio Scene Builder mode dropdown refresh and expanded the transition phrase list for camera moves, final shots, and scene progression language. +Studio now learns successful sampler seeds when the `seed` output is connected. `Perfect` and `Loved it` ratings can store concept-matched seeds under the active refinement key, and split-scene mode passes per-scene seed metadata to the Scene Chain sampler. The sampler can either use those unique scene seeds or reuse one seed for every scene with `use_same_seed`. + ## Updates in 2.6.0 Added `FunPack Studio` - a single node that combines Refinement Key management, Scene Builder, LoRA loading, Refiner V2, Conditioning Adjust, and sampler configuration into one interface. The node face shows only the rating widget and an Open Studio button. All settings live in a tabbed popup editor. @@ -30,62 +32,6 @@ Also added as standalone nodes: `FunPack Conditioning Adjust` (phrase-level cond Fixed `FunPackAdvisorLLM` crashing with newer transformers versions (`apply_chat_template` returning `BatchEncoding` instead of a plain tensor). Fixed advisor producing empty or garbled output for Qwen3 and other thinking models (thinking tokens stripped correctly, attention mask now explicitly provided). Fixed conditioning adjustments not applying for LTX/Gemma3 workflows (was reading `pooled_output` which is `None` for T5 encoders, now uses sequence mean). Fixed `prompt_repair=False` not preventing phrase injection from perfect-repair memory and emphasis. Removed the advisor gate that blocked repair for Perfect-rated outputs. -## Updates in 2.5.1 - -Added `FunPack Advisor LLM` node - loads any HuggingFace CausalLM (sharded or single-file) as an advisor for Refiner V2 or as a drop-in replacement for the built-in `TextGenerate` node. Set a HuggingFace repo ID or local path, pick dtype, connect the output to `advisor_clip`. Model is cached after first load. Compatible with `skip_template`, `min_p`, `presence_penalty`, and any model architecture that `AutoModelForCausalLM.from_pretrained` supports. - -Advisor prompt format rewritten to plain natural language so enhancement-type models (Sulphur, Qwen prompt enhancers) produce useful output rather than ignoring structured field-value instructions. System prompt reduced to one sentence. - -Training info Adaptation section now shows direction memory in plain language: run count per slot, magnitude, whether each axis is in direction mode or lerp fallback, and what was applied this run. Model patch status shows which directions were injected into cross-attention and which phrase texts are being emphasized. - -Fixed several bugs: advisor returning None when tokenization succeeded (generate was inside the wrong branch), session reset not clearing newer memory fields, system prompt bleeding into advisor output, and repetition loops from the 1.05 penalty being too weak for 8B+ models. - -Added persistent cross-run encode cache so phrase encodings are not recomputed every run. - -## Updates in 2.5.0 - -Added a two-pass CLIP text-generation advisor to `FunPack Video Refiner V2`. Pass 1 analyses what specifically needs to change in the suggested prompt. Pass 2 applies those findings. The advisor uses a structured input format — `ORIGINAL_USER_INTENT`, `LAST_PROMPT`, `RATING`, and `OPTIONAL_NOTE` — so the model knows exactly what failed and what to fix. Token budget: 1200 tokens for analysis, 1600 for repair. - -Added `feedback_prompt` input. When connected, the user's natural-language description of what was wrong is placed first in both advisor passes and the system follows it exactly, overriding axis-based repair logic and validation guards. - -Added `Prompt only` execution mode: all prompt shaping runs normally but conditioning vectors pass through unchanged. Added `prompt_repair` boolean to disable the rule-based phrase injection from memory when not enough context has been built yet. - -Added `encoded_prompts` STRING output. When the advisor ran it shows up to four sections: `Positive prompt` (what was encoded), `Advisor suggestion (applied/rejected)` (the advisor's generated prompt), `Advisor analysis` (diagnostic from the analysis pass), and `Pre-advisor prompt` (the prompt before the advisor rewrote it). - -Added `eta_final` parameter to `FunPack Hybrid Euler 2S Sampler`. When set below `eta`, ancestral noise decays toward this value as sigma approaches the quality phase boundary. The early phase now uses order-2 denoised extrapolation (Adams-Bashforth 2-step) at zero extra model-call cost, and the quality phase uses progressive correction blending to reduce the number of expensive 2S evaluations. - -Removed `negative_prompt` input and `modified_negative` conditioning output from Refiner V2. Negative conditioning has no effect at CFG=1.0 and was adding a redundant generation call to every advisor run. - -Fixed greedy decoding in the advisor: `do_sample` was `False`, causing the model to always produce its highest-probability default output. Fixed `Only prompt` mode running two generation calls per invocation (positive + the now-removed negative advisor). Fixed `encoded_prompts` always showing only the positive prompt regardless of advisor activity. Also fixed four logic bugs: intent example lookup reading a nonexistent field, streak updates contaminating conditioning strength in Prompt only mode, unfiltered global phrase memory injecting unrelated vocabulary, and the first-run rating label being forwarded when no previous output existed. - -## Updates in 2.4.2 - -Added Refiner V2 `Learning` mode. It observes prompts, conditioning, ratings, phrase memory, and diagnostics while passing positive and negative prompt conditioning through unchanged. - -Fixed Scene Builder mode handling so the live Mode control stays independent from the selected saved scene, and fixed prompt editing so the cursor can move past a final phrase chip with the mouse or right arrow key. - -## Updates in 2.4.1 - -Improved Scene Builder database editing: phrase rows now expose the full phrase as a hover hint, and double-click editing opens a wider multiline editor with explicit OK/Cancel buttons. - -## Updates in 2.4.0 - -Added `FunPack Scene Builder`, a replacement for `FunPack Template Manager`. It stores named scenes from editable positive/negative prompt text, takes prompt/intent text through connection-only inputs, passes the current LoRA stack through unchanged, and can auto-apply a scene when its name or alias appears in an intent prompt. - -Added Scene Builder `Learning` mode for changing-prompt generations: it collects phrase memory inside the selected refinement key while passing the connected prompt data through unchanged, and Refiner resets preserve that Scene Builder memory. - -Redesigned Scene Builder as a compact button-driven node with centered editor menus for scene name, mode, aliases, Positive Prompt, Negative Prompt, and Database controls. First use asks for a scene name before editing, prompt editors preserve freely typed text while phrase chips can be clicked or dragged in from the database, used chips are highlighted in the prompt editor, database words can be double-clicked for inline editing, connected prompts now teach useful words as well as phrase chunks, and wildcard random choice is a simple checkbox. - -Added searchable LoRA selection to `FunPack Apply LoRA Weights` while keeping the compact row UI and serialized LoRA stack format. - -Added advisory image/CLIP Vision context and repaired negative conditioning to `FunPack Video Refiner V2`. Vision inputs are stored as metadata and diagnostics only; they are not blended into positive conditioning. - -Added an opt-in experimental early velocity bias mode to `FunPack Hybrid Euler 2S Sampler` for capturing/applying averaged early denoise directions around normalized sigma 0.9 and 0.8. - -## Updates in 2.3.x - -Refiner V2 now supports pre-encoded conditioning workflows, original-intent alignment memory, improved prompt repair scoping, and a `Wrong appearance` rating for outputs polluted by remembered appearance concepts. Added `FunPack Refinement Key Loader`. - ## Dev Branch The `dev` branch is intended for testing unfinished changes, implementing new logic and basically, flipping everything just because I can. It can be broken, renamed, or changed without warning. @@ -145,6 +91,21 @@ Version history is available in [CHANGELOG.md](CHANGELOG.md). If you have suggestions, questions, or ideas for new nodes, feel free to open an issue or submit a pull request. +## Intent + +FunPack is a hobby project, provided to you by a fellow AI enthusiast who "lives in the trenches" and knows exactly what people seek in video/audio generation workflows. + +FunPack is provided under GNU General Public License V3, which gives you broad rights to use, modify and distribute the original/modified version of it as long as the original license text is included. FunPack places no limitations on types of content you can generate by using it, meaning both SFW and NSFW content are fine as long as you don't violate your local laws. GPLv3 does not grant you rights for such violations. + +However, I do not endorse using FunPack and/or demonstrating it alongside morally and legally questionable/prohibited content, including: +- Non-consensual explicit depiction of a real person; +- Explicit depiction of minors; +- Depiction of violence and gore targeted at a real person. + +I do not provide support to users who use FunPack in such cases, and in case I detect it, any support will be immediately ceased. + +Thanks for understanding. + ## Thank you I want to say thanks to teams behind OpenAI (ChatGPT/Codex), xAI (Grok), DeepSeek, Anthropic (Claude) and Google (Gemini) for all the help with coding and transforming my ideas into something working in real UI. Thanks to all the testers and users who regularly use FunPack in their workflows, request features and report bugs. Without all of you, this project would've been just wet dreams of a wannabe coder begging on Discord for someone to add nodes he wants. Seriously, you are cool. I love you all. <3 diff --git a/conditioning.py b/conditioning.py index 894055d9..866a7d95 100644 --- a/conditioning.py +++ b/conditioning.py @@ -19,6 +19,10 @@ from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer import folder_paths +V2_SEED_REUSE_CHANCE = 0.20 +V2_SEED_MEMORY_MAX_CONCEPTS = 128 +V2_SEED_MEMORY_MAX_PER_CONCEPT = 24 + LORA_REFINER_TYPE_PROFILES = { "general": {"step": 0.025, "max_offset": 0.20, "min_offset": -0.35, "bad_max_offset": 0.45, "bad_min_offset": -1.35, "culprit_bias": 0.28}, "action": {"step": 0.046, "max_offset": 0.35, "min_offset": -0.45, "bad_max_offset": 0.75, "bad_min_offset": -2.10, "culprit_bias": 0.16}, @@ -5967,6 +5971,7 @@ class FunPackVideoRefinerV2(FunPackVideoRefiner): data["global"].setdefault("advisor_feedback_history", []) data["global"].setdefault("vision_memory", {}) data["global"].setdefault("loss_history", []) + data["global"].setdefault("successful_seed_memory", {}) return data, "loaded" except (json.JSONDecodeError, OSError, ValueError): return self._v2_empty_state(refinement_key), "reset unreadable" @@ -6158,6 +6163,184 @@ class FunPackVideoRefinerV2(FunPackVideoRefiner): conditionings.append((cond, scene_meta)) return conditionings + def _v2_scene_seed_values(self, seed, scene_count, provided=None): + scene_count = max(0, int(scene_count or 0)) + provided = list(provided or []) + out = [] + for index in range(scene_count): + raw_seed = provided[index] if index < len(provided) else int(seed) + index + try: + out.append(int(raw_seed)) + except (TypeError, ValueError): + out.append(int(seed) + index) + return out + + def _v2_apply_scene_seed_metadata(self, conditioning, scene_seeds, seed_source): + out = [] + for index, item in enumerate(conditioning or []): + if not isinstance(item, (list, tuple)) or len(item) < 2: + out.append(item) + continue + meta = dict(item[1]) if isinstance(item[1], dict) else {} + if index < len(scene_seeds): + meta["funpack_scene_seed"] = int(scene_seeds[index]) + meta["funpack_seed_source"] = str(seed_source or "base+index") + out.append((item[0], meta)) + return out + + def _v2_seed_concepts_from_text(self, text, limit=16): + stop = { + "this", "that", "with", "from", "into", "onto", "then", "scene", "shows", + "video", "animated", "camera", "final", "next", "segment", "through", + "after", "before", "while", "where", "when", "there", "their", "about", + "style", "quality", "cinematic", "distinct", "moments", + } + concepts = [] + for word in re.findall(r"[a-z0-9][a-z0-9'_-]{2,}", str(text or "").lower()): + word = word.strip("'_-") + if len(word) < 3 or word in stop or word.isdigit(): + continue + if word not in concepts: + concepts.append(word) + if len(concepts) >= limit: + break + return concepts + + def _v2_seed_concepts_for_run(self, run, limit=16): + if not isinstance(run, dict): + return [] + concepts = [] + for phrase in run.get("phrases", []) or []: + if not isinstance(phrase, dict): + continue + for word in self._v2_seed_concepts_from_text(phrase.get("text", ""), limit=limit): + if word not in concepts: + concepts.append(word) + if len(concepts) >= limit: + return concepts + for source in (run.get("prompt", ""), run.get("encoded_prompt", "")): + for word in self._v2_seed_concepts_from_text(source, limit=limit): + if word not in concepts: + concepts.append(word) + if len(concepts) >= limit: + return concepts + return concepts + + def _v2_trim_successful_seed_memory(self, memory): + if not isinstance(memory, dict): + return {} + for entry in memory.values(): + if not isinstance(entry, dict): + continue + seeds = [item for item in entry.get("seeds", []) if isinstance(item, dict)] + seeds.sort(key=lambda item: (int(item.get("hit_count", 0)), int(item.get("last_iteration", 0))), reverse=True) + entry["seeds"] = seeds[:V2_SEED_MEMORY_MAX_PER_CONCEPT] + if len(memory) <= V2_SEED_MEMORY_MAX_CONCEPTS: + return memory + ordered = sorted( + memory.items(), + key=lambda pair: max( + [int(item.get("last_iteration", 0)) for item in pair[1].get("seeds", []) if isinstance(item, dict)] or [0] + ), + reverse=True, + ) + return dict(ordered[:V2_SEED_MEMORY_MAX_CONCEPTS]) + + def _v2_update_successful_seed_memory(self, global_state, last_run, rating_profile, iter_num, seed_output_connected=False): + if not seed_output_connected: + return "Seed memory: seed output not connected." + if not isinstance(last_run, dict) or rating_profile.get("skip_learning"): + return "Seed memory: no previous sampler seed to learn from." + if rating_profile.get("key") not in {"like", "loved_it"}: + return "Seed memory: rating not successful." + if not last_run.get("seed_output_connected", seed_output_connected): + return "Seed memory: previous seed output was not connected." + try: + seed = int(last_run.get("seed")) + except (TypeError, ValueError): + return "Seed memory: previous run had no valid seed." + + concepts = self._v2_seed_concepts_for_run(last_run) + if not concepts: + return "Seed memory: no concepts found for seed." + + memory = global_state.setdefault("successful_seed_memory", {}) + prompt_preview = re.sub(r"\s+", " ", str(last_run.get("prompt", "") or "")).strip() + if len(prompt_preview) > 180: + prompt_preview = prompt_preview[:177].rstrip() + "..." + scene_seeds = [] + for item in last_run.get("scene_seeds") or []: + try: + scene_seeds.append(int(item)) + except (TypeError, ValueError): + pass + scene_count = int(last_run.get("scene_count") or len(scene_seeds) or 1) + touched = 0 + for concept in concepts: + entry = memory.setdefault(concept, {"concept": concept, "seeds": []}) + seeds = entry.setdefault("seeds", []) + existing = None + for item in seeds: + if isinstance(item, dict) and int(item.get("seed", -1)) == seed: + existing = item + break + if existing is None: + existing = {"seed": seed, "hit_count": 0} + seeds.append(existing) + existing.update({ + "seed": seed, + "rating": rating_profile.get("label", ""), + "prompt_preview": prompt_preview, + "scene_count": scene_count, + "scene_seeds": scene_seeds, + "last_iteration": int(iter_num), + "hit_count": int(existing.get("hit_count", 0)) + 1, + }) + touched += 1 + global_state["successful_seed_memory"] = self._v2_trim_successful_seed_memory(memory) + return f"Seed memory: stored seed {seed} for {touched} concept(s)." + + def _v2_choose_successful_seed(self, refinement_key, prompt, fallback_seed, seed_output_connected=False, reset_session=False): + fallback_seed = int(fallback_seed) + if not seed_output_connected: + return fallback_seed, "fresh seed; seed output not connected", None + if reset_session: + return fallback_seed, "fresh seed; session reset", None + state, _ = self._v2_load_state(refinement_key, reset_session=False) + memory = state.get("global", {}).get("successful_seed_memory", {}) if isinstance(state, dict) else {} + if not isinstance(memory, dict) or not memory: + return fallback_seed, "fresh seed; no successful seed memory", None + concepts = self._v2_seed_concepts_from_text(prompt) + candidates = [] + for concept in concepts: + entry = memory.get(concept) + if not isinstance(entry, dict): + continue + for item in entry.get("seeds", []) or []: + if isinstance(item, dict): + candidates.append((concept, item)) + if not candidates: + return fallback_seed, "fresh seed; no concept seed match", None + if random.random() >= V2_SEED_REUSE_CHANCE: + return fallback_seed, "fresh seed; successful seed memory skipped", None + candidates.sort( + key=lambda pair: (int(pair[1].get("hit_count", 0)), int(pair[1].get("last_iteration", 0))), + reverse=True, + ) + concept, item = random.choice(candidates[:min(8, len(candidates))]) + try: + seed = int(item.get("seed")) + except (TypeError, ValueError): + return fallback_seed, "fresh seed; selected memory seed invalid", None + scene_seeds = [] + for value in item.get("scene_seeds") or []: + try: + scene_seeds.append(int(value)) + except (TypeError, ValueError): + pass + scene_seeds = scene_seeds or None + return seed, f"successful seed memory: reused {seed} from '{concept}'", scene_seeds + def _v2_conditioning_vector(self, conditioning): if not isinstance(conditioning, torch.Tensor) or conditioning.dim() <= 1: return None @@ -10547,7 +10730,8 @@ class FunPackVideoRefinerV2(FunPackVideoRefiner): refinement_key_input="", positive_conditioning=None, clip_vision_output=None, source_image=None, model=None, mode="Refine", advisor_mode="Off", advisor_thinking=True, advisor_clip=None, feedback_prompt="", prompt_repair=True, temporal_style="natural", - split_by_transitions=False, latent=None, _seed=None): + split_by_transitions=False, latent=None, seed_output_connected=False, + _seed=None, _seed_source="fresh seed", _scene_seeds=None): seed = int(_seed) if _seed is not None else random.randint(1, 0xffffffffffffffff) encode_cache = {} linked_refinement_key = str(refinement_key_input or "").strip() @@ -10582,6 +10766,7 @@ class FunPackVideoRefinerV2(FunPackVideoRefiner): global_state.setdefault("session_source_mean_count", 0) global_state.setdefault("liked_dir", {}) global_state.setdefault("bad_dir", {}) + global_state.setdefault("successful_seed_memory", {}) previous_run = state.get("last_run") previous_run_refusal = self._v2_run_looks_like_refusal(previous_run) has_previous_run = isinstance(previous_run, dict) and not previous_run_refusal @@ -10606,6 +10791,13 @@ class FunPackVideoRefinerV2(FunPackVideoRefiner): int(global_state.get("total_iterations", 0)) + 1, axis_feedback, ) + seed_memory_status = self._v2_update_successful_seed_memory( + global_state, + previous_run, + learning_profile, + int(global_state.get("total_iterations", 0)) + 1, + seed_output_connected=bool(seed_output_connected), + ) scene_sync_status = self._v2_sync_scene_builder_memory( state, global_state, @@ -10632,7 +10824,7 @@ class FunPackVideoRefinerV2(FunPackVideoRefiner): int(global_state.get("total_iterations", 0)) + 1, axis_feedback, ) - memory_status = f"{memory_status}\n{scene_sync_status}\n{intent_family_status}\n{intent_learning_status}" + memory_status = f"{memory_status}\n{seed_memory_status}\n{scene_sync_status}\n{intent_family_status}\n{intent_learning_status}" self._v2_update_conditioning_memory(global_state, previous_run, learning_profile, axis_feedback) if has_previous_run and not learning_profile.get("skip_learning"): self._v2_update_streaks(global_state, learning_profile, update_conditioning_strength=not prompt_only_mode) @@ -10970,6 +11162,23 @@ class FunPackVideoRefinerV2(FunPackVideoRefiner): }, }) + split_scene_texts = [] + current_scene_seeds = [] + current_scene_seed_source = str(_seed_source or "fresh seed") + if split_by_transitions: + try: + split_segments = self._v2_split_prompt_by_transitions(prompt_to_encode) + if len(split_segments) > 1: + split_scene_texts = self._v2_transition_scene_texts(split_segments) + current_scene_seeds = self._v2_scene_seed_values(seed, len(split_scene_texts), _scene_seeds) + current_scene_seed_source = ( + "successful seed memory" + if _scene_seeds else + "base seed + scene index" + ) + except Exception as e: + print(f"[FunPackVideoRefinerV2] Transition seed prep failed: {e}") + if current_prompt_refusal: state["last_run"] = None else: @@ -11002,6 +11211,11 @@ class FunPackVideoRefinerV2(FunPackVideoRefiner): ], "rating_label": "Unrated", "iteration": int(global_state["total_iterations"]), + "seed": int(seed), + "seed_source": str(_seed_source or "fresh seed"), + "seed_output_connected": bool(seed_output_connected), + "scene_count": len(split_scene_texts) if split_scene_texts else 1, + "scene_seeds": current_scene_seeds, } self._v2_update_advisor_feedback_history(global_state, feedback_prompt, advisor_rating_label, int(global_state.get("total_iterations", 0))) if feedback_prompt and intent_source_prompt and not current_prompt_refusal: @@ -11041,10 +11255,16 @@ class FunPackVideoRefinerV2(FunPackVideoRefiner): ) learning_flag = "trained" if has_previous_run and not learning_profile.get("skip_learning") else "not applied" refusal_flag = f" | REFUSAL DETECTED" if refusal_status else "" + seed_status = f"Seed: {int(seed)} ({str(_seed_source or 'fresh seed')})" + if current_scene_seeds: + seed_status += f" | scene seeds: {', '.join(str(item) for item in current_scene_seeds[:8])}" + if len(current_scene_seeds) > 8: + seed_status += f", ... ({len(current_scene_seeds)} total)" status_lines = [ f"V2 {state_status} | {execution_mode} | {rating_label} | iter {global_state['total_iterations']} | learning {learning_flag}{refusal_flag}", adaptation_status, + seed_status, ] for line in (repair_status, advisor_status, intent_alignment_status, vision_status, lucky_status): if _active(line): @@ -11058,6 +11278,7 @@ class FunPackVideoRefinerV2(FunPackVideoRefiner): "Run\n" f"Mode: {execution_mode} | Encoded as: {encoded_role}\n" f"Rating: {rating_label} | Learning: {learning_reason}\n" + f"Seed: {int(seed)} ({str(_seed_source or 'fresh seed')})\n" f"Prompt: {prompt_preview}" ), ( @@ -11195,45 +11416,47 @@ class FunPackVideoRefinerV2(FunPackVideoRefiner): output_conditioning = [(refined, meta)] if split_by_transitions: try: - segments = self._v2_split_prompt_by_transitions(prompt_to_encode) - if len(segments) > 1: - scene_texts = self._v2_transition_scene_texts(segments) - if scene_texts: - scene_conditionings = self._v2_transition_scene_conditionings( - clip, scene_texts, encode_cache=encode_cache, + scene_texts = split_scene_texts + if scene_texts: + scene_conditionings = self._v2_transition_scene_conditionings( + clip, scene_texts, encode_cache=encode_cache, + ) + if scene_conditionings is not None: + output_conditioning = self._v2_apply_scene_seed_metadata( + scene_conditionings, + current_scene_seeds, + current_scene_seed_source, ) - if scene_conditionings is not None: - output_conditioning = scene_conditionings - scene_mode = ( - "Scene chain mode: multi-entry conditioning for " - "FunPack LTXAV Scene Chain Sampler" - ) - else: - scene_mode = "Scene chain mode unavailable: CLIP missing, single conditioning returned" - detected_count = len(scene_texts) - status_suffix = f"\nTransition split: {detected_count} scenes detected\n{scene_mode}" - status = status + status_suffix + enhancement_status - scene_lines = "\n".join( - f" Scene {i + 1}: {t}" for i, t in enumerate(scene_texts) + scene_mode = ( + "Scene chain mode: multi-entry conditioning for " + "FunPack LTXAV Scene Chain Sampler" ) - split_encoded_prompts = ( - self._v2_encoded_prompts_output( - prompt_to_encode, - advisor_diagnostic=advisor_diagnostic, - pre_advisor_prompt=pre_advisor_prompt, - advisor_suggested=advisor_suggested, - ) - + f"\n\nDetected scenes:\n{scene_lines}" - ) - return ( - output_conditioning, - status, - training_info, - loss_graph, - split_encoded_prompts, - patched_model, - video_latent, + else: + scene_mode = "Scene chain mode unavailable: CLIP missing, single conditioning returned" + detected_count = len(scene_texts) + status_suffix = f"\nTransition split: {detected_count} scenes detected\n{scene_mode}" + status = status + status_suffix + enhancement_status + scene_lines = "\n".join( + f" Scene {i + 1}: {t}" for i, t in enumerate(scene_texts) + ) + split_encoded_prompts = ( + self._v2_encoded_prompts_output( + prompt_to_encode, + advisor_diagnostic=advisor_diagnostic, + pre_advisor_prompt=pre_advisor_prompt, + advisor_suggested=advisor_suggested, ) + + f"\n\nDetected scenes:\n{scene_lines}" + ) + return ( + output_conditioning, + status, + training_info, + loss_graph, + split_encoded_prompts, + patched_model, + video_latent, + ) except Exception as e: print(f"[FunPackVideoRefinerV2] Transition split failed: {e}") @@ -12226,23 +12449,29 @@ class FunPackStudio: "tooltip": "Optional video latent for creativity masking. Takes priority over any saved latent for this key. Connect your i2v or previous KSampler output here.", }), }, + "hidden": { + "prompt": "PROMPT", + "unique_id": "UNIQUE_ID", + }, } + @staticmethod + def _is_output_connected(prompt, unique_id, output_index): + return FunPackVideoRefiner()._is_output_connected(prompt, unique_id, output_index) + def run(self, rating, studio_settings, adjustments, model=None, clip=None, advisor_clip=None, positive_conditioning=None, negative_conditioning=None, clip_vision_output=None, source_image=None, lora_stack=None, positive_prompt=None, negative_prompt=None, user_intent_prompt=None, feedback_prompt=None, refinement_key_input="", - latent=None): + latent=None, prompt=None, unique_id=None): try: settings = json.loads(str(studio_settings or "{}")) except Exception: settings = {} - seed = random.randint(1, 0xffffffffffffffff) - # --- Scene Builder --- sb = settings.get("scene_builder", {}) if isinstance(settings.get("scene_builder"), dict) else {} sb_mode = str(sb.get("mode", "Pass-through")).strip() @@ -12308,6 +12537,18 @@ class FunPackStudio: else: key = popup_key # refine_v2 uses refinement_key_input when non-empty + # --- Seed memory --- + refiner = FunPackVideoRefinerV2() + seed_output_connected = self._is_output_connected(prompt, unique_id, 3) + fresh_seed = random.randint(1, 0xffffffffffffffff) + seed, seed_source, scene_seed_memory = refiner._v2_choose_successful_seed( + refinement_key_input if str(refinement_key_input or "").strip() and not ov.get("refinement_key") else key, + active_prompt, + fresh_seed, + seed_output_connected=seed_output_connected, + reset_session=reset_session, + ) + # --- Advisor LLM: external input takes precedence over popup-configured model --- if advisor_clip is None: llm_cfg = settings.get("advisor_llm", {}) @@ -12376,7 +12617,6 @@ class FunPackStudio: print(f"[FunPackStudio] LoRA load failed: {e}") # --- Refiner V2 --- - refiner = FunPackVideoRefinerV2() cond, status, training_info, loss_graph, encoded_prompts, out_model, video_latent = refiner.refine_v2( active_prompt, clip=clip, @@ -12400,7 +12640,10 @@ class FunPackStudio: temporal_style=temporal_style, split_by_transitions=split_by_transitions, latent=latent, + seed_output_connected=seed_output_connected, _seed=seed, + _seed_source=seed_source, + _scene_seeds=scene_seed_memory, ) # --- Conditioning Adjust --- diff --git a/docs/FunPackLTXAVSceneChainSampler.md b/docs/FunPackLTXAVSceneChainSampler.md index 63ebee3f..936689c5 100644 --- a/docs/FunPackLTXAVSceneChainSampler.md +++ b/docs/FunPackLTXAVSceneChainSampler.md @@ -14,12 +14,13 @@ Important: this sampler is resource heavy. Long chains can produce very large fi - `negative`: Negative conditioning shared by every scene. - `sampler`: ComfyUI sampler object, for example from Studio's sampler output. - `sigmas`: Sigma schedule for each chunk. -- `seed`: Base seed. Scene N uses `seed + N`. +- `seed`: Base seed. Scene N uses `seed + N` unless scene seed metadata is present. - `latent_template`: One scene-sized latent template. Plain video and nested AV latents are supported. - `num_frames_per_scene`: Pixel frame count represented by `latent_template`. - `frame_overlap`: Pixel frames to preserve and blend between scene chunks. - `cfg`: Internal CFG value. - `max_scenes`: Maximum scene entries to consume. Default is `8`, but it can be raised for longer chains. +- `use_same_seed`: When off, each scene uses `funpack_scene_seed` metadata from Studio/Refiner split mode, falling back to `seed + scene_index`. When on, every scene uses the first provided scene seed or the base `seed`. - `carry_i2v_guides`: Experimental. Appends protected frames from `latent_template`'s `noise_mask` as hidden LTX guide tokens in each continuation chunk. Default is off. ## Behavior @@ -28,6 +29,8 @@ The first scene samples from a fresh copy of `latent_template`. Each following scene copies the previous output tail into the start of the next chunk, masks that overlap so it is preserved during denoising, samples the new frames with that scene's conditioning, then blends the overlap in latent space. +When Studio or Refiner V2 split mode provides scene seed metadata, the sampler reports and uses those exact seeds. This lets successful seed memory replay a known-good scene seed set while keeping the public Studio `seed` output as a single integer. + When `carry_i2v_guides` is enabled, protected source frames from the incoming `latent_template` are appended as guide tokens with `keyframe_idxs` and `guide_attention_entries`, then cropped away after sampling. This follows the same broad idea as LTX guide/IC-LoRA conditioning: the reference is extra context, not a visible frame inserted into the generated timeline. Keep it off unless you are testing that behavior deliberately. For nested LTXAV latents, video and audio tensors are continued together. Audio overlap is derived from the audio/video latent length ratio. diff --git a/docs/FunPackStudio.md b/docs/FunPackStudio.md index 5e6488ef..6a6592ef 100644 --- a/docs/FunPackStudio.md +++ b/docs/FunPackStudio.md @@ -33,7 +33,7 @@ Only the `rating` widget and the `Open Studio` button are visible on the node. E | `model` | MODEL | Patched model: LoRAs applied, then attn2 direction injection applied on top. | | `modified_positive` | CONDITIONING | Refined positive conditioning with session memory applied. When split by transitions is enabled, this becomes one conditioning entry per detected scene for the Scene Chain sampler. | | `negative` | CONDITIONING | Negative conditioning passed through or encoded from `negative_prompt`. | -| `seed` | INT | The seed used this run. Wire to your sampler for a matching generation seed. | +| `seed` | INT | The seed used this run. Wire to your sampler for a matching generation seed. When connected, Studio can learn successful seeds for the active refinement key. | | `high_pass_sampler` | SAMPLER | Configured high-pass sampler object. | | `high_pass_sigmas` | SIGMAS | Sigma schedule for the high-pass sampler. | | `low_pass_sampler` | SAMPLER | Configured low-pass sampler object. | @@ -78,6 +78,8 @@ Each of the three text inputs (negative prompt, feedback, intent) has an **Overr Split scene conditioning is intended for `FunPack LTXAV Scene Chain Sampler`. Scene labels are order-only: `scene ten` can be the first generated chunk if it is the first detected transition. Keep character description before the first transition so Studio can prepend it to every scene. +When the `seed` output is connected, `Perfect` and `Loved it` ratings store the previous run's seed as a concept-matched successful seed. Studio occasionally reuses matching seeds on future runs. In split mode, per-scene seeds are attached to scene conditioning metadata for the Scene Chain sampler. + ### Advisor Enables and configures an internal HuggingFace CausalLM advisor. Uses the same model cache as the standalone `FunPack Advisor LLM` node, so the model is only loaded once even when both nodes are in the same workflow. diff --git a/docs/FunPackVideoRefinerV2.md b/docs/FunPackVideoRefinerV2.md index 2ef122ea..1f936c9b 100644 --- a/docs/FunPackVideoRefinerV2.md +++ b/docs/FunPackVideoRefinerV2.md @@ -57,7 +57,7 @@ The advisor receives four explicit inputs: `ORIGINAL_USER_INTENT` (the user's in **refinement_key_input**: Optional linked key input, usually from `FunPack Refinement Key Loader`. When connected, overrides the text widget. -**seed**: Optional seed for deterministic exploration paths. +**seed**: Optional seed for deterministic exploration paths. Studio passes its generated seed here and can store successful sampler seeds when its `seed` output is connected. **reset_session**: Clears all V2 session state for the selected key. Scene Builder memory is preserved. @@ -75,6 +75,8 @@ The advisor receives four explicit inputs: `ORIGINAL_USER_INTENT` (the user's in Scene splitting is order-only. Written labels such as `scene ten`, `scene -999999`, or `scene minus infinity` are treated as transition text, not as real scene numbers. Text before the first transition is used as a shared character/global anchor and prepended to every detected scene. +In Studio split mode, each scene conditioning entry also receives `funpack_scene_seed` metadata. Scene Chain uses those seeds when present. + ## Feedback History V2 maintains a rolling history of the last ten user-provided `feedback_prompt` entries in session state. Each entry is labelled with its rating, for example: diff --git a/samplers.py b/samplers.py index 09d39c2c..e670641f 100644 --- a/samplers.py +++ b/samplers.py @@ -780,6 +780,10 @@ class FunPackLTXAVSceneChainSampler: "frame_overlap": ("INT", {"default": 16, "min": 0, "max": 512, "step": 8}), "cfg": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 100.0, "step": 0.1}), "max_scenes": ("INT", {"default": 8, "min": 1, "step": 1}), + "use_same_seed": ("BOOLEAN", { + "default": False, + "tooltip": "Use the first provided scene seed for every scene. Off uses per-scene metadata seeds or seed + scene index.", + }), "carry_i2v_guides": ("BOOLEAN", { "default": False, "tooltip": "Experimental: carry protected frames from latent_template noise_mask into each continuation chunk after the overlap.", @@ -1134,8 +1138,20 @@ class FunPackLTXAVSceneChainSampler: return text return f"Scene {index + 1}" + def _scene_seed(self, scene_conditioning): + if ( + isinstance(scene_conditioning, (list, tuple)) + and len(scene_conditioning) >= 2 + and isinstance(scene_conditioning[1], dict) + ): + try: + return int(scene_conditioning[1].get("funpack_scene_seed")) + except (TypeError, ValueError): + return None + return None + def sample(self, model, vae, positive, negative, sampler, sigmas, seed, latent_template, - num_frames_per_scene, frame_overlap, cfg, max_scenes, carry_i2v_guides=False): + num_frames_per_scene, frame_overlap, cfg, max_scenes, use_same_seed=False, carry_i2v_guides=False): if not isinstance(positive, list) or not positive: raise ValueError("positive conditioning must contain at least one scene entry.") if negative is None: @@ -1151,10 +1167,17 @@ class FunPackLTXAVSceneChainSampler: output = None report_lines = [] carried_guide_frames = 0 + first_scene_seed = self._scene_seed(scene_conditionings[0]) + if first_scene_seed is None: + first_scene_seed = int(seed) for scene_index, scene_cond in enumerate(scene_conditionings): scene_positive = [scene_cond] scene_negative = negative - scene_seed = int(seed) + scene_index + provided_seed = self._scene_seed(scene_cond) + if use_same_seed: + scene_seed = first_scene_seed + else: + scene_seed = provided_seed if provided_seed is not None else int(seed) + scene_index carried = 0 if output is None: chunk = self._clone_latent(latent_template) diff --git a/tests/test_scene_chain_sampler.py b/tests/test_scene_chain_sampler.py index 52c5c1a4..2fd6ba94 100644 --- a/tests/test_scene_chain_sampler.py +++ b/tests/test_scene_chain_sampler.py @@ -155,6 +155,66 @@ def test_scene_chain_detects_scene_count_and_increments_seed(): assert "Scene 3" in report +def test_scene_chain_uses_scene_seed_metadata(): + sample_calls.clear() + node = FunPackLTXAVSceneChainSampler() + latent_template = {"samples": torch.zeros(1, 2, 5, 3, 3)} + positive = [ + (torch.ones(1, 2, 3), {"funpack_scene_text": "scene a", "funpack_scene_seed": 101}), + (torch.ones(1, 2, 3), {"funpack_scene_text": "scene b", "funpack_scene_seed": 202}), + (torch.ones(1, 2, 3), {"funpack_scene_text": "scene c", "funpack_scene_seed": 303}), + ] + + _, _, scene_count, report = node.sample( + model=object(), + vae=FakeVAE(), + positive=positive, + negative=[], + sampler=object(), + sigmas=torch.tensor([1.0, 0.0]), + seed=10, + latent_template=latent_template, + num_frames_per_scene=5, + frame_overlap=2, + cfg=1.0, + max_scenes=8, + ) + + assert scene_count == 3 + assert [call["seed"] for call in sample_calls] == [101, 202, 303] + assert "seed=202" in report + + +def test_scene_chain_use_same_seed_forces_first_seed(): + sample_calls.clear() + node = FunPackLTXAVSceneChainSampler() + latent_template = {"samples": torch.zeros(1, 2, 5, 3, 3)} + positive = [ + (torch.ones(1, 2, 3), {"funpack_scene_text": "scene a", "funpack_scene_seed": 101}), + (torch.ones(1, 2, 3), {"funpack_scene_text": "scene b", "funpack_scene_seed": 202}), + (torch.ones(1, 2, 3), {"funpack_scene_text": "scene c"}), + ] + + _, _, scene_count, _ = node.sample( + model=object(), + vae=FakeVAE(), + positive=positive, + negative=[], + sampler=object(), + sigmas=torch.tensor([1.0, 0.0]), + seed=10, + latent_template=latent_template, + num_frames_per_scene=5, + frame_overlap=2, + cfg=1.0, + max_scenes=8, + use_same_seed=True, + ) + + assert scene_count == 3 + assert [call["seed"] for call in sample_calls] == [101, 101, 101] + + def test_scene_chain_preserves_nested_av_structure_and_audio_length(): sample_calls.clear() node = FunPackLTXAVSceneChainSampler() diff --git a/tests/test_v2_category_alignment.py b/tests/test_v2_category_alignment.py index d4ba9f7e..099b07de 100644 --- a/tests/test_v2_category_alignment.py +++ b/tests/test_v2_category_alignment.py @@ -9,6 +9,7 @@ sys.modules.setdefault("folder_paths", types.SimpleNamespace(models_dir="")) import torch from conditioning import ( + FunPackStudio, FunPackVideoRefinerV2, V2_RATING_LABELS, normalize_refiner_v2_rating, @@ -2443,3 +2444,139 @@ def test_split_by_transitions_has_no_hard_scene_cap(tmp_path): assert cond[-1][1]["funpack_scene_count"] == 11 assert "capped at 8 scenes" not in status assert "Scene 11" in encoded_prompts + + +def test_studio_detects_seed_output_links(): + prompt = { + "output": { + "10": {"class_type": "FunPackStudio", "inputs": {}}, + "20": {"class_type": "Sampler", "inputs": {"seed": ["10", 3]}}, + } + } + + assert FunPackStudio._is_output_connected(prompt, "10", 3) is True + assert FunPackStudio._is_output_connected(prompt, "10", 4) is False + + +def test_successful_seed_memory_requires_connected_seed_output(tmp_path): + refiner = FunPackVideoRefinerV2() + state_path = tmp_path / "state.json" + refiner._v2_state_path = lambda refinement_key: str(state_path) + + refiner.refine_v2("KaiSa walking through neon", FakeClip(), "Perfect", "seed-test", _seed=111) + refiner.refine_v2( + "KaiSa walking through neon", + FakeClip(), + "Perfect", + "seed-test", + _seed=222, + seed_output_connected=False, + ) + + state = json.loads(state_path.read_text()) + assert state["global"]["successful_seed_memory"] == {} + + +def test_successful_seed_memory_stores_previous_seed_not_current(tmp_path): + refiner = FunPackVideoRefinerV2() + state_path = tmp_path / "state.json" + refiner._v2_state_path = lambda refinement_key: str(state_path) + + refiner.refine_v2( + "KaiSa walking through neon", + FakeClip(), + "Missing action", + "seed-test", + _seed=111, + seed_output_connected=True, + ) + refiner.refine_v2( + "KaiSa walking through neon", + FakeClip(), + "Loved it", + "seed-test", + _seed=222, + seed_output_connected=True, + ) + + state = json.loads(state_path.read_text()) + memory = state["global"]["successful_seed_memory"] + stored = [item["seed"] for entry in memory.values() for item in entry["seeds"]] + assert 111 in stored + assert 222 not in stored + + +def test_successful_seed_reuse_matches_concepts(monkeypatch, tmp_path): + refiner = FunPackVideoRefinerV2() + state_path = tmp_path / "state.json" + refiner._v2_state_path = lambda refinement_key: str(state_path) + state_path.write_text(json.dumps({ + "version": 2, + "global": { + "successful_seed_memory": { + "neon": { + "concept": "neon", + "seeds": [{ + "seed": 777, + "hit_count": 2, + "last_iteration": 5, + "scene_seeds": [777, 778, 779], + }], + } + } + }, + "prompt_histories": {}, + "last_run": None, + })) + monkeypatch.setattr("conditioning.random.random", lambda: 0.05) + monkeypatch.setattr("conditioning.random.choice", lambda items: items[0]) + + seed, source, scene_seeds = refiner._v2_choose_successful_seed( + "seed-test", + "KaiSa in neon rain", + 123, + seed_output_connected=True, + ) + + assert seed == 777 + assert scene_seeds == [777, 778, 779] + assert "successful seed memory" in source + + +def test_split_by_transitions_attaches_scene_seeds(tmp_path): + refiner = FunPackVideoRefinerV2() + state_path = tmp_path / "state.json" + refiner._v2_state_path = lambda refinement_key: str(state_path) + + cond, _, _, _, _, _, _ = refiner.refine_v2( + "anchor, scene one she walks, suddenly she turns", + FakeClip(), + "Perfect", + "split-seed-test", + split_by_transitions=True, + _seed=500, + seed_output_connected=True, + ) + + assert [item[1]["funpack_scene_seed"] for item in cond] == [500, 501] + assert all(item[1]["funpack_seed_source"] == "base seed + scene index" for item in cond) + + +def test_split_by_transitions_uses_provided_scene_seeds(tmp_path): + refiner = FunPackVideoRefinerV2() + state_path = tmp_path / "state.json" + refiner._v2_state_path = lambda refinement_key: str(state_path) + + cond, _, _, _, _, _, _ = refiner.refine_v2( + "anchor, scene one she walks, suddenly she turns", + FakeClip(), + "Perfect", + "split-seed-test", + split_by_transitions=True, + _seed=500, + _scene_seeds=[900, 901], + _seed_source="successful seed memory: reused 900 from 'anchor'", + ) + + assert [item[1]["funpack_scene_seed"] for item in cond] == [900, 901] + assert all(item[1]["funpack_seed_source"] == "successful seed memory" for item in cond)