Phase 1 amended: the engine now also saves the frozen conditioning (cond.pt) per batch
so submit can train on (cond, rating).
Phase 2 (panel + API): new batch_training.py registers /funpack/batch/{list,submit,forget}
(follows the existing /funpack/* PromptServer pattern). Studio gains a 'Batch Training'
section under the refiner: lists the pending batch for the active refinement key with each
entry's filename + seed, a Preview button (loads the temp webp via /view?type=temp), and a
per-entry V2-rating dropdown, plus Submit and 'Forget batch'. Previews served straight from
ComfyUI temp — no custom file route needed.
Phase 3 (learning intake): Submit loads cond.pt and trains the value function on
(cond, reward) for every rated entry — same conditioning, N rewards = variance-reduction
RLHF, which directly sharpens conditioning ascent. '-Just forget it-'/unknown ratings are
skipped. Submit consumes (deletes) the batch; Forget discards it without learning. Either
way the panel clears and the sampler is ready for a new batch.
Human stays the rating oracle throughout (no auto-rating) — 'user over everything'.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
LTXAVTEModel.load_sd passes the full checkpoint sd to Gemma3_12B.load_state_dict,
which has vision_model and multi_modal_projector as real attributes with matching
key names. Vision weights are already present after DualCLIPLoader runs.
Drop _vision_loaded flag check; detect vision capability by attribute existence.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Companion to DualCLIPLoader: takes the already-built CLIP object (which has
the correct LTXAVTEModel structure) and loads the SigLIP vision tower +
multi-modal projector from the Gemma3-12B-IT checkpoint, setting
_vision_loaded=True. Workflow: DualCLIPLoader → FunPackGemmaVision → Studio.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
FunPackGemmaLoader: drop-in replacement for Load CLIP Model that additionally
loads the SigLIP vision tower and multi-modal projector from Google's
Gemma3-12B-IT checkpoint into the already-initialized Gemma3_12B module,
enabling reference-image vision prompting without a separate CLIP Vision model.
Vision prompting: _v2_encode_prompt accepts reference_image; when the clip
has _vision_loaded=True, tokenizes with image=reference_image and
skip_template=False so Gemma3 sees the reference frame before generating
conditioning. Cache key includes image fingerprint to avoid stale hits.
source_image from Studio/refine_v2 flows through to _v2_conditioning_source.
K/V injection revamp (Technique 5): identity blocks in i2v mode now use
in-context conditioning instead of lerp injection. Pre-hook prepends reference
hidden-state tokens to vx and extends v_pe with neutral-rotation entries
(cos=1, sin=0) so video tokens self-attend to reference tokens in attn1.
Post-hook crops the reference prefix from the block output. Strength raised
from 0.02 → 1.0 (sigma-gated: active at 0.35–0.80). _extend_v_pe helper
handles 2-D/3-D/4-D PE tensors. Anchor blocks keep the existing lerp path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ComfyUI already has a built-in manual context windows node for long-video
windowed denoising. FunPackContextTransitionWindows duplicated that
functionality without adding anything unique, and never worked correctly
for the scene-cut use case (LTXAV's packed latent format was incompatible).
Remaining nodes:
- FunPackSceneCutWindows: attention-level scene masking (MODEL in/out)
- FunPackSceneNoise: per-scene seeded noise (LATENT out, pairs with KSamplerAdvanced)
context_transition.py is now ~180 lines vs ~320 before.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
FunPackSceneCutWindows: correct scene-cut approach for LTXAV and LTXV.
Instead of context windows (which slice x_in and fail for LTXAV's packed
format), this node patches optimized_attention at the attention level.
All frames denoised in one forward pass; self-attention is blocked across
scene boundaries (tokens in scene i can only attend to tokens in scene i).
Cross-attention to text conditioning is unchanged.
Works by: (1) FunPackSceneCutHandler reads num_scenes from conds in
should_use_context and returns False (no context windowing); (2) a
model_function_wrapper temporarily replaces optimized_attention with a
per-scene version before each model forward, then restores it.
FunPackSceneNoise: generates initial noise with a different seed per
temporal scene region. Pair with KSamplerAdvanced (add_noise=disable)
or alongside FunPackSceneCutWindows for additional bias toward scene
boundaries. Scene count auto-detected from positive conditioning entries.
Seed auto-detectable from Studio/Refiner V2 seed output.
FunPackContextTransitionWindows: kept for long-video windowed denoising
(non-LTXAV, standard-format latents only); description updated to note
LTXAV incompatibility.
Also in this commit:
- Phrase stacking fix: merged pure-transition segments with next segment
(direction was backwards - was appending to prev, now prepends to next)
- New phrases: 'the scene begins with', 'after a brief scene cut',
'as the scene transitions', 'in the next scene', 'the video concludes',
'another scene shows', 'later another scene shows', 'the final scene',
'the last scene'
- Add scene ordinals 'the final scene' / 'the last scene'
- Diagnostic log in should_use_context for context windows
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
FunPackStudio combines Refinement Key, Scene Builder, Refiner V2,
Advisor LLM, LoRA loading, and Conditioning Adjust under a single node
with a tabbed popup editor. Rating widget stays on the node face;
everything else is in the popup.
Python:
- FunPackStudio.run() chains all subsystems: scene builder prompt
construction, advisor LLM load (using shared cache), LoRA loading via
FunPackLoraLoader, FunPackVideoRefinerV2.refine_v2(), and
FunPackConditioningAdjust.adjust(). No code duplication.
- Added optional _seed parameter to FunPackVideoRefinerV2.refine_v2()
so Studio can generate a seed and output it for wiring to samplers.
- Added /funpack/available_loras endpoint for the LoRA picker.
JS (funpack_studio.js):
- Tabbed popup: Session, Scene, Refiner, Advisor, LoRA, Adjustments.
- Session: refinement key, reset toggle, scene builder mode selector.
- Scene: scene load/save, phrase bank from memory, positive prompt
editor (active when scene builder is not Pass-through).
- Refiner: all V2 settings, feedback prompt, intent override.
- Advisor: LLM enable toggle, model path, dtype.
- LoRA: dynamic list of LoRAs with name/type/model-weight/clip-weight,
loaded from /funpack/available_loras.
- Adjustments: phrase/strength pairs with session phrase bank.
- Settings serialized as JSON in hidden studio_settings widget.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Inserts between Refiner V2 and the sampler. Takes CONDITIONING + CLIP,
lets the user specify phrase/strength pairs in a floating popup panel.
For each active pair: encode the phrase, compute unit-norm direction from
base conditioning toward phrase conditioning (in pooled space), apply at
user-set strength. Positive = push toward that phrase, negative = away.
Bypasses V2's category classification entirely - adjustments target the
exact phrase the user typed, no mis-categorization possible.
Popup hides the raw JSON widget, shows "Edit Adjustments" button on the
node face with a compact summary of active adjustments. Close saves.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Strip <think>...</think> blocks in decode so Qwen3 chain-of-thought
doesn't pollute the repaired prompt and cause validation to reject it
- Pass enable_thinking kwarg to apply_chat_template (with TypeError
fallback for models that don't support it); propagate thinking flag
through tokenize -> generate so the token budget is expanded by 2048
when thinking mode is active, giving room for reasoning + response
- Return 1.0 from _v2_text_semantic_similarity when the clip has no
encode_from_tokens_scheduled - generation-only clips can't compute
similarity so they shouldn't be penalized with a 0.0 semantic gate
- Add FunPackAdvisorLLM to standalone (non-package) imports in
__init__.py so tests can run without the full ComfyUI runtime
- Update three stale advisor tests: 5-value _v2_prompt_advisor unpack,
"Encoded as:" prefix change, 6-return-value unpacking after negative
conditioning was removed
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
FunPackAdvisorLLM:
- Loads any HuggingFace CausalLM via AutoModelForCausalLM.from_pretrained
- Wraps it in _FunPackAdvisorLLMWrapper which exposes tokenize/generate/decode
matching the interface _v2_generate_advisor_text already uses
- tokenize() accepts system_prompt kwarg, applies the model's own chat
template via apply_chat_template, absorbs unsupported kwargs silently
- generate() uses max_new_tokens (not max_length) so prompt length does
not eat into the generation budget
- Model cached in _FUNPACK_ADVISOR_LLM_CACHE by (path, dtype) - no
reload on repeated runs
- Returns CLIP type so it plugs directly into advisor_clip input
- Default: huihui-ai/Huihui-Qwen3-8B-abliterated-v2
Session reset fix:
- _v2_empty_state was missing intent_expansion_memory, session_source_mean_count,
liked_dir, bad_dir - these survived reset via setdefault on next run.
Added all four to the empty state so reset is complete.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
New sampler targeting LTX2.3 distilled LoRA and similar 4-8 step
flow-matching distilled video models. Three improvements over plain Euler:
Adams-Bashforth 2-step multistep (order=2, default):
Extrapolates the denoised direction from two consecutive steps using
variable-step-size AB2 coefficients: (1 + r/2) * d_curr - (r/2) * d_prev
where r = h_current / h_previous. Gives second-order accuracy across the
large sigma jumps typical of distilled schedules at zero extra model-call cost.
Heun predictor-corrector on final steps (final_correction_steps=1, default):
Replaces the last N steps with a Heun pass: Euler predictor to sigma_next,
second model call at sigma_next for the corrector direction, averaged update.
Costs one extra model call per corrected step and significantly improves
sharpness and detail in the steps that define the final output.
Multistep history is invalidated after each Heun step.
Optional s_noise (default 0.0 = deterministic):
Small ancestral-style noise injection (scaled by sigma_up) for diversity.
Distilled models are trained as pure ODE; any non-zero value is an
intentional deviation. Keep below 0.15 if used at all.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
