7b00a551d9050945759f779fa3fc2f3dc7cae20d
1
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
092c204060 |
feat(automations): LTX Director iteration-loop framework + 4K investigation docs (#159)
* docs(investigations): kickoff for LTX Director 4K transition breakdown Captures the maintainer's problem statement (HD multi-keyframe works, 4K single-keyframe works, 4K multi-keyframe degenerates into dissolves between keyframes) and an initial conclusion about the WhatDreamsCost LTXDirector node's implementation. Traces the lineage to the Prompt Relay paper (Chen, Huang, Liu — arXiv:2604.10030) via Kijai's ComfyUI port, separates the three mechanisms the node bundles (cross-attention penalty, guide-image keyframes, audio/image preprocessing), and flags the candidates for the 4K dissolve: model-side limit at LTX sampling time vs. node-side length-independent σ in the penalty matrix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(investigations): round 2 — 2K vs 4K + two-stage upscale findings Adds the maintainer's round-2 observations: - 2K renders behave much better than 4K (segments morph instead of dissolving) — model is closer to its training distribution. - The two-stage workflow's upscale stage (stage 2) is overwriting stage 1's audio/animation: lip sync degrades and segment routing weakens. Captures the hypothesis space: - Why 2K > 4K: model prior (out of distribution at 4K) is dominant; token-grid scaling with a length-independent sigma is secondary. - Why stage 2 overwrites stage 1: KSampler denoise re-applies conditioning from scratch; if LTX Director isn't wired into stage 2 the Prompt Relay + audio_attn2 patches go missing; latent shape change between stages forces a penalty rebuild. Includes a 3-option pros/cons table (4K direct vs 2K direct vs 2-stage) and 6 investigation directions ordered cheapest-first, starting with auditing the stage-2 wiring. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(investigations): round 3 — both stages run LTX Director; new prime suspect Maintainer's screenshot confirms both stages route through LTX Director, so round 2's missing-patch hypothesis is ruled out. Audited the stock LTX nodes used in stage 2 against ComfyUI source: - LTXVLatentUpsampler explicitly pop()s the noise_mask and runs a learned (EXPERIMENTAL) neural upscaler — not interpolation. - LTXVConcatAVLatent fills a missing noise_mask with ones_like (denoise fully) for the side that's missing. - LTXVCropGuides removes keyframe slots and clears keyframe_idxs. Traced the audio noise_mask end-to-end through stage 2 and found: audio waveform stays preserved (noise_mask=0), video keyframes stay pinned, but every non-keyframe video frame goes into stage 2's sampler at noise_mask=1 — i.e. fully regenerated at denoise=0.80. That's where stage 1's audio-aligned mouth shapes live, and that's what's getting thrown away. Prime suspect: denoise=0.80 is too high for "preserve stage 1" semantics — most LTX two-stage reference workflows use 0.30–0.40. Secondary suspect: the LTXVLatentUpsampler [BETA] is a learned model that can hallucinate spatially-dense detail (faces, mouths) before stage 2 even starts denoising. Includes a 4-step cheapest-first test plan starting with dropping stage 2 denoise to 0.30. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(investigations): round 4 — scheduler-shape finding + tracking plan Captures two outputs from the planning session so they're preserved for the next pickup: 1. Promotes the maintainer's empirical finding to a locked-in baseline: `linear_quadratic` scheduler at 8 steps produces the right sigma curve shape (flat plateau near sigma=1.0 for ~5 steps, then steep drop in the last 3) for both phases. Any other scheduler degrades fast motion and keyframe transitions. 2. Records the agreed tracking-system plan (BASELINE.md + log.md + per-run screenshot folder, with three 1-5 score axes matching the live trade-off: motion / audio sync / Phase-2 sharpness). Not yet scaffolded — explicit "next step" callout for whoever picks this up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(investigations): /make-card skill + PNG card renderer toolkit Per-iteration tracking card for the LTX Director 4K investigation. Solves the documentation-drift problem we hit on v01->v02 (multiline summary diverged from actual node values) by treating the saved workflow JSON as the source of truth. Adds: - scripts/make_card.py — reads a ComfyUI workflow JSON, paints a self-contained 540px PNG card via Pillow. Extracts Phase 1 run params (Format/Mult/Scheduler/Steps/Denoise), locked base (Sampler/CFG/Seed/epsilon/Audio src), scene (duration/segments with prompt/audio/keyframe checkboard), free-text BASE notes from `OVERLAY - INFO` Text Multiline, FEEDBACK from `OVERLAY - FEEDBACK` Text Multiline, and a POST-RENDER block that auto-computes render duration from JSON->newest-video mtime delta in the same folder. Layout mirrors the project's snapshot-dialogs design language (panelled sections, sky/amber/ green accents matching docs/designs/snapshot-dialogs.html). - scripts/watch_cards.py — background poller. Watches a working folder; auto-renders <stem>_card.png whenever a workflow JSON appears or is updated. Skips up-to-date PNGs. - .claude/skills/make-card/SKILL.md — /make-card slash command; globs the newest JSON in the working folder, calls make_card.py, shows the PNG inline. - .claude/skills/make-card/work-folder.txt — one-line config of the working folder path. Edit to switch projects. - docs/investigations/experiments/{baseline,combined-card,run-001-card}.html — design prototypes that fed into the PNG layout; kept as history. - docs/investigations/experiments/run-001-card.png and run-v04-card.png — sample card outputs from v03 and v04 of the workflow JSON, included for reference. Issue #151 tracks the eventual native ComfyUI custom node so this flow happens inside the graph; for now the external script + watcher is the loop. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(launch): use absolute Windows python path for design-server The default `python3` runtimeExecutable fails on this Windows host (triggers the Microsoft Store install stub). Hard-coding C:\Python313\python.exe lets the preview MCP and the design-server target work on the maintainer's machine. Worktree-scoped change — main keeps the cross-platform `python3` default. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(automations): docs/loop scaffold + env-var working folder + _AI subdir Promotes the LTX Director card-rendering loop from one-off scripts to a documented, project-portable automation framework. Adds `docs/automations/` as the home for AI-assisted iteration loops: - README.md — the pattern; how to start a loop for a new model - CONVENTIONS.md — cross-cutting: working folder, Text Multiline node contract (Working_Folder_PATH / OVERLAY - INFO / OVERLAY - FEEDBACK), card format, iterations log, skill, watcher, run-id rules, tool locations - LTX-2.3/README.md — entry point for the LTX 2.3 loop; per-iteration steps; full extraction table; active investigation status - LTX-2.3/handoff-checklist.md — 5-minute onboarding for a new machine or new person - LTX-2.3/findings.md — promoted findings (linear_quadratic scheduler locked-in; resolution behaviour; audio mode; open two-stage suspects) Replaces the committed `.claude/skills/make-card/work-folder.txt` with an env-var-driven path. Follows the project's existing `KOLOOK_*` convention — `.env.example` documents the new `KOLOOK_AUTOMATIONS_WORK_DIR`. Real path lives in `.env` (gitignored). No more leaking client-specific absolute paths into committed files. `scripts/make_card.py`: - Loads `.env` automatically; worktree-aware (walks the .git common-dir marker to find the main repo when run from a worktree). - Output folder priority: Working_Folder_PATH node in JSON → env var → JSON's parent dir. - Run with no args: auto-discovers newest *.json in the working folder. - Writes `_AI/card.png` + `_AI/iterations.md` inside the working folder, not the root. Lets the maintainer wipe AI artifacts or user content (JSONs/MP4s) independently. `/make-card` skill simplified to a single command — `python scripts/make_card.py` — the script handles env loading and discovery itself. baseline.html: stripped the project-specific label. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(automations): sync all path references to _AI/ + add layout diagram Audit pass after introducing _AI/ subfolder. Updated every doc and script that still referenced root-level card.png / iterations.md: - docs/automations/CONVENTIONS.md (3 spots) - docs/automations/LTX-2.3/README.md (per-iteration steps + composite) - docs/automations/LTX-2.3/handoff-checklist.md (4 spots, including the off-boarding section which conflated repo commits with working- folder artifacts) - docs/automations/LTX-2.3/findings.md (promotion criterion) - scripts/watch_cards.py (target PNG path now matches make_card.py's _AI/card.png — was writing to a stale <stem>_card.png location) Adds a clear "Where things live" section to docs/automations/README.md showing both layouts side-by-side: - the repo tree (where the tooling, skill, env-example, and per-model loop docs sit) - the per-project working-folder tree (user content at root, _AI/ subfolder for managed artifacts) So anyone picking this up can answer "what's in the repo vs. what's in my project folder" without reading prose. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(automations): add CHEATSHEET.md — one-page dense reference Compact reminder card for returning users. Structured by application context: ComfyUI · Scripts · Folder structure · Conventions · Quick-fail diagnostics. Skippable for first-timers; targets the maintainer who already knows the loop and just needs the right node names, score syntax, and command list at a glance. Linked from docs/automations/README.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(automations): add Bootstrap section to CHEATSHEET — bare-bones setup What's needed to get a fresh machine running the loop, separated from the daily-loop reference. Three checklists: - Repo side: Python + Pillow + .env config - ComfyUI side: was-node-suite + the three required Text Multiline nodes, with exact titles - First card: /make-card → expect _AI/card.png Deliberately skips model-specific setup (checkpoints, LoRAs, per-model custom nodes) — those live in each model's handoff-checklist.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(automations): skip *loop* files in auto-discovery + watcher The maintainer's workflow includes a "loop" stage in ComfyUI that composites the card next to the video and writes the result back into the working folder as `<base>_loop_<seq>.{mp4,json}`. Without a skip rule, auto-discovery treats this composited-card output as the newest input — the script then re-renders the card from yesterday's metadata. Now: any JSON or video file with `loop` in its stem (case-insensitive) is ignored by: - scripts/make_card.py — auto-discover newest JSON, newest video - scripts/watch_cards.py — polling glob Explicit invocation still works: `python scripts/make_card.py <wf_loop.json>` will render a card for that file specifically. Documents the rule in docs/automations/CONVENTIONS.md (new §7) and adds a one-line bullet to CHEATSHEET conventions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style(automations): satisfy Ruff lint — split if-on-one-line, rename l, drop os CI's Lint & Smoke step flagged 17 violations across the new scripts. Cleaned up: - scripts/make_card.py: 11 × E701 (multi-statement lines) split every `if cond: stmt` onto two lines. - scripts/make_card.py: 5 × E741 (ambiguous variable `l`) renamed to `link` in the two link-dict comprehensions and their downstream lookups. - scripts/watch_cards.py: 1 × F401 (unused `os` import). No functional change — `make_card.py` re-renders the working folder's card identically post-cleanup. Verified `ruff check` returns "All checks passed!" on both files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(automations): render time from ComfyUI log, not disk timestamps Disk mtimes don't reliably bracket the model run — the maintainer's workflow saves the workflow JSON manually after rendering, assembles QuickTime from EXRs separately, and ComfyUI's metadata bundle writes JSON + MP4 at the same instant. None of those give us actual render duration. Render time now sourced from ComfyUI's own log (`<comfyui-root>/user/comfyui.log`): - Find the latest `saving images: 100%` line → pair it with the next `Prompt executed in X` → that's the most recent EXR-producing render wall-time. - Supports both `Prompt executed in 18.21 seconds` and `Prompt executed in HH:MM:SS` log formats. - Skips trivial follow-up prompts (e.g. audio-VAE load between render and /make-card invocation). Log path resolution: 1. `KOLOOK_COMFYUI_LOG` env var (explicit override). 2. Inferred from existing `KOLOOK_COMFYUI_DEV_PATH` → `<comfyui-root>/user/comfyui.log`. Fallbacks remain in priority order: EXR sequence span (oldest to newest in the latest version subfolder) → MP4 mtime delta → "pending". Docs: - .env.example: documents the optional KOLOOK_COMFYUI_LOG. - CONVENTIONS.md: new §4 "Render duration" describing the resolution chain; section numbers renumbered. - CHEATSHEET.md: updated the "Render time —" diagnostic. Lint: ruff clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |