From 9e054e6105998da5ca3abb436f6d59719669ff5c Mon Sep 17 00:00:00 2001 From: silveroxides Date: Sun, 13 Sep 2026 06:39:19 +0200 Subject: [PATCH] docs: Record per-layer parameter plan --- plans/per-layer-parameter-rules.md | 237 +++++++++++++++++++++++++++++ 1 file changed, 237 insertions(+) create mode 100644 plans/per-layer-parameter-rules.md diff --git a/plans/per-layer-parameter-rules.md b/plans/per-layer-parameter-rules.md new file mode 100644 index 0000000..5a42367 --- /dev/null +++ b/plans/per-layer-parameter-rules.md @@ -0,0 +1,237 @@ +# Per-layer parameter rules for merge, extraction, and resize + +## 1. Summary and initial scope + +Add one multiline configurator node that supplies optional per-layer numeric overrides to existing nodes. Matching layers receive the explicitly assigned values; unassigned parameters and unmatched layers retain the main node’s settings. Existing exclusions, inclusion filters, discard rules, mismatch handling, and safety guards remain authoritative. + +The initial implementation covers these 39 nodes: + +| Family | Included variants | Count | +|---|---|---:| +| Standard mergers | Two- and three-input checkpoint, diffusion-model, text-encoder, LoRA, and embedding mergers | 10 | +| Extraction | Fixed, Ratio, Quantile, Knee, and Frobenius for LoRA, DoRA, learned DoRA, text-encoder LoRA, and text-encoder DoRA | 25 | +| LoRA resize | Fixed, Ratio, Frobenius, and Cumulative | 4 | + +Do not expand this implementation to CWB configuration, weighted multi-LoRA merging, LoRA application, Lodestone, optimizer settings, iteration counts, seeds, device selection, dtype selection, or new normalization behavior. These remain post-implementation considerations. + +No changes are made during planning. After approval and execution is enabled, the first implementation action is to save this entire approved plan verbatim to `plans/per-layer-parameter-rules.md`. Commit that planning document before editing implementation files. Commit the completed feature after verification. + +Preserve the pre-existing analysis/report edits currently outside the filter commit. + +## 2. Public interface, syntax, and bindings + +### Configurator and connection + +Implement `LayerParameterConfiguration`, displayed as **Layer Parameter Configuration**, in a new `nodes/layer_parameters.py` module. + +Its inputs are: + +| Input | Type | Default | Meaning | +|---|---|---|---| +| `rules` | Multiline string | Empty | One pattern-and-assignment rule per line | +| `glob_patterns` | Boolean | False | False uses regex substring search; True uses glob substring matching | + +Its outputs are: + +| Output | Type | Meaning | +|---|---|---| +| `layer_parameters` | `io.Custom("MODELUTILS_LAYER_PARAMETERS")` | Immutable, parsed rules | +| `documentation` | String | Syntax, bindings, supported nodes, fallback behavior, errors, and examples | + +This follows the existing `io.Custom(...)` configurator/socket mechanism already used by CWB. No frontend extension or dynamic socket-type changes are needed. + +Append an optional `layer_parameters` input to each supported receiver. Append `layer_parameters=None` to its execution interface and forward it by keyword. Preserve every existing input identifier, widget order, output, and default. + +An unconnected input or an empty rule set leaves the operation unchanged. + +### Rule grammar + +Accept the following forms: + +```text +(blocks\.4[589]\.attn\.qkv_proj) a:0.5 b:0.25 c:0.75 d:1.0 +(blocks\.4[589]\.attn\.qkv_proj) alpha:0.5, beta:0.25, gamma:0.75, delta:1.0 +(blocks\.4[589]\.attn\.qkv_proj) a:64, b:32, c:0.99, d:0.0 +(blocks\.4[589]\.attn\.qkv_proj) max_rank:128 min_rank:1 target:0.9 +``` + +Rules: + +- Require a parenthesized pattern followed by at least one `name:value` assignment. +- Accept whitespace, commas, or a mixture between assignments, including whitespace around colons. +- Preserve the pattern’s backslashes, internal whitespace, nested groups, and character classes. +- Accept signed decimal and scientific-notation numeric values. Reject nonfinite values. +- Ignore blank lines and whole-line comments whose first nonwhitespace character is `#`. Do not support inline comments. +- Reject empty patterns, missing values, trailing garbage, duplicate assignments, and invalid regex. +- Use `re.search` for regex matching. For glob matching, use case-sensitive `fnmatchcase` with surrounding wildcards to provide substring matching. +- Keep configurator pattern syntax independent of the main node’s existing exclusion-filter syntax. + +Short aliases and full parameter names may be mixed. Two aliases assigning the same parameter on one line are an error, even when their values agree. + +### Binding tables + +The shared binding definitions must generate both receiver tooltips and configurator documentation, preventing the two from drifting. + +#### Standard merge coefficients + +Bindings remain stable across calculation modes: + +| Short name | Full name | +|---|---| +| `a` | `alpha` | +| `b` | `beta` | +| `c` or `g` | `gamma` | +| `d` | `delta` | +| `e` | `epsilon` | +| `f` or `z` | `zeta` | + +Accept only coefficients actually used by the selected calculation mode: + +| Calculation mode | Supported coefficients | +|---|---| +| Weight-Sum; Train-Difference | alpha | +| Add-Difference; Power-Up (DARE) | alpha, beta | +| Comparative-Interpolation; Add-Dissimilarities; SVD LoRA Extraction; Enhanced Auto Interp; Weight-Sum Cutoff; Power-Up (DARE+TIES) | alpha, beta, gamma | +| Extract-Features; Enhanced Man Interp; Power-Up Enhanced (DARE) | alpha, beta, gamma, delta | +| Power-Up Enhanced (DARE+TIES) | beta, gamma, delta, epsilon, zeta | + +Reject assignments to ignored coefficients rather than pretend they affect the selected mode. Seeds remain main-node settings. + +#### Extraction + +Use the same bindings across LoRA, DoRA, learned DoRA, and both text-encoder extraction families: + +| Method | `a` | `b` | `c` | `d` | `e` | `f` | +|---|---|---|---|---|---|---| +| Fixed | linear_dim | conv_dim | clamp_quantile | min_diff | — | — | +| Ratio | linear_ratio | conv_ratio | clamp_quantile | min_diff | linear_max_rank | conv_max_rank | +| Quantile | linear_quantile | conv_quantile | clamp_quantile | min_diff | linear_max_rank | conv_max_rank | +| Frobenius | linear_target | conv_target | clamp_quantile | min_diff | linear_max_rank | conv_max_rank | +| Knee | linear_max_rank | conv_max_rank | clamp_quantile | min_diff | — | — | + +Knee method selection, probe offsets, SVD iterations, chunking controls, and learned-DoRA optimization settings remain on the receiving node. + +#### LoRA resize + +| Method | `a` | `b` | `c` | +|---|---|---|---| +| Fixed | new_rank | — | — | +| Ratio | max_rank | ratio | — | +| Frobenius | max_rank | min_rank | target | +| Cumulative | max_rank | target | — | + +### Value validation + +Inherit values from the receiving node—not hardcoded configurator defaults. + +For supplied overrides: + +- Extraction dimensions and rank caps are integral values in `[1, 16384]`; ratios are `[1, 100]`; quantile/Frobenius targets are `[0, 1]`; `clamp_quantile` is `[0.5, 1]`; `min_diff` is `[0, 1]`. +- Resize ranks are integral values in `[1, 3072]`; ratios are `[1, 100]`; targets are `[0.1, 1]`. +- Validate resolved Frobenius resize `min_rank <= max_rank`, including values inherited from the main node. +- Standard merge coefficients retain existing widget bounds. Where a coefficient represents a quantile or probability, validate `[0, 1]`; generic SVD rank coefficients must also be positive integral values. +- Do not silently truncate fractional ranks or repair invalid assignments. Retain existing mathematical rank caps imposed by tensor dimensions. + +## 3. Matching and implementation data flow + +### Matching contract + +For ordinary model merging and extraction, match the receiving operation’s target tensor names. + +For LoRA inputs, match the normalized logical layer name after the existing parsing and normalization path. Do not match factor or alpha suffixes independently. + +Use the established dotted naming produced by extraction and existing reference mapping. Preserve internal identifiers such as `qkv_proj`. Do not introduce underscore normalization as the new public convention, rewrite regex punctuation, or redesign model-format conversion. + +Retain the existing unresolved flattened-name fallback when no reference mapping exists. Documentation must distinguish this fallback from the normal dotted path. A dotted rule that cannot match the resulting normalized input names produces the ordinary unmatched-rule error; do not silently reinterpret it. + +A single rule matching multiple keys belonging to one logical LoRA layer counts as one match. That layer receives one parameter set shared by its factors. + +### Strict errors and fallback + +Validate all rules before opening the output writer or beginning tensor streaming: + +1. Parse syntax and compile patterns once in the configurator. +2. At each receiver, resolve aliases and reject unsupported parameters. +3. Use header/key information to identify target layers and their existing normalized names. +4. Detect matching rules per logical layer. +5. Reject multiple matching rule lines for any one layer—even if their assignments concern different parameters. +6. Reject any rule line matching no target layer. +7. Combine the single matching rule with main-node values and validate the resulting parameters. + +Errors identify the receiving node, configuration line number, pattern, and affected layer or parameter. Overlap errors include every conflicting line number. + +Determine name matches before existing exclusions so an explicitly excluded layer is not falsely reported as nonexistent. Overrides never reactivate an excluded, discarded, guarded, or otherwise unprocessable layer. + +### Shared implementation + +Keep parsing, immutable rule records, binding definitions, and matching/error handling in the new shared module. + +Store only lightweight per-layer assignments and resolved scalar parameters. Do not store tensors, mutate the shared configuration payload, or modify global/main-node parameter dictionaries between layers. + +Register the configurator through the existing extension node list. Do not add dependencies. + +### Standard merger integration + +In `MergerLogic.execute_merge`: + +- Prepare the rule-to-layer mapping alongside existing header-based work planning. +- For LoRA mode, use parsed logical groups to assign the same override to their factor keys; retain existing merge and alpha-normalization behavior. +- Immediately before `calc_mode_class.create_recipe`, construct a layer-local parameter dictionary containing the base settings plus that layer’s overrides. +- Keep tensor references and preloaded-source data local to the current work unit. +- Do not mutate `recipe_params` with overrides that could affect later layers. + +Do not change recipe formulas, calculation-mode selection, or source-loading behavior. + +### Extraction integration + +Thread the optional rules through all 25 node entry points into the four extraction backends. + +Prepare resolved parameters before streaming, then use them in each backend’s `_process_layer` before `min_diff` testing and SVD selection. + +Specific requirements: + +- Fixed-rank overrides update both the requested dimension and corresponding maximum-rank argument; otherwise the original cap would negate an increased rank. +- Adaptive overrides update the existing linear/conv method parameter and rank cap without altering method selection or probe settings. +- Normal execution, CPU retry, and chunked fallback use the same resolved layer values. +- Extend the existing chunked extraction helpers with a trailing `clamp_quantile` argument and pass the effective value through. These helpers currently omit it when calling `_svd_extract_linear`; text-encoder extraction shares the SVD helper. +- Perform configuration validation outside broad extraction exception handlers so a configuration error cannot become a silently skipped layer. + +Do not alter SVD, chunk recombination, learned optimization, or tensor-lifecycle algorithms. + +### Resize integration + +Attach resolved rank/target values to the existing logical-layer work units. + +When calling `_resize_lora_factors`, translate the public bindings into the existing arguments: + +- Fixed `new_rank` → `new_rank`. +- Adaptive `max_rank` → `new_rank`. +- Ratio/target → `dynamic_param`. +- Frobenius `min_rank` → `min_rank`. + +Leave `dynamic_method` on the main node. Preserve whole-layer factor handling, alpha behavior, passthrough data, low-bit handling, dtype selection, and CPU fallback. + +## 4. Verification and delivery + +Extend the repository-owned test selector with focused rule-parser and integration coverage. Map the new production module explicitly; do not rely solely on the broad repository-contract group. + +Required tests: + +- Both delimiter styles, mixed short/full names, scientific notation, nested regex groups, CRLF, blank lines, and comments. +- Invalid syntax, invalid regex, unsupported parameters, duplicate aliases, nonfinite values, fractional ranks, and invalid resolved rank bounds. +- Overlapping rules fail with layer and line numbers; unmatched rules fail before streaming or writing. +- Partial assignments inherit remaining main-node values. Unmatched layers retain main-node behavior. +- Existing include/exclude/discard and safety paths remain authoritative. +- LoRA matching uses normalized logical names, preserves internal underscores, and assigns one configuration to paired factors. +- All 39 receiver schemas append the optional socket without moving existing controls; every execution wrapper forwards it correctly. +- Standard merge outputs reflect different coefficients on different layers, without leaking overrides to later layers. +- Every extraction family uses the requested per-layer rank, clamp, and `min_diff`; cover adaptive methods, CPU retry, and forced chunk fallback. +- All four resize variants use their resolved rank/target values, including inherited `min_rank` validation. +- Absent or empty configuration preserves existing behavior. +- Configuration failures preserve any existing destination file. + +Use small deterministic tensors and existing I/O fixtures. Do not run model inference or add unrelated performance, architecture, or normalization test projects. + +Run focused groups during development, `--changed` after accumulated changes, and the required full gate before the feature commit. Run changed-file lint, compilation, and `git diff --check`; distinguish pre-existing lint findings from new ones. + +Deliver the saved approved plan, implementation, generated binding documentation/tooltips, regression tests, and a verified feature commit. Leave unrelated analysis/report edits untouched.