olivv-cs-ComfyUI-FunPack/docs/module-reference.md

Module reference

Facts for module authors, in tables. For the how and why, read Writing a module. Everything here is read from core/contract.py, core/schema.py, core/registry.py and the shipped modules. If this page and the code ever disagree, the code wins: please fix the page.

Announcement constants

Read by core/registry.py from the module's __init__.py (the FIELDS table). A constant the registry does not know is ignored.

Constant	Python type	Default	Rule
ID	str	none	Required, non-empty. Unique across modules (a second claimant is left out and said). Keep it free of .: a fault is traced back to a module from its hook key funui.<id>.
TITLE	str	none	Required, non-empty.
MOUNT	str	""	Required and non-empty when the module has SETTINGS.
SETTINGS	dict	{}	{key: setting}. Must be a dict.
STAGE	str	"sampling"	One of load, conditioning, latent, guidance, sampling, post.
CATEGORY	str	""	"" or one of continuity, guidance, conditioning, sampling, post, system. Empty shows under "Other".
STATUS	str	"experimental"	proven or experimental.
REQUIRES	list[str]	[]	Model traits that must all be present.
USES	list[str]	[]	Capabilities this module asks other modules for.
USES_WHEN	dict	{}	Same shape as when: own settings under which it asks. Keys must be its own settings.
ROLES	list[str]	[]	Pipeline role at names it acts through. Inert in a pipeline where none is filled.
AFTER, BEFORE	list[str]	[]	Module ids. Order within a stage.
HOOKS	list[str]	[]	Hook points a sampler must offer for a sampler_modifier.
NODES	list[type]	[]	Classes with define_schema and execute (V3 io.ComfyNode).
TRAITS	callable	none	traits(model) -> iterable[str].
PROVIDES	dict[str, callable]	{}	Capability name to function. Names must be non-empty strings.

CONTRACT_VERSION in core/contract.py is 1. It is sent to the browser as contract in /fun/api/modules. Core does not read a version from modules, so there is nothing for a module to declare.

Setting keys

Key	Applies to	Notes
type	all	bool, int, float, enum, text, multiline, path, color.
default	all	Required. bool is a bool; int is an int (not a float); float/int are finite and inside min..max; enum is one of the options; color is #rgb, #rrggbb or #rrggbbaa; the rest are strings.
min, max	int, float	Finite numbers, min <= max. A value outside is refused at run time.
step	int, float	Positive.
options	enum	Non-empty list of {"value": scalar, "label": str}. No repeated values.
label	all	String. Unique within the module (test).
hint	all	String. Required by test, one plain sentence.
unit	numbers	String. Shown after the hint.
placeholder	text types	String. Used by the panel renderer, not by the Engine page.
ui	all	A renderer allowed for the type (next table).
when	all	{sibling_key: value or [values]}. Row shown only while every condition holds. Sibling must exist; an enum value must be one of its options; no self-reference.

The panel renderer also reads precision, presets (for macroSlider) and an option's icon and hint. The schema does not check them.

Renderers

Allowed ui per type (the first is the default for composer/panel.js):

Type	Allowed ui
bool	checkboxRow, toggle
int, float	number, slider, stepper, macroSlider
enum	select, segmented, radioGroup, filterList, wheel
text	input, search
multiline	textarea, autoTextarea
path	filterList
color	swatch

What Settings > Engine and the Sampler window draw (features/settings/field.js), whatever ui says:

Type	Drawn as
bool	toggle with label and hint
int, float	slider if ui is absent or slider and both min and max are set, else a number box
enum	select
color	swatch
multiline	text area (3 rows)
text, path	text box (commits on blur or Enter)

Stages and order

STAGE order is the list above. Within a stage modules run in id order unless AFTER/BEFORE say otherwise. Shipped modules today: load (loaders, models, pipeline, taste), conditioning, latent, guidance (taste-steered techniques), sampling, post.

Categories

Used only to group settings in Settings > Engine and to pick what the Sampler window shows (sampling). Order on the page: Continuity, Guidance, Conditioning, Sampling, Post, System, Other.

Pipeline roles

A slot in a pipeline lists roles: {"at": ..., "input": ..., "label": ..., "drives": ...}. at is where the input appears in the app. Names in use:

at	Shown as
generation.prompt	the prompt
project.negative	the project's negative prompt
project.video	width, height, length, frame rate (drives: frames or fps)
generation.sampling	steps, sampler, scheduler (Sampler window)
generation.seed	the seed
generation.model	the model choice
assets.source_image	the start picture (what continuity needs)

A role naming a place the app does not offer is simply not shown.

Traits

Read off any loaded model by core (traits.universal):

Trait	True when
temporal_latent, spatial_latent, waveform_latent	the latent has 3, 2 or 1 axes
temporal_compression	the latent format compresses time
predict_<type>	the model type, lowercased (predict_eps, predict_flow, ...)

Announced by model modules: audio_stream (H3, LTX), dit_block_hooks (H3: honours block replacement and routes attention through the override), adaln_modalities (H3 with the AdaLN projection), ltx_av (LTX audio+video). Add new ones in the model's own module. Modality is deliberately not a trait.

Capabilities

A capability is a name in PROVIDES. Core defines none: the contract is between the provider and the askers. All of these are in use today. None as an answer from a model-level provider means "not my model"; a raise means "it was mine and it broke".

Plugging into a run

Name	Provided by	Signature	Called by
modifier	most sampling and guidance modules	install(patcher, values, key) -> str or None	FunUI Load Modifiers
sampler_modifier	none yet	make(values, accepts) -> object with active(ctx), process(ctx, latent) or None	FunUI Sampler via core/chain.py
settings_sink	sampling_modifiers	() -> {"node": node id, "input": input name}	the manifest; the app writes settings there
cache_key	phrase_probe	() -> value that changes with its switch	FunUI Load Modifiers cache key
guards_off	full_control	(settings) -> bool	modifier nodes and the sampler
routes	several	routes(table, base, web)	core/routes.py, base /fun/api/m/<id>

Pipelines

Name	Provided by	Signature
default_pipeline	pipeline	() -> list of slots
pipeline_presets	each model module	() -> [{"id", "title", "slots"}]
optional_nodes	H3, LTX	() -> {node class: {"pass": {output index: input name}}} slots that may be skipped when their module is absent
upgrade_slots	output_save	(slots, outputs) -> slots migrates old saved pipelines

Knowing a model

Name	Signature	Notes
detect	(keys) -> bool	keys are tensor names from a safetensors header. Never load to answer.
probe_traits	(keys) -> list[str]	traits knowable from the header alone
empty_latent	(model, width, height, length, batch_size=1) -> {"samples": ...} or None	claim by identity
decode	(latent, model=None, vae=None, audio_vae=None, tile_size=0) -> result or None	claim by identity
video_stream	(x, named) -> (video, rebuild) or None	the picture inside a packed latent; core/streams.video_of asks
text_rows	(model_kwargs, length) -> bool mask or None	which conditioning rows are the prompt's words
text_tokenizer	(clip) -> tokenizer or None	a tokenizer that returns offsets
prompt_rows	(meta, cond_len, prompt_tokens) -> (start, end) or None
target_rows	(mod_segments, seq_len, device, stream) -> bool mask or None	stream is "video" or "audio"
query_rotation	(rope_freqs, seq_len) -> rotate(x, rows, inverse=False) or None
anchor_pin	(kwargs, transform) -> kwargs or None
rescale_conditioning	(conditioning, height, width) -> (conditioning, count) or None
latent_upscaler	(state_dict) -> object with funui_latent_upscale(x, scale) or None
training_adapt	(model) -> {name, step, video, batch_limit, fps, frames, pack, shims, silence} or None	what LoRA training must know about a model whose latent is not a plain tensor (H3 packs picture and sound). shims() is a context manager that applies the repairs only for the length of a run

Learning

Name	Provided by	Signature
taste_store	taste	(patcher) -> handle with collect, rows, capture, direction, directions, describe; None without a key
taste_kind	taste	(key, kind) -> Kind for one kind of one key
taste_latest_key	taste	() -> the key the most recent capture went to, or None
on_rating	taste calls every module that provides it	(prompt_id, rating, axis) -> None

An add-on module: LoRA training

modules/training/lora/ is announced as training_lora and switched on in Settings > Modules (the lora_training add-on, which asks for onnxruntime, needed only by the automatic tagger). It is a node-and-routes module with no settings of its own: the window passes every value to the node.

File	What it holds
store.py	Training sets as JSON under training/sets/. An item points at a Bin file or an output file; the path is resolved at run time, inside its root.
rated.py	Finds rated results in the saved projects and maps a rating to liked, disliked, bad_composition or bad_image.
captions.py	Tag text helpers, shared in spirit with app/ui/features/training/captions.js.
taggers.py	WD tagger models (ONNX) downloaded from Hugging Face on request, or any repo with model.onnx and selected_tags.csv; one tagging job at a time.
media_io.py, frames.py	Pictures and clips as tensors, fitted to a size bucket.
plan.py	Builds the graph a run is queued as: the pipeline's own loaders (ids model, clip, vae, audio_vae, or the first of each kind) feed FunUITrainLora.
train.py, nodes.py	The run, on ComfyUI's own Train LoRA node. Latents and conditioning are built from the set; each step's loss goes to jobs.py; Stop and ComfyUI's interrupt end it at the next step.
jobs.py	The run in progress and finished run records under training/runs/.

"Learn what I like" trains on liked results and your own media; "learn what to avoid" trains on the disliked ones, for a LoRA applied at negative strength. This is not VADER (reward gradients), which stays parked.

The diffusion model loader sets cached_patcher_init for a plain file, because the trainer clones the model from it. An H3 run uses the repairs in modules/models/minimax_h3/training.py. UNTESTED on real weights or hardware: proven only on a tiny H3 and a stand-in text encoder and VAEs, on CPU.

Helpers a module may import from _core

from ..._core import ... offers exactly these: addons, chain, config, contract, control, dit_hooks, graph, input_steer, log, media, patching, projects, rated_dial, registry, relations, run, schema, shortcuts, streams, temp_files, traits, vae_tiles. Adding one means editing modules/_core.py in both of its branches.

Most used:

Helper	For
log	info, alert, warning, critical, once, broke, failed
registry	registry.current().ask(...), .providers(...), .specs
patching	tag, strip, guard, Dropped
dit_hooks	add_block_hook, add_attention_override, current_step, late_half, last_step, probing, weak_branch, parse_blocks, block_count, publish
streams	video_of, av_video_stream, model_args, with_options
traits	has_block, traits_of
control	Unavailable

Served paths

Path	What
/fun/	the app (index.html)
/fun/api/modules	the manifest (?traits=a,b narrows to a model)
/fun/api/m/<id>/...	a module's own routes
/fun/app/...	the app. Allowed types: .js .css .html .woff2 .svg.

Diffusion Model Loader: speed settings in detail

The in-app hints are short; the details are here.

• mxfp8: MiniMax H3 only. Stores attention and MLP weights of the middle blocks as MXFP8 (one scale per 32 weights); the first 2 and last 3 blocks stay bf16. Blackwell GPUs multiply it at about twice the bf16 rate. Needs torch 2.10+ with CUDA 13 and a current comfy-kitchen. A LoRA applied to these layers slows them back down.
• int8_convrot: MiniMax H3 from a bf16/fp16 file. Weights are rotated, then stored as int8 (about half the memory). Speed-up is lost on layers a LoRA touches or when compute dtype is forced.
• fp16_accumulation: sets torch's fp16 matmul accumulation flag; needs a torch build that has it.
• sla / sla_method: block-sparse attention for H3, run alongside the chosen attention backend. sla keeps a fixed share (sla_sparsity, 0.90 validated, break-even near 0.60), sol-attn uses a per-block threshold (sla_tau), vsa works on video cubes and needs a model trained for it, veda uses a learned predictor file from models/veda (Veda, arXiv 2605.30325). sla_sparsity_start ramps the share up over the steps (arXiv 2610.21129).
• sla_block_size / sla_protect_audio: H3 packs audio at 80 rows per second, so large blocks force seconds of speech through one pattern; protecting audio keeps words and sound at full attention.