'Loaded fine' and 'runs fast' are different claims. Every NVFP4 load now
prints a diagnostics block that settles which one you have:
- comfy_kitchen backend census (the big one: torch built for CUDA < 13
disables ck's CUDA backend, so quantized matmuls silently fall through
to the pure-PyTorch eager backend - correct output, ZERO speedup, with
an explicit warning line when detected)
- torch/cuda build + device capability
- loaded-model layer census: nvfp4 layers vs layers flagged
full-precision emulation (pre-Blackwell GPUs)
- a timed 4096x4096 GEMM probe (incl. dynamic input quantization):
bf16 vs nvfp4 wall time with a FAST PATH ACTIVE / NO SPEEDUP verdict
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three live-run findings from the rental:
1. Block steering said "profile not ready (needs 2+ rated runs)" with 16
paired runs. The v1 liked/disliked-EMA attribution had no contrast when
ratings cluster on one pole (pole-vs-all-runs-mean cancels to zero) and
dropped rewards in (0, 0.3) entirely. Attribution is now the per-block
COVARIANCE between reward and activity over a rolling 32-run history:
every admissible reward contributes with its actual value, contrast
comes from reward variance, and the sampler log now states the real
blocker ("all rated alike - rate both good AND bad runs") instead of a
misleading run count. v1 profiles are discarded on first pairing.
2. output_guidance stayed "not ready (needs 10+)" forever on LTX-AV: the
end-of-run x0 snapshot was gated on output samples being a plain
torch.Tensor, but AV chain output is a NESTED tensor (video+audio) - so
the output value function never received a single sample. The snapshot
now extracts the video stream from nested outputs.
3. FunPackNVFP4ModelLoader hit a size-mismatch wall (9-vs-6 ada params,
connector dims): it loaded the state dict without the safetensors
METADATA that carries the LTX-AV architecture config, so detection
built the wrong model variant. Metadata now flows through, mirroring
comfy.sd.load_diffusion_model.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FunPackNVFP4ModelLoader quantizes an existing bf16/fp16 checkpoint to NVFP4
at load time as a pure state-dict transform: selected 2D Linears are
re-serialized in ComfyUI core's quantized-checkpoint format (packed fp4
weights + fp8-e4m3 block scales + fp32 tensor scale + per-layer comfy_quant
metadata) and handed to the stock loading path. Core's MixedPrecisionOps
then does everything: real FP4 tensor-core GEMMs on sm_100+ (sm_120 incl.),
dynamic activation scales (no calibration pass), per-layer LoRA fallback,
and dequant emulation on older GPUs. No custom forward, no monkeypatching.
Zone-aware precision map for the joint AV transformer: default scope
quantizes only video-branch Linears (attn1/attn2/ff) of the middle blocks;
the audio branch, the audio<->video cross-modal bridges, the first/last N
blocks, and all non-matmul weights stay full precision. Scopes widen to
+cross-modal, all blocks, and all 2D layers for experimentation.
Tests drive the REAL comfy.quant_ops/comfy.ops via comfy_kitchen's eager
backend on CPU: scope selection, serialized-format contract, prefix
handling, refusal of pre-quantized checkpoints, and a load+forward
roundtrip through MixedPrecisionOps on both the NVFP4 and emulation paths.
The Movie Editor Models modal picks the node up automatically as a
Unet/Diffusion Model candidate (object_info-driven).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
