The dropdown already offered the realtime option while no
asset backed it, so selecting it could only raise. The two
tests that pinned that gap are retargeted, not deleted.
The package installed its own handler, Formatter and level on a
module-level logger, then logged through that logger. ComfyUI core already
installs app.logger.ColoredFormatter on the ROOT logger, so a bare
logging.<level>() call whose message starts with "[ComfyUI-VibeVoice] " is
tagged and coloured for free. This removes the duplicate machinery rather
than extending it -- no vv_logging module, Logger subclass, LoggerAdapter,
vendored ANSI table, new handler, setLevel or propagate.
Three hazards that motivated deleting the setup block rather than moving it:
- propagate=False severed pytest's caplog; the tests only passed because
__init__.py's `if "pytest" in sys.modules` guard skipped the block.
Bare root calls propagate by default.
- logger.setLevel(INFO) on the package logger pinned every descendant, so
ComfyUI's --verbose never applied to this node. Deleting the block fixes it.
- The level must stay a literal at the call site; no env-var switch was
added. modules/diagnostics.py gates diagnostic CONTENT and is untouched.
Every call site (235 across 34 files) is now a bare root call with the
prefix. The vendored src/vibevoice tree used transformers.utils.logging,
which has no module-level info/warning/debug; those files now import stdlib
logging, so a new call site there would fail loudly instead of silently.
Level audit against ERROR=failure / WARNING=degradation / INFO=user-facing
progress / DEBUG=internals. Deleted as noise: the resample notice (44.1 kHz
reference audio is the normal case) and the two "Successfully loaded
external VibeVoice" confirmations, which duplicated the patcher's load line.
Promoted to WARNING: a mixed-naming GGUF, which is resolved by heuristic
majority vote and silently aliases the rest. Demoted to DEBUG: the four
SageAttention kernel-selection lines, model discovery, shard counts,
per-retry download attempts, attention-mode confirmation and the
save_pretrained notice. Kept at INFO: generation complete, transcription
results, model downloads and load starts.
Judgement calls recorded in docs/2026-10-01-vv-logging-cleanup-design.md.
The two I am least sure of: the ungated memory-census profile line was
demoted rather than gated (adding a gate would not be presentation-only),
and patcher.py's "Loading VibeVoice models for..." was kept at INFO against
the ask's example list because it is the only load-start line the package has.
48 new tests in tests/test_logging_idiom.py plus tests/test_audio_utils.py:
byte-exact ColoredFormatter rendering, no-leftover-machinery, AST prefix and
two-direction level policy, and the 44.1 kHz resample regression (the call
still fires with (44100, 24000); nothing is logged). 37 caplog.at_level
pins that named a module logger were stripped -- they only lowered a named
logger and left root at WARNING, so the record was discarded before capture.
Gate: 1924 passed, 30 skipped, 0 failed (1876 before this change). Twelve
deliberate mutations -- re-deleting a message, re-leveling, re-adding
setLevel and propagate=False, stripping a prefix, moving the resample out
of its branch -- were each caught by at least one test.
Not run: ComfyUI was never launched and no checkpoint was loaded.
The [vvcensus], [vvrss], [vvpull], [vvload], [vvdtype], 'Load diagnostics:'
and 'GGUF forward diagnostics:' lines exist to make a slow load or a slow
autoregressive step explainable. None of them produce audio, and all of
them default to ON, so a normal run prints several of them.
Gate them at the six choke points that already existed (report_census,
RssSampler.report, _install_vbar_observer, _log_conv_dtype_mismatch,
_log_load_diagnostics, log_gguf_forward_counters, plus the two emit sites
in generation.py and base_loader.py) rather than commenting out 18 call
sites: the formatting helpers stay importable and unconditional, so a
probe or a test can still ask for the string directly.
New modules/diagnostics.py owns the switch. VIBEVOICE_DIAGNOSTICS=1 turns
everything on; VIBEVOICE_RAM_CENSUS and VIBEVOICE_VBAR_OBSERVER keep
their existing per-family meaning. Production leaves all three unset.
Tests that assert the CONTENT of a line now set the env var explicitly,
and three tests pin the new silent-by-default contract (census, gguf
counters, vbar observer).
Live 2026-09-30, VibeVoice-7B-q8_0.gguf: host RAM 24 -> 41 GB with sustained
SSD reads. The census names it exactly:
pre-h2d params=total=10.31GB private=8.04GB offhost=2.27GB
GGUFLinear/params=8.04GB
8.04 GB of the model's parameters are private host memory. The copy is in
GGUFTensor.from_reader_tensor (modules/gguf_quant.py), which materialises the
reader's mmap view into a private tensor before anything can place it. The gguf
package has no aimdo file slice, so place_tensor_on_device correctly falls
back -- but the copy has already happened by then.
It is also why generation is slow: GGUFLinear.forward takes _forward_streamed
whenever weight.device != x.device, so every forward was paging the blocks.
The blocks ARE the model's storage -- q8_0 is 8.04 GB, and dequantizing to
bf16 at load would be ~16 GB and would not fit the card, defeating the
checkpoint. So they go where the rest of the model goes, at the same byte cost:
module.set_raw_weight(place_tensor_on_device(raw, target_device))
dequantize_blocks is device-agnostic (reshape -> kernel -> to(dtype)), so the
forward now dequantizes in place on the GPU. The no-device path is unchanged.
Side effect: with the blocks genuinely in VRAM, _adopt_resident_weights
(ddc3e84) now counts them, so core's "loaded completely; 10554.74 MB" line
becomes honest for GGUF too.
2 new tests in tests/test_gguf_flat_pool.py. GGUF suite: 94 passed / 1
pre-existing failure (error wording changed by the external agent's rewrite).
NOT yet confirmed on a live load -- per the standing rule the agent does not
load huge models. Analysis and handoff:
.dev/docs/2026-09-30-vibevoice-load-investigation/12-gguf-resident-blocks.md
Loading problem closed for every model type. Live, 2026-09-30 (RTX 4070 Ti
SUPER 16GB), 1.5B bf16 and 7B fp8 both load straight into VRAM with
+0.69 GB / +1.45 GB of machine RAM and no sustained SSD traffic.
What was wrong
--------------
The per-tensor placement used `Tensor.to(cuda)` on a memory-mapped view. That
is a HOST-side read: the copy engine faults every page in through the CPU, at
0.55 GB/s and +2.18 GB of machine RAM per 2 GB (report 2026-09-30 section F6).
Every route paid it, on every model type.
The 7B fp8 load had looked SSD-free only because pass 1's `safe_open` was
committing the whole file as private memory first, so those host-side copies
were served from RAM. The 19.7 -> 28 -> 20 GB spike was the receipt for that.
What changed
------------
* base_loader.place_tensor_on_device(): one placement helper for every route.
With an aimdo mapping (main.py sets aimdo_enabled at startup) it uses core's
own `read_tensor_file_slice_into` to DMA the file byte range straight into a
preallocated CUDA tensor -- the same primitive core already uses to page
weights into VRAM, measured at 2.6 GB/s with the page cache left clean.
Falls back to `.to()` when core declines or when the aimdo native library
raises; a one-time `[vvload]` log line confirms the DMA is live.
* One streaming assign for all routes. `_stream_apply_dense` gained
`target_device`, and a new `_stream_apply_dense_safetensors` is the dense
twin of `_stream_apply_safetensors`. The TTS and ASR dense branches no longer
build a whole CPU state dict and call `model.to(cuda)`; GGUF and the internal
official-model loader joined the same path.
* `read_safetensors_tensors_by_name()`: pass-1 dequant scales are read from the
byte ranges the safetensors header already records, instead of `safe_open`.
That drops a genuine ~1x-file private commit.
* select_patcher_class() always returns the standard ModelPatcher, so core can
full-load instead of trapping weights in VBAR for an autoregressive model.
Tests
-----
5 new tests for DMA routing and both fallbacks; load-path tests updated to the
new read seam. Targeted run: 373 passed / 34 failed, all 34 pre-existing and
asserting the ModelPatcherDynamic/vbar architecture this change removes
(26 in test_dynamic_patcher_selection.py). Full suite not run, per user rule.
Loading filled host RAM 20->40GB while streaming from SSD and never released
it; inference then re-served the whole model per AR step. Teardown against
core's aimdo/DynamicVRAM machinery, measured on the real 9.47GB 7B fp8 and
5.16GB 1.5B bf16 checkpoints, found four loader defects plus one that was
never ours to fix.
Defects fixed (each with before/after numbers):
* Quant pass 1 pre-read each dequant-at-load layer's scale with
safetensors.safe_open + get_tensor. On Windows one such call commits ~1x
file size as private, untouched memory, pinned for the lifetime of the
returned tensor: +9,050MB on the 7B fp8, invisible to both the working-set
counter and the storage census. Now maps the file once through core's
aimdo arm and clones only the tiny scales. Whole load: 1.51x -> 0.13x file
size; retained after free 11.3GB -> 96MB.
* Quant families were excluded from core's dynamic patcher by an inherited
"quant streams natively" stop condition. Because a legacy patcher has
nowhere to page from, those routes cloned every tensor into host RAM and
fully H2D'd it. select_patcher_class now follows core's availability rule
for every family, with gguf_block still excluded for a measured reason
(GGUFTensor.from_reader_tensor clones the reader's view).
* replace_linears_for_quant built its resident modules outside the meta
context: 8.08GB of never-written host allocation per 7B fp8 load.
* Dormant weight_function/bias_function double application: core already
applies them inside cast_bias_weight (ops.py:431-438).
Exonerated with numbers, not assumed: the dense read path is core's own
(13.5MB private for a 5.16GB file), and core's file->VRAM paging read is
cache-clean (+0.13GB per 2GB). Only host-side view reads reproduce the
reported 1:1 RAM at 0.55GB/s signature.
Inference: reproduced headlessly that when the tree does not fit in the
VRAM that is actually free, every forward re-reads every weight from the
checkpoint file (3072/3072 file reads, 21ms -> 125ms per step). Our wrappers
are protocol-correct; residency is stable at 100% with headroom. [vvpull]
now reports resident vs reread with bytes and free VRAM so one live line
settles it, behind VIBEVOICE_VBAR_OBSERVER=0.
Not changed after being tried and reverted: deriving fast_disk from the
checkpoint. It measured as a no-op on the dev host but converted RAM-speed
pinned re-reads into disk-speed reads live, which made loading dramatically
slower. Reverted in full; see the report for the probe-design lesson.
Adds host-RAM instrumentation ([vvrss] with machine-level start/end, [vvcensus]
storage census, tts-generate bracket), four standalone probes, and ~4000
lines of tests pinning the invariants above.
Loading
- GGUF install is now two passes: a metadata pass that decides each
tensor's disposition, then an install pass that assigns residents in
place and streams dense tensors. The whole checkpoint is no longer
buffered in a dict alongside the model being built.
- dequantize_reader_tensor takes a target dtype, so dequant-at-load
writes straight into the destination parameter and the fp32
intermediate is never allocated.
- A bundle whose heavy fields were released is rebuilt from its recorded
source_path instead of failing the consumer.
- Host memory is released after install.
Numerics
- Q8_0 dequant computes in fp32 so the result is rounded once, at the
final cast. The activation-dtype path was reverted: it rounded twice
and moved stored weights.
- Removed a redundant weight-sized copy from the dequant kernel.
Bitwise-identical, ~1.16x.
- Precision gates are bitwise rather than tolerance-based.
Docs and tests
- README condensed; changelog moved to CHANGELOG.md.
- Third-party project references removed from source comments.
- Tests no longer assert README prose; the e2e smoke contract follows
the developer script to its new location and skips when absent.
- Version guard reads CHANGELOG.md.
Garbled, parameter-insensitive speech came from a randomized EOS head:
5.3 re-runs _initialize_weights over acoustic_connector and
tts_eos_classifier because the vendored _init_weights override had no
_is_hf_initialized guard. Guard it; only checkpoint-absent weights are
initialized now.
- MockCacheLayer exposes both the 4.x and 5.x cache APIs, so the
prefilled voice prompt is visible to 5.3's mask builder.
- _ensure_cache_has_layers covers the container: offload/prefetch,
batch ops, crop, and a copyable lazy prefetch stream.
- max_new_tokens is a combined text+speech budget, not latents.
- cfg_scale floor of 1.5 for the realtime family.
- voice presets resolve against every registered TTS root.
- sage excluded from the realtime path: it ignores the attention mask.
- bound transformers to >=5.3.0,<5.4, the measured line.
Loading VibeVoice-*-fp8_e4m3.safetensors materialized the full state
dict and dequantized all ~380 fp8 layers to bf16 in RAM: 18->43 GB
spike, Pin-error flood, ~4.5 GB partial offload on 16 GB cards.
- FP8Linear keeps fp8 weight + scalar fp32 scale resident in VRAM
(~1 byte/weight); forward dequantizes per-tensor via comfy-kitchen
into the activation dtype. Missing kitchen backend falls back to
dequant-at-load.
- Quantized safetensors load streams per-tensor (never a full
in-memory state dict); dense files keep the batch path.
- Non-Linear fp8 targets (real exports quantize embed_tokens) demote
to dequant-at-load; ConvRot non-Linear targets keep the hard fail.
- Diffusion head resolves the timestep-mlp input dtype via the
module's compute_dtype, never the fp8 storage dtype — casting
activations to weight.dtype fed fp8 into the dequant kernel and
crashed generate() (NoCapableBackendError).
GPU-gated on 1.5B + 7B fp8: full VRAM fit, no partial offload, no
pin flood, audio generates.
transformers 5.x deprecated PretrainedConfig.torch_dtype; both getter and setter log a warning on every load. Add set_config_dtype/get_config_dtype helpers (configuration_vibevoice.py + modules/dtype_utils.py) that use config.dtype on v5 and torch_dtype on 4.x, and route the loader/external_loader writes plus the 5 vendored modeling reader blocks through them. +17 regression tests incl. no-warning assertions on the real load path.
Config auto-detection:
- config_detect: read the checkpoint embedding shape as an architecture
fingerprint (header-only for safetensors, reuses the open GGUF reader);
7B=[152064,3584], 1.5B=[151936,1536]; orientation-agnostic for
shape-reversed GGUF files
- Auto-detect is the new default config_name; resolves the family before
any heavy load, or fails fast with an actionable error (.bin/.pt and
unknown families cannot be fingerprinted)
- an explicit config_name that contradicts the weights self-corrects to
the detected family with one WARNING (reconcile_config)
- loader: friendly shape pre-check in _apply_state_dict names the
offending tensors and hints at config_name instead of torch's raw
size-mismatch RuntimeError
Dropdown dedup:
- VibeVoice-Large removed from config_name options (duplicate of 7B);
kept as a legacy alias so saved workflows still load (normalize at node
+ loader entry; validate_inputs(**kwargs) override makes core skip its
combo-membership check)
- node resolves Auto-detect before computing the cache identity so the
request key matches the consumer's bundle-derived key (no per-run churn)
Console noise:
- demote ~30 internal INFO logs to DEBUG across loaders/patcher/registry
- drop two stray tie_weights prints; tied lm_head.weight no longer warned
as missing (expected under tie_word_embeddings)
Tests: 1060 passed / 5 pre-existing failures / 4 skipped
- patcher: complete the H2D transfer for modules ComfyUI's lowvram
arbiter leaves on CPU during partial loads (plain transformers trees
have no comfy.ops streaming hooks; crashed at forward with
'Input type CUDABFloat16Type and weight type CPUBFloat16Type' on
models larger than free VRAM, e.g. 7B)
- finish Large->7B model_info rename: tokenizer repo selection now
matches '7B' (previously fell back to the 1.5B tokenizer), 7B added
to external-loader config options (shares Large packaged defaults),
stale test assertions updated, tests for the removed external-model
example workflow deleted
- loader: packaged tokenizer.json loads directly from the node folder
instead of being copied into user model directories; HF download
stays as last resort; model dirs stay untouched
Fix A: generate_audio() now raises a clear RuntimeError instead of AttributeError when speech_outputs is None/empty (corrupt or over-quantized weights emit no speech tokens). Fix B: warn_if_lowbit_quantization() detects naive int casts (raw I8 tensors with no scale metadata) in safetensors and sub-4-bit I-quants in GGUF at load time. Root-caused via diagnostics: iq3_xxs garbled output = over-quantization (relerr~0.22/cosine~0.978 vs original, controls cosine=1.0); int8 crash = naive cast + unguarded None. 15 new tests; full regression 725p/5f/4s (5f pre-existing).
ComfyUI's get_filename_list('diffusion_models') filters by supported_pt_extensions which excludes .gguf, so GGUF files never appeared in the dropdown. Also, comfy.utils.load_torch_file routes .gguf to torch.load which fails.
- modules/external_loader.py: _load_gguf_state_dict() parses GGUF via the gguf package (GGUFReader + dequantize, writable copy for torch); _load_weight_state_dict() dispatches by extension; both TTS and ASR load branches route through it
- nodes/external_loader_node.py: list_external_model_files() merges diffusion_models list with .gguf files scanned from the same folders plus the ComfyUI-GGUF unet_gguf folder; resolve_weight_path() falls back diffusion_models -> unet_gguf
- README: GGUF support note + v2.2.1 changelog
Tests: new tests/test_gguf_loading.py (19 tests, real GGUF round-trip). Full suite: 706 passed, 5 pre-existing failures, 4 skipped.
WildAi