* feat(core): stream CSV records and add a configurable read limit
CSV reading used to build a full List[List[str]] of the raw file before
parsing, so cold loads kept both the raw rows and the final objects in
memory at once. Records now stream straight into the parsers.
- artists: _read() feeds _iter_csv_rows() into parse_artist_rows() instead
of _csv_rows(); a shared _iter_csv_rows() owns counting, early stop and
error containment for both read paths.
- characters: build_records() is rewritten as a single pass and accepts
Iterable. The previous Sequence signature iterated twice, so a generator
input silently dropped the first data row (6 rows -> 4 records, no error).
- characters: load_records()/load_index() read via the new
ArtistRepository.open_rows(), so the raw rows are no longer parked in
_rows_cache. load_rows() stays as the public compatibility entry.
- limits: normalize_max_rows()/resolve_max_rows() accept only None or a
positive integer. The effective limit is part of every cache key, and the
count is defined once for both paths: csv.reader records, header and
blank records included.
- exact-N no longer claims truncation: the streaming reader flags
"truncated" only when it actually reaches record N+1, with no extra read.
- nodes: both take an optional csv_max_rows (default 0 = unlimited),
appended last in optional and in the signature, marked advanced. ComfyUI
1.55.18 restores widgets_values positionally (NamedValuesRestore is off
by default), so appending at the end is what keeps old workflows intact.
Default behaviour is unchanged: same seeds produce byte-identical output
for both nodes.
* fix(core): tighten path boundaries and add administrator allowlists
Two boundary defects in ra_core/paths.py, plus an opt-in allowlist channel
for administrators who want to narrow the allowed roots.
- _resolve_unc_to_drive() used to map the admin-share form of ANY host to a
local drive: \\evil-host\D$\data\x.csv became D:\data\x.csv and was then
ALLOWED. The local path was what actually got opened, so the remote intent
was silently downgraded to a local read/write. Only localhost, 127.0.0.1
and [::1] are restored now; every other host stays UNC and is rejected by
the existing UNC branch.
- _is_within() compared os.path.commonpath() against the root without
normcase, so mixed-case nesting was judged outside. Under the existing
blacklist this was unreachable (forbidden_roots() only keeps existing
directories and realpath normalises their case), but an allowlist root that
does not exist yet was rejected. Both sides are normcased now.
- New administrator allowlist, read from ra_config.json in the node directory
(csv_allowed_roots / db_allowed_roots). No environment variable is read:
reading one would add a python_environment_manipulation finding to the
Registry scan, and the package is already flagged for the five existing
reads. The file is absent by default, which keeps the current blacklist
behaviour.
- The allowlist only narrows. Passing it still has to clear the UNC rejection
and the forbidden-root loop. It is checked after _canonicalize() and before
is_path_guard_disabled(), so "allowlist plus escape hatch" fails closed
instead of the escape hatch silently winning.
- Invalid configuration fails closed with an actionable message: bad JSON,
non-object top level, empty or missing lists, non-string roots, relative
roots and bare drive letters are all rejected.
- for_write is no longer a dead parameter: it selects db_allowed_roots over
csv_allowed_roots, so the two kinds stay separate.
- Node-directory data files are exempt: the bundled default CSV/DB are
trusted runtime resources, and users sometimes write the full default path
into custom_csv_path. Node source files are still blocked.
TOCTOU is unchanged and not claimed: check_db_path validates once and
sqlite3 reopens by path, and sqlite3 has no fd parameter.
CSV reading used to build a full List[List[str]] of the raw file before
parsing, so cold loads kept both the raw rows and the final objects in
memory at once. Records now stream straight into the parsers.
- artists: _read() feeds _iter_csv_rows() into parse_artist_rows() instead
of _csv_rows(); a shared _iter_csv_rows() owns counting, early stop and
error containment for both read paths.
- characters: build_records() is rewritten as a single pass and accepts
Iterable. The previous Sequence signature iterated twice, so a generator
input silently dropped the first data row (6 rows -> 4 records, no error).
- characters: load_records()/load_index() read via the new
ArtistRepository.open_rows(), so the raw rows are no longer parked in
_rows_cache. load_rows() stays as the public compatibility entry.
- limits: normalize_max_rows()/resolve_max_rows() accept only None or a
positive integer. The effective limit is part of every cache key, and the
count is defined once for both paths: csv.reader records, header and
blank records included.
- exact-N no longer claims truncation: the streaming reader flags
"truncated" only when it actually reaches record N+1, with no extra read.
- nodes: both take an optional csv_max_rows (default 0 = unlimited),
appended last in optional and in the signature, marked advanced. ComfyUI
1.55.18 restores widgets_values positionally (NamedValuesRestore is off
by default), so appending at the end is what keeps old workflows intact.
Default behaviour is unchanged: same seeds produce byte-identical output
for both nodes.
StarAsh042