Author SHA1 Message Date
jetthuangai 51822fe912 Bump version 1.8.0 2026-10-18 18:17:09 +07:00
jetthuangai ee93e7133c feat: masks from one SAM3 Segment node, faces and clothes parsed close up
Vision wave A (1.8.0):
- SAM3 Segment takes a mask formula in concepts (commas/new lines list,
  + - & ~ parentheses, words in a row are one label, background keyword),
  translates common Vietnamese words, adds a report output and an optional
  on-node preview (mask bright, instances labelled, report below), and is an
  output node. Masks of things running off the picture reach its edge.
- Segs Compose shares the formula language; Segs Select reads Vietnamese.
- Portrait Segment: whole-frame parse refined per face (YuNet, MIT, pinned,
  two sizes plus a second look with a margin); faces confirmed by the parser;
  wide crop for hair/neck, tight crop for features inside an ellipse.
- Garment Segment: whole-frame parse refined per group of people (crops of
  the model's shape, grown when people run past an edge, merged by
  confidence, only parts anchored on people already found).
- Parsers work on at most 4096 px and pre-shrink big inputs.
- Docs EN/VI, changelog, notices (YuNet), node card and examples.
2026-10-18 18:16:56 +07:00
jetthuangai f269c16ca5 Bump version 1.7.1 2026-10-18 12:33:12 +07:00
jetthuangai c41cbc2dd0 fix: SAM3 Segment fails when another pack's bundled sam3 is imported first
ComfyUI-RMBG puts its vendored copy of sam3 ahead of the installed package
on sys.path, so "sam3" resolves to that copy. Its build_sam3_image_model
looks for the tokenizer vocabulary one folder too high when bpe_path is not
given (RMBG always passes it), which raised FileNotFoundError on
models/assets/bpe_simple_vocab_16e6.txt.gz.

The vocabulary is now always found inside the sam3 package that was actually
imported (assets/ next to it, or one level up) and passed to the builder;
a clear error names the package folder when it is missing. Verified with
RMBG's copy first on sys.path (balanced and fast) and with Meta's package.
2026-10-18 12:33:06 +07:00
jetthuangai 3c053e09df Bump version 1.7.0 2026-10-18 12:10:29 +07:00
jetthuangai 5a617560a3 docs: SAM 3 and NH_SEGS nodes in the EN/VI guides
- vision guide (EN/VI): requirements and install, how the two passes work
  and their limits, the NH_SEGS type, the six nodes, three workflows and
  troubleshooting
- READMEs (75 nodes), CHANGELOG 1.7.0, THIRD_PARTY_NOTICES (SAM License,
  user-installed sam3 and checkpoint)
- six node cards, sam3-segment.jpg example from the public-domain astronaut
  photo; build_examples.py builds it with --gpu
2026-10-18 12:10:18 +07:00
jetthuangai 5389e93994 feat: SAM 3 segmentation and NH_SEGS instance nodes (Vision v2 P0)
Six new Vision nodes, 69 -> 75:
- SAM3 Segment: every instance of the concepts you name, with Meta's SAM 3.
  The whole image is encoded once for all concepts (pass 1); in balanced mode
  each instance is segmented again on a full-resolution close-up with the
  box-prompt decoder (pass 2), so small and medium objects keep crisp edges
  on 4K-8K photos. A close-up that does not show the pass-1 instance is
  dropped (tolerant check that keeps thin poles and cables).
- Segs Select, Segs Compose (label formulas: | - & ~ all image none),
  Segs To Masks, Segs To Impact SEGS, Segs Preview.

NH_SEGS keeps per-instance masks only inside each box, so dozens of
instances on an 8K image stay small; full-size masks are built only on
output (with a RAM guard in Segs To Masks).

Self-contained: no other custom node pack is used. Meta's sam3 package is
imported lazily with an API check (install hint pinned to the tested
commit), TF32 switches changed by its import are restored, and the gated
sam3.pt checkpoint in models/sam3 is checked before loading and never
downloaded. CUDA only. Weights and sam3's GPU-side tables go back to RAM
after each run unless keep_model_loaded.
2026-10-18 12:10:18 +07:00
jetthuangai f4843d30d3 docs: bilingual EN/VI guide, node cards, examples and new README
- README rewritten: banner, highlights with real outputs, install incl.
  ONNX Runtime and extra_model_paths keys, quick start, full catalog,
  licence summary, PDF guide links
- docs/en and docs/vi: guide home + 13 pages covering all 69 nodes (card,
  inputs/outputs from the live definitions, behaviour, recipes, tips)
- docs/assets: 69 node cards generated from INPUT_TYPES, 14 example images
  and a banner rendered from real node outputs on public-domain/CC0 photos
- docs/_tools: card, example, spec and PDF builders (reportlab) so the docs
  can be regenerated from the code
2026-10-18 09:17:48 +07:00
38 changed files with 5477 additions and 77 deletions
+23 -9
View File
@@ -11,10 +11,11 @@
# NH-Nodes for ComfyUI
**69 production-focused nodes** for ComfyUI: load and save image sets with their metadata, tile and composite large
**75 production-focused nodes** for ComfyUI: load and save image sets with their metadata, tile and composite large
images, pick the right resolution for current models and put results back into the original frame, refine masks,
segment garments and faces, build VTON masks, route workflows with lazy switches, generate prompts, and test many
LoRAs or models from one node.
segment garments and faces (also in half-body and group photos), get the mask of anything you name with SAM 3 from a
short formula such as `person - bag` or `background`, build VTON masks, route workflows with lazy switches, generate
prompts, and test many LoRAs or models from one node.
Every node is documented with a visual card, its inputs and outputs, and examples — in
[English](docs/en/README.md) and [Vietnamese](docs/vi/README.md).
@@ -32,8 +33,10 @@ Every node is documented with a visual card, its inputs and outputs, and example
</tr>
<tr>
<td><a href="docs/en/vision.md"><img src="docs/assets/examples/garment-segment.jpg" alt="Garment segmentation"></a><br>
<b>Segmentation</b> — garment and face-part masks from SegFormer models, plus VTON garment masks with pose and
parsing.</td>
<b>Segmentation</b> — name what you want (people, windows, the sky) or write a formula such as
<code>person - bag</code> or <code>background</code>, and SAM 3 returns the mask, with crisp edges on small and
medium objects even in 4K–8K photos, and shows it on the node; garment and face-part masks from SegFormer models
that parse faces and people close up; VTON garment masks with pose and parsing.</td>
<td><a href="docs/en/mask.md"><img src="docs/assets/examples/mask-morphology.jpg" alt="Mask morphology"></a><br>
<b>Mask tools</b> — fast grow / shrink on the CPU or the GPU, hole filling, soft edges, bounding boxes and
aspect-ratio matching.</td>
@@ -66,7 +69,8 @@ cd NH-Nodes
python -m pip install -r requirements.txt
```
Restart ComfyUI afterwards. Requirements: a recent ComfyUI (tested with 0.41) and Python 3.10+.
Restart ComfyUI afterwards. Requirements: a recent ComfyUI (tested with 0.41) and Python 3.10+. **SAM3 Segment** has
more requirements, listed below.
### Optional: VTON and segmentation models
@@ -74,14 +78,20 @@ Restart ComfyUI afterwards. Requirements: a recent ComfyUI (tested with 0.41) an
existing GPU build). Install one of them into ComfyUI's Python:
`python -m pip install onnxruntime-gpu` (NVIDIA) or `python -m pip install onnxruntime` (CPU).
- **Garment Segment**, **Portrait Segment** and **VTON** download their models from Hugging Face on first use (pinned
revisions, only the needed files). To keep models elsewhere, add these keys to `extra_model_paths.yaml`:
revisions, only the needed files). Garment Segment and Portrait Segment also fetch YuNet, a 230 KB face detector
(MIT) that runs on the CPU. To keep models elsewhere, add these keys to `extra_model_paths.yaml`:
```yaml
my_models:
base_path: D:/ai-models/
nh_nodes: NH-Nodes/ # SegFormer models (garment, face parsing)
nh_nodes: NH-Nodes/ # SegFormer models (garment, face parsing) and the YuNet face detector
vton_mask: ComfyUI-Vton-Mask/ # VTON pose + parsing ONNX models
sam3: sam3/ # SAM 3 checkpoint (you download sam3.pt yourself)
```
- **Requirements for SAM 3** (SAM3 Segment only): an NVIDIA GPU with CUDA, Meta's `sam3` package and the `sam3.pt`
checkpoint. NH-Nodes installs and downloads none of them: you install the package yourself and request the weights
from Meta. The steps, and what to do if another SAM 3 pack is already installed, are in
[Before you start with SAM 3](docs/en/vision.md#before-you-start-with-sam-3).
## Quick start
@@ -90,6 +100,7 @@ Restart ComfyUI afterwards. Requirements: a recent ComfyUI (tested with 0.41) an
| Upscale or refine a huge image piece by piece | Load Image → **Image Tile** → your sampler/upscaler per tile → **Image Untile** | [image-compose](docs/en/image-compose.md#image-tile-nh) |
| Generate at the model's native size, then restore the original framing | Load Image → **NH Smart Ratio Image Resize** → edit → **NH Restore Image Geometry** | [resolution](docs/en/resolution.md) |
| Inpaint only the clothes | Load Image → **Garment Segment** → **Mask Morphology** (grow) → inpaint | [vision](docs/en/vision.md) |
| Mask every person, window or the sky in a large photo, or everything except the person | Load Image → **SAM3 Segment** (`concepts`: `window`, `person - bag` or `background`) → mask → edit or inpaint; add **Segs Select** to pick single instances | [vision](docs/en/vision.md#sam-3-workflows) |
| Compare many LoRAs, one per queue | **Counter** → **Load LoRA Model Index** (index) → sampler → **Image Label** (prints the LoRA name) → Save Image | [loaders](docs/en/loaders.md) |
| Run one of two branches without executing the other | condition → **If/Else** (lazy) → … | [logic](docs/en/logic.md#ifelse-nh) |
@@ -103,7 +114,7 @@ Restart ComfyUI afterwards. Requirements: a recent ComfyUI (tested with 0.41) an
| Image — mask-based retouch | NH Mask-Aware Resize, NH Agnostic Image Generator, NH Simple Face Paste | [EN](docs/en/image-retouch.md) · [VI](docs/vi/image-retouch.md) |
| Resolution | NH Smart Resolution Picker, NH Smart Ratio Image Resize, NH Ratio Preset Image Resize, NH Remove Resize Padding, NH Restore Image Geometry | [EN](docs/en/resolution.md) · [VI](docs/vi/resolution.md) |
| Mask | Mask Morphology, Mask Properties, NH Create Box Mask, NH Mask Aspect Ratio Match | [EN](docs/en/mask.md) · [VI](docs/vi/mask.md) |
| Vision | Garment Segment, Portrait Segment | [EN](docs/en/vision.md) · [VI](docs/vi/vision.md) |
| Vision | Garment Segment, Portrait Segment, SAM3 Segment, Segs Select, Segs Compose, Segs To Masks, Segs To Impact SEGS, Segs Preview | [EN](docs/en/vision.md) · [VI](docs/vi/vision.md) |
| VTON | VTON Ultimate Processor | [EN](docs/en/vton.md) · [VI](docs/vi/vton.md) |
| Logic | Compare, Logic Gate, If/Else, Switch N, Any Switch, Any Branch Switch, Gate Switch, Value Match Index, Math Eval, Random Choice | [EN](docs/en/logic.md) · [VI](docs/vi/logic.md) |
| Text | Prompt Template, Prompt Scheduler, String Operations, Prompt Join, Text Split, Regex Extract, Text Index, Text Concatenate, Text Split Lines, Text Random Line | [EN](docs/en/text.md) · [VI](docs/vi/text.md) |
@@ -128,6 +139,9 @@ NH-Nodes' own code is [MIT](LICENSE). Some features rely on third-party code or
- **VTON Ultimate Processor** and **Portrait Segment** are for **non-commercial use only** (CC BY-NC-SA 4.0 code and
parsing models; non-commercial face-parsing model).
- **Garment Segment**'s model does not declare a licence.
- **SAM3 Segment** uses Meta's SAM 3 under the SAM License: royalty-free, with no non-commercial clause, but it bars
military or warfare, nuclear, espionage, and gun or illegal-weapon uses. The weights and the `sam3` package are not
included; you get them from Meta.
## Development
+37 -1
View File
@@ -16,6 +16,7 @@ MIT license.
| `VTON Ultimate Processor (NH)` | **No** | Its code and parsing models are CC BY-NC-SA 4.0 (FitDiT / OOTDiffusion) |
| `Portrait Segment (NH)` | **No** | The face-parsing model card restricts use to non-commercial research and education |
| `Garment Segment (NH)` | Unclear | The model uploader does not declare a license |
| `SAM3 Segment (NH)` | Yes, within trade-control limits | Meta's SAM License is royalty-free and has no non-commercial clause, but it forbids ITAR-regulated activities and uses that trade controls prohibit (military or warfare, nuclear, espionage, guns and illegal weapons). The model and the `sam3` package are not part of NH-Nodes: you get them from Meta |
## Source code bundled in this repository
@@ -54,9 +55,44 @@ They are looked up first in the folders registered as `vton_mask` and `nh_nodes`
| VTON Ultimate Processor | same | `humanparsing/parsing_atr.onnx`, `humanparsing/parsing_lip.onnx` | From the OOTDiffusion/FitDiT checkpoints (CC BY-NC-SA 4.0); trained on the ATR and LIP datasets, which are distributed for research use |
| Portrait Segment | [`jonathandinu/face-parsing`](https://huggingface.co/jonathandinu/face-parsing) @ `758b82e` | SegFormer fine-tuned on CelebAMask-HQ | Model card: "non-commercial research and educational purposes" |
| Garment Segment | [`1038lab/segformer_fashion`](https://huggingface.co/1038lab/segformer_fashion) @ `e5bae2d` | SegFormer trained for Fashionpedia classes | No license declared by the uploader (Fashionpedia annotations are CC BY 4.0) |
| Portrait Segment, Garment Segment | [`opencv/face_detection_yunet`](https://huggingface.co/opencv/face_detection_yunet) @ `3cc26e7` | `yunet/face_detection_yunet_2023mar.onnx`, the YuNet face detector (about 230 KB), and its `LICENSE` file | MIT, Copyright (c) 2020 Shiqi Yu (OpenCV Zoo) |
YuNet is the face detector that Portrait Segment and Garment Segment use to find the people in a
picture. It runs on the CPU through OpenCV, and its MIT license does not change the "At a glance"
table: the terms of those two nodes come from their SegFormer models.
## Software and models you install yourself
`SAM3 Segment (NH)` runs Meta's SAM 3. NH-Nodes neither bundles nor downloads it, and the
nodes only load it when you use them. You get both parts from Meta and accept Meta's terms
yourself.
| What | Where it comes from | Terms |
|------|---------------------|-------|
| `sam3` package (code) | [`facebookresearch/sam3`](https://github.com/facebookresearch/sam3), installed by the user with pip | SAM License |
| SAM 3 checkpoint `sam3.pt` | [`facebook/sam3`](https://huggingface.co/facebook/sam3) on Hugging Face, a gated repository: access is granted after the user accepts the license. It goes in the `sam3` model folder (`models/sam3`, or any location added under `sam3` in `extra_model_paths.yaml`), which other SAM 3 packs share | SAM License |
The [SAM License](https://github.com/facebookresearch/sam3/blob/main/LICENSE) (Meta, last
updated 19 November 2025) is a non-exclusive, worldwide, non-transferable, royalty-free
limited license to use, reproduce, distribute, copy, modify and create derivative works of
the SAM Materials. It does not mention commercial use. What matters to a user of NH-Nodes:
- **Trade controls.** You must not be a target of trade controls and must comply with them.
The materials may not be used for activities subject to the International Traffic in Arms
Regulations (ITAR), or for end uses that trade controls prohibit, including military or
warfare purposes, nuclear industries or applications, espionage, and the development or
use of guns or illegal weapons.
- Whoever redistributes the SAM Materials must include a copy of the license. NH-Nodes does
not redistribute them.
- Research publications that use the SAM Materials must acknowledge it.
- The materials may not be used to reverse engineer or decompile them, nor to encourage
others to.
- They are provided "as is", and Meta may terminate the license if you breach it.
## Python dependencies
Dependencies listed in `requirements.txt` are installed from PyPI and are not
redistributed here; each keeps its own license. ONNX Runtime (MIT) is optional
and only needed by the VTON node.
and only needed by the VTON node. The `sam3` package (SAM License) and its extras
(`timm`, `ftfy`, `regex`, `iopath`) are optional too: only `SAM3 Segment (NH)` needs
them, and you install them yourself.
+108
View File
@@ -0,0 +1,108 @@
"""Face boxes for Portrait Segment, from YuNet (OpenCV Zoo, MIT license).
YuNet is a 230 KB ONNX model that runs on the CPU through the OpenCV the pack already needs, so finding faces costs
no VRAM and no extra package. The weights are fetched once, pinned to a fixed Hugging Face revision, into the same
``nh_nodes`` model folder as the parsers (models/NH-Nodes/yunet by default).
"""
import os
import cv2
import numpy as np
from .model_files import download_pinned, register_model_folder
from .segformer_backend import FOLDER_EXTENSIONS, FOLDER_NAME, default_folder
YUNET = {
"repo_id": "opencv/face_detection_yunet",
"revision": "3cc26e7f1014a5ee5d74a42acee58bafc9d0a310",
"file": "face_detection_yunet_2023mar.onnx",
"folder": "yunet",
}
MIN_SCORE = 0.6 # YuNet's own samples use 0.9; a box that is not a face is turned down later by the face parser
DETECT_SIDES = (2048, 640) # faces are looked for on copies at most this long: small faces show on the first, big or
# grainy ones (scanned prints, close-ups) are often found only on the second
MAX_FACES = 32 # the largest ones; each face costs two parses
_detector = None
def model_path(download=True):
"""Path of the YuNet weights, from any registered ``nh_nodes`` folder; downloaded into the default one if absent."""
for base in register_model_folder(FOLDER_NAME, default_folder(), FOLDER_EXTENSIONS):
candidate = os.path.join(base, YUNET["folder"], YUNET["file"])
if os.path.isfile(candidate):
return candidate
if not download:
return None
target = os.path.join(default_folder(), YUNET["folder"])
download_pinned(YUNET["repo_id"], YUNET["revision"], target, (YUNET["file"], "LICENSE"))
path = os.path.join(target, YUNET["file"])
if not os.path.isfile(path):
raise RuntimeError(f"[NH-Nodes] Face detector missing after download: {path}")
return path
def _get_detector(size):
"""The YuNet detector, set up for pictures of ``size`` (width, height). Loaded from bytes read by Python, so a
model folder whose path has accents (which OpenCV cannot open on Windows) works too."""
global _detector
if _detector is None:
weights = np.fromfile(model_path(), dtype=np.uint8)
_detector = cv2.FaceDetectorYN.create("onnx", weights, np.empty(0, np.uint8), size, MIN_SCORE, 0.3, 5000)
_detector.setInputSize(size)
return _detector
def _detect_at(image, max_side, min_score):
"""Face boxes found on a copy of ``image`` at most ``max_side`` pixels long, in the image's pixels."""
height, width = image.shape[:2]
scale = min(1.0, max_side / max(height, width))
small = image
if scale < 1.0:
small = cv2.resize(image, (max(1, round(width * scale)), max(1, round(height * scale))),
interpolation=cv2.INTER_AREA)
detector = _get_detector((small.shape[1], small.shape[0]))
_, found = detector.detect(cv2.cvtColor(np.ascontiguousarray(small), cv2.COLOR_RGB2BGR))
faces = []
for row in [] if found is None else found:
score = float(row[-1])
if score < min_score:
continue
x, y, w, h = (float(v) / scale for v in row[:4])
box = (max(0.0, x), max(0.0, y), min(float(width), x + w), min(float(height), y + h))
if box[2] > box[0] and box[3] > box[1]:
faces.append((*box, score))
return faces
def _area(box):
return max(0.0, box[2] - box[0]) * max(0.0, box[3] - box[1])
def _overlap(a, b):
return _area((max(a[0], b[0]), max(a[1], b[1]), min(a[2], b[2]), min(a[3], b[3])))
def merge_faces(faces):
"""One box per face from boxes found at several scales: best score first, a box is dropped when it overlaps a
kept one (IoU 0.3 or more) or lies mostly (60%) inside one, as a half face found at one scale does."""
kept = []
for face in sorted(faces, key=lambda f: f[4], reverse=True):
if all(_overlap(face, other) < 0.3 * (_area(face) + _area(other) - _overlap(face, other))
and _overlap(face, other) < 0.6 * min(_area(face), _area(other)) for other in kept):
kept.append(face)
return kept
def detect_faces(image, min_score=MIN_SCORE, sides=DETECT_SIDES, limit=MAX_FACES):
"""[(x1, y1, x2, y2, score)] of the faces in the uint8 RGB ``image``, in its pixels, largest first.
Looks on copies of each length in ``sides`` and merges what they find (see merge_faces), which finds faces from
about 1/150 of the picture's long side up to close-ups. A face cut by the frame is often missed; the caller can
look again on a copy with a margin, and parses the whole frame when nothing is found.
"""
found = [face for side in sides for face in _detect_at(image, side, min_score)]
faces = merge_faces(found)
faces.sort(key=_area, reverse=True)
return faces[:limit]
+339
View File
@@ -0,0 +1,339 @@
"""Where to run a parsing model on a big picture, and how to put its answers back (model-free, unit tested).
Parsing models see a small fixed-size picture (512 x 512 for the SegFormer face and fashion parsers). Squeezing a
whole 8K photo into that leaves a face a few dozen pixels wide, so the parsers also run on crops: a square around
each face, or a box of the model's shape around each group of people. A crop only refines the parse of the whole
frame: it is pasted inside its own region, away from the region's cut edges, so whatever it misses keeps the
whole-frame answer and no straight crop edge shows. Regions are (x1, y1, x2, y2) in image pixels with exclusive ends,
and may reach past the image; the missing part is padded with grey.
"""
import cv2
import numpy as np
PAD_VALUE = 114 # grey, for the parts of a crop outside the picture
def square_around(box, scale, lift=0.0):
"""Square region ``scale`` times the longer side of ``box``, centred on it and moved up by ``lift`` times its
side (a head's hair sits above its face box)."""
x1, y1, x2, y2 = box[:4]
side = max(1.0, scale * max(x2 - x1, y2 - y1))
cx, cy = (x1 + x2) / 2.0, (y1 + y2) / 2.0 - lift * side
left, top = int(round(cx - side / 2.0)), int(round(cy - side / 2.0))
size = int(round(side))
return left, top, left + size, top + size
def _span(lo, hi, size, limit):
"""[start, start + size) holding [lo, hi), slid to stay inside [0, limit) when it fits there."""
start = (lo + hi) / 2.0 - size / 2.0
if size <= limit:
start = min(max(0.0, start), limit - size)
start = int(round(start))
return start, start + int(size)
def fit_box(box, aspect, margin, image_size):
"""Region of the model's shape (``aspect`` = width / height) around ``box`` plus ``margin`` (a fraction of its
longer side) on each side. It stays inside the image where it fits there, and is padded where it does not."""
height, width = image_size
x1, y1, x2, y2 = box[:4]
pad = margin * max(x2 - x1, y2 - y1)
w, h = (x2 - x1) + 2 * pad, (y2 - y1) + 2 * pad
w, h = max(w, h * aspect), max(h, w / aspect)
w, h = int(np.ceil(w)), int(np.ceil(h))
left, right = _span(x1, x2, w, width)
top, bottom = _span(y1, y2, h, height)
return left, top, right, bottom
SPLIT = 1.3 # a box this much wider than the parser's shape is covered by a row of regions instead of one
def _centres(lo, hi, tile, overlap):
count = max(2, int(np.ceil((hi - lo - tile) / (tile * (1.0 - overlap)))) + 1)
return np.linspace(lo + tile / 2.0, hi - tile / 2.0, count)
def cover_box(box, aspect, margin, image_size, overlap=0.2):
"""Regions of the parser's shape (``aspect`` = width / height) that together cover ``box`` plus ``margin``.
One region (see fit_box) unless the box is much wider than that shape, as a group photo is: then a row of
overlapping regions as tall as the box (plus a margin of its height), so each person keeps the parser's resolution
instead of the group being squeezed into one region with grey bands above and below. A tall box keeps one region:
splitting a standing person into halves would hide the rest of the body from the parser.
"""
x1, y1, x2, y2 = box[:4]
pad = margin * (y2 - y1)
bx1, by1, bx2, by2 = x1 - pad, y1 - pad, x2 + pad, y2 + pad
if bx2 - bx1 <= (by2 - by1) * aspect * SPLIT:
return [fit_box(box, aspect, margin, image_size)]
tile = (by2 - by1) * aspect
return [fit_box((c - tile / 2.0, by1, c + tile / 2.0, by2), aspect, 0.0, image_size)
for c in _centres(bx1, bx2, tile, overlap)]
def square_frame(image_size):
"""The square region centred on the picture that holds all of it (the short side is padded)."""
height, width = image_size
side = max(height, width)
left, top = (width - side) // 2, (height - side) // 2
return left, top, left + side, top + side
def visible(region, image_size):
"""The part of ``region`` inside an image of ``image_size`` (height, width)."""
height, width = image_size
x1, y1, x2, y2 = region
return max(0, x1), max(0, y1), min(width, x2), min(height, y2)
def cut_padded(image, region):
"""``image[region]`` (H x W x C), padded with grey where the region leaves the image."""
height, width = image.shape[:2]
x1, y1, x2, y2 = region
vx1, vy1, vx2, vy2 = visible(region, (height, width))
crop = image[vy1:vy2, vx1:vx2]
pads = (vy1 - y1, y2 - vy2, vx1 - x1, x2 - vx2)
if any(pads):
crop = cv2.copyMakeBorder(np.ascontiguousarray(crop), *pads, cv2.BORDER_CONSTANT,
value=(PAD_VALUE,) * image.shape[2])
return np.ascontiguousarray(crop)
def nearest_owner(region, boxes, regions, index):
"""Bool mask over ``region`` (inside the image): the pixels that belong to ``boxes[index]`` because no other box
whose region also covers them is nearer (distance from the box centre, in box sizes). Where the crops around two
faces overlap, each face keeps the parse of its own crop."""
x1, y1, x2, y2 = region
overlapping = [other for other in range(len(boxes)) if other != index and regions[other][2] > x1
and regions[other][0] < x2 and regions[other][3] > y1 and regions[other][1] < y2]
if not overlapping:
return np.ones((y2 - y1, x2 - x1), dtype=bool)
# decided on a grid of at most OWNER_GRID points a side, then enlarged: the border between two faces runs through
# hair or background, where a few pixels do not matter, and an 8K crop would otherwise cost a second per face
step = max(1, int(np.ceil(max(x2 - x1, y2 - y1) / OWNER_GRID)))
ys = np.arange(y1, y2, step, dtype=np.float32)[:, None] + step / 2.0
xs = np.arange(x1, x2, step, dtype=np.float32)[None, :] + step / 2.0
def distance(box):
bx1, by1, bx2, by2 = box[:4]
size = max(1.0, float(max(bx2 - bx1, by2 - by1)))
return np.hypot(xs - (bx1 + bx2) / 2.0, ys - (by1 + by2) / 2.0) / size
own = distance(boxes[index])
keep = np.ones(own.shape, dtype=bool)
for other in overlapping:
ox1, oy1, ox2, oy2 = regions[other]
covered = (xs >= ox1) & (xs < ox2) & (ys >= oy1) & (ys < oy2)
keep &= ~covered | (own <= distance(boxes[other]))
if step > 1:
keep = cv2.resize(keep.astype(np.uint8), (x2 - x1, y2 - y1), interpolation=cv2.INTER_NEAREST) > 0
return keep
OWNER_GRID = 512
CUT_BAND = 0.08 # share of a crop's side, along each of its edges that cuts the picture, left to the whole frame
NEAR_PEOPLE = 0.02 # how far (share of the long side) from the whole-frame parse's people a crop may add labels
def inner_mask(region, image_size, band=CUT_BAND):
"""Bool mask over the part of ``region`` inside the picture: True except within ``band`` of the region's size
along its edges that lie inside the picture. The parser sees things cut there, so its answer is less sure; an edge
that is the picture's own edge (or beyond it) keeps everything."""
height, width = image_size
x1, y1, x2, y2 = region
vx1, vy1, vx2, vy2 = visible(region, image_size)
keep = np.ones((vy2 - vy1, vx2 - vx1), dtype=bool)
cut_x, cut_y = int(round(band * (x2 - x1))), int(round(band * (y2 - y1)))
if x1 > 0:
keep[:, :max(0, x1 + cut_x - vx1)] = False
if x2 < width:
keep[:, max(0, (x2 - cut_x) - vx1):] = False
if y1 > 0:
keep[:max(0, y1 + cut_y - vy1), :] = False
if y2 < height:
keep[max(0, (y2 - cut_y) - vy1):, :] = False
return keep
def parse_region(predict, image, region):
"""(``predict`` run on the padded crop, cut to the part of ``region`` inside the picture; that part). When
``predict`` returns several maps (labels and their confidence), each is cut the same way."""
seen = visible(region, image.shape[:2])
result = predict(cut_padded(image, region))
x1, y1 = region[:2]
rows, cols = slice(seen[1] - y1, seen[3] - y1), slice(seen[0] - x1, seen[2] - x1)
if isinstance(result, tuple):
return tuple(part[rows, cols] for part in result), seen
return result[rows, cols], seen
GROW_ROUNDS = 3 # times a crop may grow when its people run past one of its cut edges
GROW_STEP = 0.5 # how much it grows each time, as a share of its size
def _cut_sides_with_people(labels, region, image_size, background=0, share=0.02):
"""Edges of ``region`` that cut the picture and where the parse (of the part inside the picture) still finds
people within CUT_BAND: there the crop sliced through somebody."""
height, width = image_size
x1, y1, x2, y2 = region
vx1, vy1, vx2, vy2 = visible(region, image_size)
band_x, band_y = max(1, int(round(CUT_BAND * (x2 - x1)))), max(1, int(round(CUT_BAND * (y2 - y1))))
found = labels != background
sides = []
if x1 > 0 and found[:, :band_x].mean() > share:
sides.append("left")
if x2 < width and found[:, max(0, (vx2 - vx1) - band_x):].mean() > share:
sides.append("right")
if y1 > 0 and found[:band_y, :].mean() > share:
sides.append("top")
if y2 < height and found[max(0, (vy2 - vy1) - band_y):, :].mean() > share:
sides.append("bottom")
return sides
def _grown(region, sides, aspect, image_size):
"""``region`` enlarged by GROW_STEP of its size towards ``sides``, back in the parser's shape."""
x1, y1, x2, y2 = region
step_x, step_y = GROW_STEP * (x2 - x1), GROW_STEP * (y2 - y1)
box = (x1 - step_x * ("left" in sides), y1 - step_y * ("top" in sides),
x2 + step_x * ("right" in sides), y2 + step_y * ("bottom" in sides))
return fit_box(box, aspect, 0.0, image_size)
COARSE_TRUST = 0.5 # the first look's confidence counts this much against a crop's: a crop sees people sharper
def person_box(face, image_size):
"""Rough box of the person a face (x1, y1, x2, y2[, score]) belongs to: five face widths wide, from just above
the head to about eight face heights below its top (a standing person), inside the picture."""
height, width = image_size
x1, y1, x2, y2 = face[:4]
face_w, face_h = x2 - x1, y2 - y1
centre = (x1 + x2) / 2.0
return (max(0, int(centre - 2.5 * face_w)), max(0, int(y1 - 0.6 * face_h)),
min(width, int(np.ceil(centre + 2.5 * face_w))), min(height, int(np.ceil(y1 + 8.0 * face_h))))
def people_labels(predict, image, aspect, margin=0.08, people=()):
"""Label map of the people in the uint8 RGB ``image``, from ``predict`` (picture -> (label map with 0 for
background, confidence map)), a parser trained on pictures of the shape ``aspect`` (width / height).
A first look parses the whole frame and is kept as the answer to refine. Each group of people (those it found,
and the ``people`` boxes found from faces, which it may have missed) gets regions of the parser's shape (a row of
them for a wide group, see cover_box), parsed again at the parser's full resolution; a region whose people run past
one of its cut edges grows that way and is parsed again. A region's label replaces the current one where it is
surer and away from the region's cut edges, for the parts of it that lie on people already found (see anchored)
and for background near them; so nothing is invented on an empty backdrop and overlapping regions meet where their
answers do, not along a straight line.
"""
height, width = image.shape[:2]
coarse, trust = predict(image)
regions = [region for box in foreground_clusters(coarse, extra=people)
for region in cover_box(box, aspect, margin, (height, width))]
if not regions or regions == [(0, 0, width, height)]:
return coarse # nobody found, or the first look already saw the whole picture in the parser's own shape
anchor = coarse != 0 # where the first look (or a face) puts people
for x1, y1, x2, y2 in people:
anchor[y1:y2, x1:x2] = True
near = near_people(anchor.astype(np.uint8))
canvas, trust = coarse.copy(), trust.astype(np.float32) * COARSE_TRUST
for region in regions:
(labels, sure), seen = parse_region(predict, image, region)
for _ in range(GROW_ROUNDS):
sides = _cut_sides_with_people(labels, region, (height, width))
if not sides:
break
region = _grown(region, sides, aspect, (height, width))
(labels, sure), seen = parse_region(predict, image, region)
if labels.size == 0:
continue
x1, y1, x2, y2 = seen
sure = sure.astype(np.float32)
allowed = anchored(labels, anchor[y1:y2, x1:x2]) | ((labels == 0) & near[y1:y2, x1:x2])
keep = inner_mask(region, (height, width)) & allowed & (sure > trust[y1:y2, x1:x2])
canvas[y1:y2, x1:x2][keep] = labels[keep]
trust[y1:y2, x1:x2][keep] = sure[keep]
return canvas
ANCHOR_SHARE = 0.1 # share of a crop's part that must lie on people already found for the part to be kept
def anchored(labels, anchor, share=ANCHOR_SHARE, grid=1024):
"""Bool mask of the labelled pixels of a crop's ``labels`` whose part (the connected pixels of one label) lies on
``anchor`` (people already found) for at least ``share`` of its area. A crop may so extend a person the first look
saw only in part, but anything it finds on its own (clothes "seen" on a plain backdrop, even right next to
somebody) is dropped. Worked out on a copy at most ``grid`` pixels long."""
height, width = labels.shape
scale = min(1.0, grid / max(height, width, 1))
small, held = labels, anchor
if scale < 1.0:
size = (max(1, round(width * scale)), max(1, round(height * scale)))
small = cv2.resize(labels.astype(np.int32), size, interpolation=cv2.INTER_NEAREST)
held = cv2.resize(anchor.astype(np.uint8), size, interpolation=cv2.INTER_NEAREST) > 0
kept = np.zeros(small.shape, dtype=bool)
for value in np.unique(small):
if value == 0:
continue
count, parts = cv2.connectedComponents((small == value).astype(np.uint8), connectivity=8)
if count <= 1:
continue
areas = np.bincount(parts.ravel(), minlength=count)
on_people = np.bincount(parts[held], minlength=count)
good = on_people >= share * areas
good[0] = False
kept |= good[parts]
if scale < 1.0:
kept = cv2.resize(kept.astype(np.uint8), (width, height), interpolation=cv2.INTER_NEAREST) > 0
return kept & (labels != 0)
def near_people(labels, background=0, reach=NEAR_PEOPLE, grid=1024):
"""Bool map of the pixels within ``reach`` (a share of the long side) of a labelled pixel. Worked out on a copy at
most ``grid`` pixels long and enlarged, since a few pixels do not matter here and a 300 px dilation of an 8K map
would take seconds."""
height, width = labels.shape
scale = min(1.0, grid / max(height, width))
found = (labels != background).astype(np.uint8)
if scale < 1.0:
found = cv2.resize(found, (max(1, round(width * scale)), max(1, round(height * scale))),
interpolation=cv2.INTER_AREA)
found = (found > 0).astype(np.uint8)
radius = max(1, int(round(reach * max(found.shape))))
grown = cv2.dilate(found, cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (2 * radius + 1, 2 * radius + 1)))
if scale < 1.0:
grown = cv2.resize(grown, (width, height), interpolation=cv2.INTER_NEAREST)
return grown > 0
def foreground_clusters(labels, background=0, min_fraction=0.001, gap=0.05, extra=()):
"""Boxes around the groups of labelled pixels of a label map (and the ``extra`` boxes): specks under
``min_fraction`` of the picture are left out, and boxes closer to each other than ``gap`` of the picture's long
side are merged, so people standing together form one group and people far apart get one box each."""
found = (labels != background).astype(np.uint8)
count, _, stats, _ = cv2.connectedComponentsWithStats(found, connectivity=8)
boxes = [[int(stats[i, cv2.CC_STAT_LEFT]), int(stats[i, cv2.CC_STAT_TOP]),
int(stats[i, cv2.CC_STAT_LEFT] + stats[i, cv2.CC_STAT_WIDTH]),
int(stats[i, cv2.CC_STAT_TOP] + stats[i, cv2.CC_STAT_HEIGHT])]
for i in range(1, count) if stats[i, cv2.CC_STAT_AREA] >= min_fraction * labels.size]
boxes += [[int(v) for v in box[:4]] for box in extra]
reach = gap * max(labels.shape)
merged = True
while merged:
merged = False
for i in range(len(boxes)):
for j in range(i + 1, len(boxes)):
a, b = boxes[i], boxes[j]
if (max(a[0], b[0]) - min(a[2], b[2]) < reach) and (max(a[1], b[1]) - min(a[3], b[3]) < reach):
boxes[i] = [min(a[0], b[0]), min(a[1], b[1]), max(a[2], b[2]), max(a[3], b[3])]
del boxes[j]
merged = True
break
if merged:
break
return [tuple(box) for box in sorted(boxes)]
+710
View File
@@ -0,0 +1,710 @@
"""SAM 3 for the NH vision nodes: finding the model, loading it, and segmenting concepts in two passes.
NH-Nodes does not ship SAM 3. It uses Meta's ``sam3`` package, installed separately (see ``INSTALL_HINT``), and the
``sam3.pt`` checkpoint, which Hugging Face hands out only after Meta's license is accepted, so it is never
downloaded automatically. Checkpoints are looked up in the ``sam3`` model folder (``models/sam3`` by default, more
locations in ``extra_model_paths.yaml``); other SAM 3 packs use the same folder, so one copy serves all of them.
Pass 1 looks at the whole image once: the image is resized to the model's square input, encoded once and asked for
every concept. Each instance's mask is sampled back to full resolution inside its own box only.
Pass 2 (``detail_level="balanced"``) cuts each instance out of the full-resolution image with a margin and segments it
again with SAM 3's box-prompt decoder, the pass-1 mask given as a hint. Objects that are small or medium in the frame
then get their edges from the original pixels instead of a ~1000 px view of the whole frame. SAM 3 draws masks on a
grid of about 288 cells across whatever it looks at, so a subject that fills most of an 8K frame still has edges at
about 1/300 of its size; hair-level edges need matting. A close-up that does not show the pass-1 instance is dropped
and the pass-1 mask kept.
The official sam3 code builds some of its lookup tables on the GPU, so SAM 3 runs on CUDA only.
"""
import contextlib
import importlib
import importlib.util
import inspect
import itertools
import math
import os
import sys
import cv2
import numpy as np
import torch
import torch.nn.functional as F
from .model_files import make_room_on_gpu, models_dir, register_model_folder
from .segs import SegsFrame, make_segment
try:
import folder_paths
except Exception: # running outside ComfyUI (tests)
folder_paths = None
try:
import comfy.model_management as _mm
except Exception:
_mm = None
FOLDER_NAME = "sam3" # shared with the other SAM 3 packs
# Listed in ComfyUI for the folder: SAM 3 checkpoints in the formats SAM 3 packs use, not Hugging Face cache files.
FOLDER_EXTENSIONS = (".pt", ".pth", ".safetensors")
CHECKPOINT_EXTENSIONS = (".pt", ".pth") # what sam3's own loader reads (torch.load)
DEFAULT_CHECKPOINT = "sam3.pt"
BPE_FILE = "bpe_simple_vocab_16e6.txt.gz" # sam3's tokenizer vocabulary, in the package's assets folder
DETAIL_LEVELS = ("balanced", "fast")
SAM3_COMMIT = "b26a5f330e05d321afb39d01d3d4881f258f65ff" # the sam3 build NH-Nodes is tested with
INSTALL_HINT = (
"SAM 3 needs Meta's sam3 package, which NH-Nodes does not install for you. In ComfyUI's Python environment run:\n"
f" pip install --no-deps git+https://github.com/facebookresearch/sam3@{SAM3_COMMIT}\n"
" pip install timm ftfy regex iopath\n"
"--no-deps keeps your numpy: sam3 asks for numpy<2, which ComfyUI does not need. "
"Restart ComfyUI afterwards."
)
CHECKPOINT_HINT = (
"Ask for access at https://huggingface.co/facebook/sam3 (Meta has you accept the SAM License first), "
"download sam3.pt from that page and put it in {folder}. Then refresh ComfyUI (press R) and pick it."
)
DEDUP_IOU = 0.9 # a second instance of the same concept overlapping a better one this much is a duplicate
# A close-up replaces the pass-1 mask when it shows the same object: at least REFINE_MIN_INSIDE of it lies within the
# pass-1 blur of the pass-1 mask, and it covers at least REFINE_MIN_CORE of the pass-1 core (see close_up_agrees).
REFINE_MIN_INSIDE = 0.8
REFINE_MIN_CORE = 0.6
_PASS1_GRID = 288 # SAM 3's detector draws masks on a 288 x 288 grid of its square view of the whole image
_AGREE_PIXELS = 1 << 21 # masks larger than this are compared at a reduced scale
_WIDE_ASPECT = 4.0 # beyond this aspect ratio SAM 3's square view squeezes the picture enough to miss small things
_CROP_MARGIN = 0.2 # pass-2 crop margin on each side, as a fraction of the instance's longer side
_CROP_MIN_MARGIN = 16 # px
_CROP_MAX_ASPECT = 2.0 # pass-2 crops are widened to at most 2:1, the model squeezes them into a square
_SAMPLE_CHUNK = 1 << 22 # output pixels per grid_sample call when resampling masks
# SAM 3's masks fade out along the border of the square view it segments: on a 1008 px view the last 3 rows and
# columns hold logits near 0 even where the object runs off the picture (79% / 23% / 23% of an edge's object pixels
# still positive), which left a ragged gap along the image edge. That band is lifted to the first row inside it, only
# where it is still in the fade (see extend_to_border). A close-up fades over 2 to 3 rows of its own view. The price:
# a thing that ends within that band (about 0.3% of the picture's side) above the edge is drawn down to the edge,
# which costs far less than the ragged edge every person cut by the frame used to get.
_BORDER_CELLS = 3
_CLOSE_UP_CELLS = 3
_FADE_FLOOR = -2.5 # band logits below this are a sure "no" and are left alone (the fade reaches about -1.6)
_VIEW_SIZE = 1008 # side of SAM 3's square view
NEAR_MISS = 0.1 # candidates scoring at least this (but under the threshold) are reported as near misses
_WORKING_BYTES = 3 << 30 # VRAM for activations and the full-resolution image, on top of the weights
_API = None
_CACHE = {}
# ---------------------------------------------------------------------------
# Checkpoints
# ---------------------------------------------------------------------------
def default_folder():
"""<models dir>/sam3, the folder ComfyUI and the other SAM 3 packs use."""
return os.path.join(models_dir(), FOLDER_NAME)
def checkpoint_folders():
"""Every folder searched for checkpoints, best first (registers the folder name with ComfyUI)."""
return register_model_folder(FOLDER_NAME, default_folder(), FOLDER_EXTENSIONS)
def checkpoint_choices():
"""Checkpoint names for the node's list: .pt/.pth files in the sam3 folders, or the default name if none yet."""
names = set()
folders = checkpoint_folders()
if folder_paths is not None:
try:
names.update(folder_paths.get_filename_list(FOLDER_NAME))
except Exception as exc:
print(f"[NH-Nodes] Could not list the {FOLDER_NAME} model folder: {exc}")
else:
for folder in folders:
if os.path.isdir(folder):
names.update(os.listdir(folder))
names = sorted(name for name in names if name.lower().endswith(CHECKPOINT_EXTENSIONS))
return names or [DEFAULT_CHECKPOINT]
def resolve_checkpoint(name):
"""Full path of checkpoint ``name`` in the sam3 folders; a RuntimeError says where to get it when it is missing."""
name = str(name or DEFAULT_CHECKPOINT)
parts = name.replace("\\", "/").split("/")
if os.path.isabs(name) or ".." in parts:
raise RuntimeError(f"[NH-Nodes] SAM 3 checkpoint name must be a file in the {FOLDER_NAME} folder: {name!r}")
if not name.lower().endswith(CHECKPOINT_EXTENSIONS):
raise RuntimeError(
f"[NH-Nodes] {name} is not a SAM 3 .pt checkpoint. The safetensors export on Hugging Face is for the "
"transformers library and uses other weight names; download sam3.pt instead."
)
folders = checkpoint_folders()
path = folder_paths.get_full_path(FOLDER_NAME, name) if folder_paths is not None else None
if path is None:
path = next((os.path.join(folder, *parts) for folder in folders
if os.path.isfile(os.path.join(folder, *parts))), None)
if path is None or not os.path.isfile(path):
raise RuntimeError(
f"[NH-Nodes] SAM 3 checkpoint {name} not found in: {', '.join(folders)}.\n"
+ CHECKPOINT_HINT.format(folder=folders[0] if folders else default_folder())
)
return path
# Make the folder name known to ComfyUI as soon as the pack loads.
checkpoint_folders()
# ---------------------------------------------------------------------------
# The sam3 package and the model
# ---------------------------------------------------------------------------
def _tf32_flags():
"""(matmul, cudnn) TF32 switches, or None when torch refuses to report them (its newer precision API in use)."""
try:
return torch.backends.cuda.matmul.allow_tf32, torch.backends.cudnn.allow_tf32
except Exception:
return None
def _restore_tf32(flags):
if flags is not None:
try:
torch.backends.cuda.matmul.allow_tf32, torch.backends.cudnn.allow_tf32 = flags
except Exception:
pass
def _missing_parameters(func, names):
"""Names from ``names`` that ``func`` does not accept (none when its signature cannot be read or takes **kwargs)."""
try:
parameters = inspect.signature(func).parameters
except (TypeError, ValueError):
return []
if any(p.kind == p.VAR_KEYWORD for p in parameters.values()):
return []
return [name for name in names if name not in parameters]
def ensure_sam3():
"""(build_sam3_image_model, Sam3Processor) from the installed sam3 package.
A RuntimeError explains what to do when the package is missing, fails to import or lacks the image API
NH-Nodes uses. Importing sam3's model builder switches TF32 on for the whole process; the previous setting is
put back, so ComfyUI's own models keep theirs.
"""
global _API
if _API is not None:
return _API
if importlib.util.find_spec("sam3") is None:
raise RuntimeError("[NH-Nodes] " + INSTALL_HINT)
flags = _tf32_flags()
try:
builder = importlib.import_module("sam3.model_builder")
processors = importlib.import_module("sam3.model.sam3_image_processor")
except Exception as exc:
raise RuntimeError(f"[NH-Nodes] The sam3 package is installed but failed to load: {exc}\n{INSTALL_HINT}") from exc
finally:
_restore_tf32(flags)
build = getattr(builder, "build_sam3_image_model", None)
processor_cls = getattr(processors, "Sam3Processor", None)
missing = []
if not callable(build):
missing.append("build_sam3_image_model")
else:
missing += [f"build_sam3_image_model({name}=)" for name in _missing_parameters(
build, ("bpe_path", "checkpoint_path", "load_from_HF", "enable_inst_interactivity", "device", "eval_mode"))]
if processor_cls is None:
missing.append("Sam3Processor")
else:
missing += [f"Sam3Processor({name}=)" for name in
_missing_parameters(processor_cls, ("device", "confidence_threshold"))]
missing += [f"Sam3Processor.{name}" for name in ("set_image", "set_text_prompt", "reset_all_prompts")
if not callable(getattr(processor_cls, name, None))]
if missing:
raise RuntimeError(
f"[NH-Nodes] The installed sam3 package lacks {', '.join(missing)}; it is older or newer than the "
f"versions NH-Nodes supports. Reinstall it:\n{INSTALL_HINT}"
)
_API = (build, processor_cls)
return _API
def sam3_device():
"""ComfyUI's torch device, which must be a CUDA GPU for SAM 3."""
if _mm is not None:
device = _mm.get_torch_device()
else:
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
if device.type != "cuda":
raise RuntimeError(
f"[NH-Nodes] SAM 3 needs an NVIDIA GPU with CUDA (ComfyUI runs on {device}): the official sam3 code "
"builds part of the model on the GPU."
)
return device
def bpe_path():
"""The tokenizer vocabulary inside the sam3 package that was imported, whichever copy that is.
Other packs bundle their own copy of sam3 and may put it first on sys.path (ComfyUI-RMBG does), so "sam3" is not
always Meta's install. Such a copy can look for the vocabulary in the wrong folder when it is not given one
(ComfyUI-RMBG's looks one level too high), so NH-Nodes always finds it here and passes it.
"""
package = sys.modules.get("sam3") or importlib.import_module("sam3")
folders = [os.path.dirname(package.__file__)] if getattr(package, "__file__", None) else []
folders += [folder for folder in getattr(package, "__path__", []) if folder not in folders]
for folder in folders:
for candidate in (os.path.join(folder, "assets", BPE_FILE), os.path.join(folder, os.pardir, "assets", BPE_FILE)):
if os.path.isfile(candidate):
return os.path.normpath(candidate)
where = folders[0] if folders else "its folder"
raise RuntimeError(f"[NH-Nodes] The sam3 package in {where} has no assets/{BPE_FILE}; reinstall it:\n{INSTALL_HINT}")
def check_checkpoint(path):
"""Fail early, with the download advice, when ``path`` is not a readable SAM 3 checkpoint.
sam3 loads weights non-strictly and only prints the names it misses, so another model's .pt in the shared folder
would run with random weights, and a truncated download fails deep inside torch. Reading the names is quick:
the file is memory-mapped, not loaded.
"""
advice = CHECKPOINT_HINT.format(folder=os.path.dirname(path))
try:
state = torch.load(path, map_location="cpu", weights_only=True, mmap=True)
except Exception as exc:
raise RuntimeError(f"[NH-Nodes] {path} cannot be read as a SAM 3 checkpoint ({exc}); it may be an "
f"incomplete download.\n{advice}") from exc
if isinstance(state, dict) and isinstance(state.get("model"), dict):
state = state["model"]
names = list(state.keys()) if isinstance(state, dict) else []
del state
if not any(name.startswith("detector.") for name in names) or not any(name.startswith("tracker.") for name in names):
raise RuntimeError(f"[NH-Nodes] {path} is not a SAM 3 checkpoint (it has no SAM 3 detector and tracker "
f"weights).\n{advice}")
def load_model(checkpoint):
"""The SAM 3 image model for ``checkpoint`` (a name in the sam3 folders); kept in RAM between runs."""
path = resolve_checkpoint(checkpoint)
stamp = (os.path.normcase(os.path.abspath(path)), os.path.getmtime(path), os.path.getsize(path))
if _CACHE.get("model") is not None and _CACHE.get("stamp") == stamp:
return _CACHE["model"]
unload()
build, _ = ensure_sam3()
device = sam3_device()
check_checkpoint(path)
print(f"[NH-Nodes] Loading SAM 3 from {path}")
with torch.cuda.device(device): # the tables sam3 builds on "cuda" go to ComfyUI's GPU
model = build(bpe_path=bpe_path(), checkpoint_path=path, load_from_HF=False,
enable_inst_interactivity=True, device="cpu", eval_mode=True)
if not callable(getattr(model, "predict_inst", None)) or getattr(model, "inst_interactive_predictor", None) is None:
raise RuntimeError("[NH-Nodes] This sam3 build has no box-prompt decoder (predict_inst). " + INSTALL_HINT)
transforms = getattr(model.inst_interactive_predictor, "_transforms", None)
if getattr(transforms, "max_hole_area", 0):
# sam3 fills holes of close-up masks by default; keep real ones (between an arm and the body), as pass 1 does
transforms.max_hole_area = 0.0
model.requires_grad_(False)
_CACHE.update(model=model, stamp=stamp)
return model
def _model_device(model):
return next(model.parameters()).device
def _is_on(current, wanted):
"""True when ``current`` satisfies ``wanted``; an index-less "cuda" accepts any cuda:N."""
return current.type == wanted.type and (wanted.index is None or current.index == wanted.index)
_TABLE_ATTRIBUTES = ("freqs_cis", "freqs_cis_real", "freqs_cis_imag") # sam3's rotary tables, plain attributes
def _all_on(model, device):
return all(_is_on(p.device, device) for p in model.parameters())
def _move_tables(model, device):
"""Move sam3's rotary tables, which model.to() leaves behind because they are not buffers."""
for module in model.modules():
for name in _TABLE_ATTRIBUTES:
value = getattr(module, name, None)
if isinstance(value, torch.Tensor) and not _is_on(value.device, device):
setattr(module, name, value.to(device))
def move_model(model, device):
"""Put the weights on ``device``, asking ComfyUI to free VRAM first; back to the CPU if the move fails."""
if device.type == "cuda":
weights = 0 if _all_on(model, device) else sum(
t.numel() * t.element_size() for t in itertools.chain(model.parameters(), model.buffers()))
make_room_on_gpu(weights + _WORKING_BYTES)
try:
model.to(device)
_move_tables(model, device)
except Exception:
offload(model) # a half-moved model would fail every later run
raise
def offload(model):
"""Move the weights and sam3's GPU-side tables to the CPU, so nothing holds VRAM between runs.
The position-encoding caches sam3 fills at build time (about 200 MB on the GPU) are emptied; sam3 rebuilds them
on the GPU when it needs them again.
"""
cpu = torch.device("cpu")
if not _all_on(model, cpu):
model.to(cpu)
_move_tables(model, cpu)
for module in model.modules():
if type(module).__name__ == "PositionEmbeddingSine" and isinstance(getattr(module, "cache", None), dict):
module.cache.clear()
if torch.cuda.is_available():
torch.cuda.empty_cache()
def unload():
"""Forget the cached model (another checkpoint was chosen)."""
model = _CACHE.pop("model", None)
_CACHE.pop("stamp", None)
if model is not None:
offload(model)
def _native_bf16():
try:
return torch.cuda.is_bf16_supported(including_emulation=False)
except TypeError: # torch without the including_emulation switch
return torch.cuda.is_bf16_supported()
def _autocast(device):
"""bfloat16 autocast on GPUs with native bfloat16 (Ampere and newer), as Meta runs SAM 3; full precision elsewhere."""
if device.type == "cuda" and _native_bf16():
return torch.autocast("cuda", dtype=torch.bfloat16)
return contextlib.nullcontext()
# ---------------------------------------------------------------------------
# Geometry helpers (model-free, unit tested)
# ---------------------------------------------------------------------------
def to_logits(masks):
"""sam3 returns its masks after a sigmoid (under the key "masks_logits"); turn them back into logits.
Values outside [0, 1] are taken as logits already, in case a later sam3 returns them as the key says.
"""
masks = masks.float()
if masks.numel() and (float(masks.min()) < 0.0 or float(masks.max()) > 1.0):
return masks
masks = masks.clamp(1e-6, 1.0 - 1e-6)
return torch.log(masks) - torch.log1p(-masks)
def sample_region(logits, image_size, region, out_size):
"""Resample the square model-space map ``logits`` [S, S] over ``region`` of the image, bilinearly.
``region`` is (x1, y1, x2, y2) in pixels of an image of ``image_size`` (height, width), which the model saw
squeezed into S x S; the result is [h, w] for ``out_size`` (h, w), each value taken at the centre of its output
pixel. Runs in row chunks, so a full 8K region needs a few hundred MB at most.
"""
height, width = image_size
size = logits.shape[-1]
x1, y1, x2, y2 = region
out_h, out_w = int(out_size[0]), int(out_size[1])
device = logits.device
xs = x1 + (torch.arange(out_w, device=device, dtype=torch.float32) + 0.5) * ((x2 - x1) / out_w)
ys = y1 + (torch.arange(out_h, device=device, dtype=torch.float32) + 0.5) * ((y2 - y1) / out_h)
# model-map pixel index of each centre, as grid_sample coordinates (align_corners=True: -1/+1 = first/last centre)
grid_x = (xs * (size / width) - 0.5) * (2.0 / max(1, size - 1)) - 1.0
grid_y = (ys * (size / height) - 0.5) * (2.0 / max(1, size - 1)) - 1.0
source = logits.float()[None, None]
result = torch.empty((out_h, out_w), device=device, dtype=torch.float32)
rows = max(1, _SAMPLE_CHUNK // max(1, out_w))
for start in range(0, out_h, rows):
gy = grid_y[start:start + rows]
grid = torch.stack(torch.broadcast_tensors(grid_x[None, :], gy[:, None]), dim=-1)[None]
result[start:start + rows] = F.grid_sample(source, grid, mode="bilinear", padding_mode="border",
align_corners=True)[0, 0]
return result
def _lift(band, inner):
"""``band`` raised to ``inner`` where that is higher and ``band`` is still in the fade (above _FADE_FLOOR)."""
if torch.is_tensor(band):
return torch.where(band > _FADE_FLOOR, torch.maximum(band, inner), band)
return np.where(band > _FADE_FLOOR, np.maximum(band, inner), band)
def extend_to_border(logits, band_y, band_x, edges=("top", "bottom", "left", "right")):
"""Copy of ``logits`` [..., H, W] (torch or numpy) whose outer ``band_y`` rows / ``band_x`` columns on the given
``edges`` are lifted to the first row / column inside the band, where they are still in the fade. A mask can only
grow there, and only where the object reaches the band from inside."""
out = logits.clone() if torch.is_tensor(logits) else np.array(logits, copy=True)
height, width = out.shape[-2:]
band_y, band_x = min(int(band_y), (height - 1) // 2), min(int(band_x), (width - 1) // 2)
if band_y > 0 and "top" in edges:
out[..., :band_y, :] = _lift(out[..., :band_y, :], out[..., band_y:band_y + 1, :])
if band_y > 0 and "bottom" in edges:
out[..., height - band_y:, :] = _lift(out[..., height - band_y:, :],
out[..., height - band_y - 1:height - band_y, :])
if band_x > 0 and "left" in edges:
out[..., :, :band_x] = _lift(out[..., :, :band_x], out[..., :, band_x:band_x + 1])
if band_x > 0 and "right" in edges:
out[..., :, width - band_x:] = _lift(out[..., :, width - band_x:], out[..., :, width - band_x - 1:width - band_x])
return out
def border_band(pixels, view=_VIEW_SIZE, cells=_BORDER_CELLS):
"""Width in pixels of the faded band (``cells`` rows of SAM 3's view) along an edge ``pixels`` long that SAM 3
saw ``view`` pixels long."""
return math.ceil(cells * pixels / view)
def mask_box(mask):
"""Tight (x1, y1, x2, y2) box of a bool [H, W] tensor with exclusive ends; None when empty."""
rows = torch.nonzero(mask.any(dim=1)).flatten()
if rows.numel() == 0:
return None
cols = torch.nonzero(mask.any(dim=0)).flatten()
return int(cols[0]), int(rows[0]), int(cols[-1]) + 1, int(rows[-1]) + 1
def model_box_to_image(box, size, image_size, margin=1):
"""An S x S model-space box as an image-space region, ``margin`` model pixels wider on each side, in the image."""
height, width = image_size
sx, sy = width / size, height / size
x1, y1, x2, y2 = box
return (max(0, int((x1 - margin) * sx)), max(0, int((y1 - margin) * sy)),
min(width, int(math.ceil((x2 + margin) * sx))), min(height, int(math.ceil((y2 + margin) * sy))))
def _fit_span(center, size, limit):
size = int(round(min(size, limit)))
start = int(round(center - size / 2.0))
start = min(max(0, start), limit - size)
return start, start + size
def crop_region(box, image_size):
"""Pass-2 crop around an image-space box: a margin on each side, widened to at most 2:1, kept inside the image."""
height, width = image_size
x1, y1, x2, y2 = box
margin = max(_CROP_MIN_MARGIN, _CROP_MARGIN * max(x2 - x1, y2 - y1))
crop_w, crop_h = (x2 - x1) + 2 * margin, (y2 - y1) + 2 * margin
crop_w, crop_h = max(crop_w, crop_h / _CROP_MAX_ASPECT), max(crop_h, crop_w / _CROP_MAX_ASPECT)
left, right = _fit_span((x1 + x2) / 2.0, crop_w, width)
top, bottom = _fit_span((y1 + y2) / 2.0, crop_h, height)
# the box itself always stays inside, whatever the rounding did
return min(left, x1), min(top, y1), max(right, x2), max(bottom, y2)
def drop_stray_parts(refined, coarse):
"""``refined`` without the connected parts that do not touch ``coarse`` at all (specks around the object)."""
count, labels = cv2.connectedComponents(refined.astype(np.uint8), connectivity=8)
if count <= 2:
return refined
touching = np.bincount(labels[coarse], minlength=count) > 0
touching[0] = False
return touching[labels]
def close_up_agrees(refined, coarse, tolerance):
"""True when the close-up mask ``refined`` shows the same object as the pass-1 mask ``coarse`` (bool arrays).
The pass-1 mask is blurred by up to ``tolerance`` px, so plain IoU would turn down a sharp mask of a thin pole or
cable. Instead most of the close-up must lie within ``tolerance`` of the pass-1 mask, and it must cover most of
the pass-1 core (pixels deeper than twice ``tolerance`` inside it); thin objects have no core and need only the
first. Large masks are compared at a reduced scale.
"""
if not refined.any() or not coarse.any():
return False
step = max(1, int(math.sqrt(refined.size / _AGREE_PIXELS)))
if step > 1 and not refined[::step, ::step].any():
step = 1 # something thinner than the step: compare at full scale
refined, coarse = refined[::step, ::step], coarse[::step, ::step]
tolerance = max(1.0, tolerance / step)
to_coarse = cv2.distanceTransform((~coarse).astype(np.uint8), cv2.DIST_L2, 3)
inside = np.count_nonzero(refined & (to_coarse <= tolerance)) / np.count_nonzero(refined)
core = cv2.distanceTransform(coarse.astype(np.uint8), cv2.DIST_L2, 3) > 2.0 * tolerance
covered = np.count_nonzero(refined & core) / np.count_nonzero(core) if core.any() else 1.0
return inside >= REFINE_MIN_INSIDE and covered >= REFINE_MIN_CORE
def _box_overlap(a, b):
"""Intersection box of two (x1, y1, x2, y2) boxes, or None."""
x1, y1, x2, y2 = max(a[0], b[0]), max(a[1], b[1]), min(a[2], b[2]), min(a[3], b[3])
return (x1, y1, x2, y2) if x1 < x2 and y1 < y2 else None
def find_instances(processor, state, concept, max_instances=0, threshold=0.0, misses=None):
"""[(score, logits [S, S], model-space box)] for one concept on an encoded image, best first, duplicates dropped.
Candidates scoring under ``threshold`` are left out; the best of them goes to ``misses[concept]`` when a dict is
given, so the node can say how close a concept it did not find came. The logits are extended to the border of
the view first (SAM 3's masks fade out there, see extend_to_border).
"""
processor.reset_all_prompts(state)
output = processor.set_text_prompt(concept, state)
masks, scores = output["masks_logits"], output["scores"]
if masks is None or len(scores) == 0:
return []
scores = scores.float().flatten()
under = scores <= threshold # SAM 3 itself keeps only scores above the threshold
if misses is not None and bool(under.any()):
misses[concept] = max(misses.get(concept, 0.0), float(scores[under].max()))
band = border_band(masks.shape[-1])
found, kept = [], [] # kept: (bool mask, box, area) of accepted instances
for i in torch.argsort(scores, descending=True, stable=True).tolist():
if float(scores[i]) <= threshold:
break
logits = extend_to_border(to_logits(masks[i].reshape(masks.shape[-2], masks.shape[-1])), band, band)
inside = logits > 0
box = mask_box(inside)
if box is None:
continue
area = int(inside.sum())
duplicate = False
for other, other_box, other_area in kept:
overlap = _box_overlap(box, other_box)
if overlap is None:
continue
ox1, oy1, ox2, oy2 = overlap
inter = int((inside[oy1:oy2, ox1:ox2] & other[oy1:oy2, ox1:ox2]).sum())
if inter / max(1, area + other_area - inter) >= DEDUP_IOU:
duplicate = True
break
if duplicate:
continue
kept.append((inside, box, area))
found.append((float(scores[i]), logits, box))
if max_instances and len(found) >= max_instances:
break
return found
# ---------------------------------------------------------------------------
# Segmenting
# ---------------------------------------------------------------------------
def _hint_size(model):
"""Side of the low-resolution mask the box-prompt decoder takes as a hint (288 for SAM 3)."""
encoder = getattr(getattr(getattr(model, "inst_interactive_predictor", None), "model", None),
"sam_prompt_encoder", None)
size = getattr(encoder, "mask_input_size", (288, 288))
return int(size[0]), int(size[1])
def _upsample(concept, score, logits, box, image_size):
"""Pass 1 only: the instance's mask at full resolution, inside its own region."""
region = model_box_to_image(box, logits.shape[-1], image_size)
x1, y1, x2, y2 = region
mask = sample_region(logits, image_size, region, (y2 - y1, x2 - x1)) > 0
return make_segment(concept, score, mask, origin=(x1, y1)), False
def _refine(model, processor, image, concept, score, logits, box, image_size, hint_size):
"""Pass 2: segment the instance again on a full-resolution crop, the pass-1 mask as a hint.
Returns (segment, refined) where ``refined`` is False when the pass-1 mask was kept.
"""
target = model_box_to_image(box, logits.shape[-1], image_size)
cx1, cy1, cx2, cy2 = crop_region(target, image_size)
coarse = (sample_region(logits, image_size, (cx1, cy1, cx2, cy2), (cy2 - cy1, cx2 - cx1)) > 0).cpu().numpy()
hint = sample_region(logits, image_size, (cx1, cy1, cx2, cy2), hint_size).clamp(-20.0, 20.0)
state = processor.set_image(image[:, cy1:cy2, cx1:cx2])
x1, y1, x2, y2 = target
prompt_box = np.array([x1 - cx1, y1 - cy1, x2 - cx1, y2 - cy1], dtype=np.float32)
masks, _, _ = model.predict_inst(state, box=prompt_box, mask_input=hint[None], multimask_output=False,
return_logits=True)
logits = np.asarray(masks, dtype=np.float32).reshape(-1, cy2 - cy1, cx2 - cx1)[0]
# the close-up fades out along its own border too, which matters where that border is the image's
height, width = image_size
edges = [edge for edge, at_image_edge in (("top", cy1 == 0), ("bottom", cy2 == height), ("left", cx1 == 0),
("right", cx2 == width)) if at_image_edge]
view = int(getattr(processor, "resolution", _VIEW_SIZE))
refined = extend_to_border(logits, border_band(cy2 - cy1, view, _CLOSE_UP_CELLS),
border_band(cx2 - cx1, view, _CLOSE_UP_CELLS), edges) > 0
refined = drop_stray_parts(refined, coarse)
accepted = close_up_agrees(refined, coarse, max(image_size) / _PASS1_GRID + 2.0)
return make_segment(concept, score, refined if accepted else coarse, origin=(cx1, cy1)), accepted
def _image_chw(image):
"""One ComfyUI image [H, W, C] -> float [3, H, W] in [0, 1]."""
channels = image.shape[-1]
if channels == 1:
image = image.expand(-1, -1, 3)
elif channels not in (3, 4):
raise ValueError(f"[NH-Nodes] SAM 3 expects an IMAGE with 1, 3 or 4 channels, got {channels}")
return image[..., :3].permute(2, 0, 1).float().clamp(0.0, 1.0)
def segment_images(model, make_processor, images, concepts, threshold=0.5, detail_level="balanced",
max_per_concept=0, device=None, interrupt=None, progress=None, misses=None):
"""NH_SEGS (one SegsFrame per image) for ``concepts`` found in the ComfyUI IMAGE batch ``images``.
``make_processor(threshold)`` returns a Sam3Processor for ``model`` on ``device``; the model must already be
there. It is asked for candidates down to NEAR_MISS, so that ``misses`` (a dict, when given) learns the best score
under ``threshold`` of each concept. Instances come in concept order, best score first within a concept.
``interrupt()`` runs before each concept and each close-up (it raises to stop), ``progress(done, total)`` after
each concept of each image.
"""
device = device or _model_device(model)
refine = detail_level != "fast"
hint_size = _hint_size(model) if refine else None
frames, kept_pass1, warned_wide = [], 0, False
done, total = 0, len(images) * len(concepts)
with torch.inference_mode(), _autocast(device):
for image in images:
height, width = int(image.shape[0]), int(image.shape[1])
if not warned_wide and max(height, width) > _WIDE_ASPECT * min(height, width):
warned_wide = True
print(f"[NH-Nodes] SAM 3 sees the whole {width}x{height} picture squeezed into a square, so on images "
"wider than 4:1 it can miss small things; split such a picture into tiles for better results")
pixels = _image_chw(image).to(device)
processor = make_processor(min(threshold, NEAR_MISS))
size = int(getattr(processor, "resolution", 1008))
square = F.interpolate(pixels[None], size=(size, size), mode="bilinear", antialias=True,
align_corners=False)[0].clamp_(0.0, 1.0)
state = processor.set_image(square)
segments = []
for concept in concepts:
if interrupt is not None:
interrupt()
for score, logits, box in find_instances(processor, state, concept, max_per_concept, threshold,
misses):
if refine:
if interrupt is not None:
interrupt()
seg, refined = _refine(model, processor, pixels, concept, score, logits, box,
(height, width), hint_size)
kept_pass1 += 0 if refined else 1
else:
seg, _ = _upsample(concept, score, logits, box, (height, width))
if seg is not None:
segments.append(seg)
done += 1
if progress is not None:
progress(done, total)
frames.append(SegsFrame(height, width, tuple(segments)))
del pixels, square, state
if kept_pass1:
print(f"[NH-Nodes] SAM 3: kept the first-pass mask for {kept_pass1} instance(s) whose close-up disagreed")
return tuple(frames)
def segment(images, concepts, threshold=0.5, detail_level="balanced", max_per_concept=0, checkpoint=DEFAULT_CHECKPOINT,
keep_model_loaded=False, interrupt=None, progress=None, misses=None):
"""Load SAM 3 and run ``segment_images`` on ComfyUI's GPU; the weights go back to RAM afterwards unless kept."""
if images.shape[0] == 0:
raise ValueError("[NH-Nodes] SAM 3 got an empty image batch")
model = load_model(checkpoint)
_, processor_cls = ensure_sam3()
device = sam3_device()
try:
move_model(model, device)
return segment_images(
model, lambda value: processor_cls(model, device=device, confidence_threshold=value), images, concepts,
threshold, detail_level, max_per_concept, device, interrupt, progress, misses,
)
finally:
if not keep_model_loaded:
offload(model)
+19 -3
View File
@@ -170,20 +170,36 @@ def offload(bundle):
torch.cuda.empty_cache()
def predict_labels(bundle, image_uint8):
"""Run SegFormer on one HxWx3 uint8 RGB array -> HxW int64 label map at input size."""
def predict_labels(bundle, image_uint8, confidence=False):
"""Run SegFormer on one HxWx3 uint8 RGB array -> HxW int64 label map at input size.
With ``confidence`` it returns (labels, HxW float16 probability of each pixel's label), which lets crops of the
same picture be merged by how sure each one is (see parse_crops.people_labels).
"""
_ensure_on_device(bundle)
processor, model = bundle["processor"], bundle["model"]
height, width = image_uint8.shape[:2]
scale = min(1.0, _MAX_LOGIT_SIDE / max(height, width))
logit_size = (max(1, round(height * scale)), max(1, round(width * scale)))
# The processor squeezes the picture to its fixed size anyway; doing that here first, with area averaging, is
# many times faster on a big photo than its own resize and keeps fine detail from aliasing.
size = getattr(processor, "size", None)
if isinstance(size, dict) and "height" in size and "width" in size and (
height > 2 * size["height"] or width > 2 * size["width"]):
image_uint8 = cv2.resize(image_uint8, (size["width"], size["height"]), interpolation=cv2.INTER_AREA)
inputs = processor(images=image_uint8, return_tensors="pt")
inputs = {k: v.to(model.device) for k, v in inputs.items()}
with torch.no_grad():
logits = model(**inputs).logits
logits = F.interpolate(logits, size=logit_size, mode="bilinear", align_corners=False)
labels = logits.argmax(dim=1)[0].to(torch.int32).cpu().numpy()
best, labels = logits.float().softmax(dim=1)[0].max(dim=0)
labels = labels.to(torch.int32).cpu().numpy()
best = best.cpu().numpy() if confidence else None
if labels.shape != (height, width):
labels = cv2.resize(labels, (width, height), interpolation=cv2.INTER_NEAREST)
if best is not None:
best = cv2.resize(best, (width, height), interpolation=cv2.INTER_LINEAR)
if best is not None:
return labels.astype(np.int64), best.astype(np.float16)
return labels.astype(np.int64)
+726
View File
@@ -0,0 +1,726 @@
"""NH_SEGS: the instances the NH vision nodes pass between each other.
An ``NH_SEGS`` value is a tuple with one ``SegsFrame`` per image of a batch. A frame holds the image size and the
``Segment`` instances found in that image. A segment keeps its label, score and box, and only the mask pixels inside
its box, so dozens of instances on an 8K image stay small. Full-size masks are built only when a node outputs a MASK.
Plain torch/numpy, no model: the select, compose, preview and convert nodes and their tests run anywhere.
"""
import fnmatch
import functools
from collections import namedtuple
from dataclasses import dataclass
import cv2
import numpy as np
import torch
import torch.nn.functional as F
from PIL import Image, ImageDraw, ImageFont
from .vi_words import translate
SEGS_TYPE = "NH_SEGS"
@dataclass(frozen=True, eq=False)
class Segment:
"""One instance found in an image.
``box`` is (x1, y1, x2, y2) in image pixels with exclusive ends, so the instance covers ``image[y1:y2, x1:x2]``;
it is the tight box around the mask. ``mask`` is a bool [y2 - y1, x2 - x1] CPU tensor over that box.
"""
label: str
score: float
box: tuple
mask: torch.Tensor
source: str = "sam3"
@property
def area(self):
"""Number of mask pixels."""
return int(self.mask.sum())
@property
def center(self):
x1, y1, x2, y2 = self.box
return (x1 + x2) / 2.0, (y1 + y2) / 2.0
@dataclass(frozen=True, eq=False)
class SegsFrame:
"""The instances found in one image of ``height`` x ``width`` pixels, in the order they were found."""
height: int
width: int
segments: tuple = ()
def with_segments(self, segments):
return SegsFrame(self.height, self.width, tuple(segments))
def _comfy_cache_tensors(self):
"""The tensors this frame holds, so ComfyUI's RAM-pressure cache can size it."""
return [seg.mask for seg in self.segments]
def make_segment(label, score, mask, origin=(0, 0), source="sam3"):
"""Segment for the True pixels of the bool [h, w] ``mask`` whose top-left pixel sits at ``origin`` (x, y) of the image.
The stored box is the tight box around those pixels and the stored mask a compact CPU copy of that box. Works on
any device, cropping before the copy to the CPU. Returns None when the mask is empty.
"""
mask = torch.as_tensor(mask)
if mask.dtype != torch.bool:
mask = mask > 0.5
rows = torch.nonzero(mask.any(dim=1)).flatten()
if rows.numel() == 0:
return None
cols = torch.nonzero(mask.any(dim=0)).flatten()
y1, y2 = int(rows[0]), int(rows[-1]) + 1
x1, x2 = int(cols[0]), int(cols[-1]) + 1
crop = mask[y1:y2, x1:x2].to("cpu").clone(memory_format=torch.contiguous_format)
ox, oy = int(origin[0]), int(origin[1])
return Segment(str(label), float(score), (ox + x1, oy + y1, ox + x2, oy + y2), crop, source)
def frame_mask(frame, segments=None):
"""Bool [H, W] union of ``segments`` (every segment of ``frame`` by default)."""
canvas = torch.zeros((frame.height, frame.width), dtype=torch.bool)
for seg in frame.segments if segments is None else segments:
x1, y1, x2, y2 = seg.box
canvas[y1:y2, x1:x2] |= seg.mask
return canvas
def segs_union(segs):
"""MASK [B, H, W] float: the union of every instance, one mask per image."""
return torch.stack([frame_mask(frame) for frame in segs]).float()
def check_memory(nbytes, what):
"""Raise MemoryError with advice when ``nbytes`` would take more than half of the RAM that is free now."""
try:
import psutil
available = psutil.virtual_memory().available
except Exception: # no psutil: nothing to compare with
return
if nbytes > available // 2:
raise MemoryError(
f"[NH-Nodes] {what} would need {nbytes / 2**30:.1f} GiB but only {available / 2**30:.1f} GiB of RAM is "
"free. Keep fewer instances with Segs Select, or combine them into one mask with Segs Compose."
)
def instance_masks(segs):
"""(MASK [N, H, W] float with one full-size mask per instance of every image, in order; the labels).
With no instance at all the MASK holds a single empty mask, as nodes downstream may not accept an empty batch.
Each full-size mask takes H x W x 4 bytes (about 145 MB on an 8K image), so a MemoryError explains what to do
when they would not fit in RAM.
"""
labels = [seg.label for frame in segs for seg in frame.segments]
height, width = segs[0].height, segs[0].width
check_memory(max(1, len(labels)) * height * width * 4, f"{len(labels)} full-size instance masks")
masks = torch.zeros((max(1, len(labels)), height, width))
index = 0
for frame in segs:
for seg in frame.segments:
x1, y1, x2, y2 = seg.box
masks[index, y1:y2, x1:x2] = seg.mask.float()
index += 1
return masks, labels
def count_segments(segs):
return sum(len(frame.segments) for frame in segs)
# ---------------------------------------------------------------------------
# Labels
# ---------------------------------------------------------------------------
def normalize_label(text):
"""Lower case, '_' as a space, single spaces: "Red_Car " and "red car" both become "red car"."""
return " ".join(str(text).replace("_", " ").lower().split())
def label_matches(label, pattern):
"""True when ``label`` matches ``pattern``, ignoring case; the pattern may use * and ? wildcards."""
return fnmatch.fnmatchcase(normalize_label(label), normalize_label(pattern))
def parse_label_list(text):
"""Commas and new lines separate entries: "person, red car" -> ["person", "red car"].
Blank entries and lines starting with '#' are dropped; a repeated entry (ignoring case and spacing) is kept once,
with its first spelling.
"""
entries, seen = [], set()
for line in str(text or "").splitlines():
if line.strip().startswith("#"):
continue
for part in line.split(","):
entry = " ".join(part.split())
key = normalize_label(entry)
if entry and key not in seen:
seen.add(key)
entries.append(entry)
return entries
# ---------------------------------------------------------------------------
# Select
# ---------------------------------------------------------------------------
SORT_ORDERS = ("score", "area", "left_to_right", "right_to_left", "top_to_bottom", "bottom_to_top",
"center_first", "as_found")
def sort_segments(segments, order, frame):
"""``segments`` in ``order`` (one of SORT_ORDERS); positions use each box's centre. Ties keep their found order."""
segments = list(segments)
if order == "score":
return sorted(segments, key=lambda s: -s.score)
if order == "area":
return sorted(segments, key=lambda s: -s.area)
if order == "left_to_right":
return sorted(segments, key=lambda s: s.center[0])
if order == "right_to_left":
return sorted(segments, key=lambda s: -s.center[0])
if order == "top_to_bottom":
return sorted(segments, key=lambda s: s.center[1])
if order == "bottom_to_top":
return sorted(segments, key=lambda s: -s.center[1])
if order == "center_first":
cx, cy = frame.width / 2.0, frame.height / 2.0
return sorted(segments, key=lambda s: (s.center[0] - cx) ** 2 + (s.center[1] - cy) ** 2)
return segments
def select_segments(frame, labels=(), min_score=0.0, min_area=0.0, max_area=1.0, order="score", start=0, count=0):
"""Frame with the segments that pass the filters, sorted by ``order``, from position ``start`` (0 = first).
``labels`` are patterns (any one must match; none = every label). Areas are fractions of the image. ``count`` = 0
keeps every segment from ``start`` on.
"""
total = max(1, frame.height * frame.width)
kept = []
for seg in frame.segments:
if labels and not any(label_matches(seg.label, pattern) for pattern in labels):
continue
if seg.score < min_score:
continue
if not min_area <= seg.area / total <= max_area:
continue
kept.append(seg)
kept = sort_segments(kept, order, frame)
start = max(0, int(start))
kept = kept[start:] if count <= 0 else kept[start:start + int(count)]
return frame.with_segments(kept)
# ---------------------------------------------------------------------------
# Compose: a small boolean language over labels
# ---------------------------------------------------------------------------
class ExpressionError(ValueError):
"""An NH_SEGS mask expression that cannot be read."""
class EmptyExpressionError(ExpressionError):
"""An expression with nothing in it (blank, or only commas and comments)."""
_OPERATORS = {"+": "|", "|": "|", "-": "-", "&": "&", "~": "~", "!": "~", "(": "(", ")": ")"}
_SEPARATORS = ",\n" # list items, joined like '|' but read last
_WORD_OPERATORS = {"or": "|", "and": "&", "not": "~"}
_KEYWORDS = ("all", "image", "none", "background")
_QUOTES = "\"'"
def _is_word_char(ch):
return ch.isalnum() or ch == "_"
def _word_tokens(text, word_operators):
"""[(kind, value, position, written)] with kinds "op", "sep", "word" and "quoted"; positions count from 1.
A quote opens a quoted label only where a token starts, and a '-' between two word characters is part of the
word, so ``women's`` and ``t-shirt`` are plain words while ``person - hair`` subtracts. A '#' where a token would
start begins a comment that runs to the end of its line (inside a word, as in ``car#2``, it is part of the word).
"""
tokens, i, n = [], 0, len(text)
while i < n:
ch = text[i]
if ch == "#":
end = text.find("\n", i)
i = n if end < 0 else end
elif ch in _SEPARATORS:
tokens.append(("sep", ",", i + 1, "a new line" if ch == "\n" else ch))
i += 1
elif ch.isspace():
i += 1
elif ch in _OPERATORS:
tokens.append(("op", _OPERATORS[ch], i + 1, ch))
i += 1
elif ch in _QUOTES:
end = text.find(ch, i + 1)
if end < 0:
raise ExpressionError(f"quote opened at position {i + 1} is never closed")
tokens.append(("quoted", text[i + 1:end], i + 1, text[i:end + 1]))
i = end + 1
else:
j = i
while j < n and not text[j].isspace() and text[j] not in _SEPARATORS:
hyphen_in_word = (text[j] == "-" and _is_word_char(text[j - 1])
and j + 1 < n and _is_word_char(text[j + 1]))
if text[j] in _OPERATORS and not hyphen_in_word:
break
j += 1
word = text[i:j]
if word_operators and word.lower() in _WORD_OPERATORS:
tokens.append(("op", _WORD_OPERATORS[word.lower()], i + 1, word))
else:
tokens.append(("word", word, i + 1, word))
i = j
return tokens
def _tokenize(text, word_operators=True):
"""Tokens with each run of words joined into one "label" (or "keyword") token: ``red car`` is one label.
Labels are read through ``vi_words.translate``, so Vietnamese becomes the English the instances are named in;
the token keeps (english, status) as its value. A run that translates to all, image, none or background, unquoted,
is that keyword.
"""
raw = _word_tokens(text, word_operators)
tokens = []
for index, token in enumerate(raw):
kind, value, position, written = token
if kind == "sep" and written == "a new line":
# a formula laid out over several lines: a line break next to an operator joins, it does not list
after = next((t for t in raw[index + 1:] if not (t[0] == "sep" and t[3] == "a new line")), None)
before = next((t for t in reversed(tokens) if t[0] != "sep"), None) if tokens and tokens[-1][0] != "sep" \
else None
if (before is not None and before[0] == "op" and before[1] != ")") or (
after is not None and after[0] == "op" and after[1] in ("|", "-", "&", ")")):
continue
if kind in ("word", "quoted") and tokens and tokens[-1][0] == "run":
_, parts, start, quoted = tokens[-1]
tokens[-1] = ("run", parts + [value], start, quoted or kind == "quoted")
elif kind in ("word", "quoted"):
tokens.append(("run", [value], position, kind == "quoted"))
else:
tokens.append(token)
result = []
for token in tokens:
if token[0] != "run":
result.append(token)
continue
_, parts, position, quoted = token
written = " ".join(parts)
english, status = translate(written)
if not quoted and english.lower() in _KEYWORDS:
result.append(("keyword", english.lower(), position, written))
else:
result.append(("label", (english, status), position, written))
return result
class _Parser:
"""list := expr? (',' expr?)* ; expr := term (('|' | '-') term)* ; term := unary ('&' unary)* ;
unary := '~' unary | atom ; atom := label | keyword | '(' list ')'. Commas and new lines join the items of a list
like '|', after every other operator. Builds a tuple tree evaluated by ``evaluate``: a label node is
("label", english, written, status), a keyword node ("keyword", name, written)."""
def __init__(self, text, word_operators=True):
self.tokens = _tokenize(text, word_operators)
self.pos = 0
def _peek(self):
return self.tokens[self.pos] if self.pos < len(self.tokens) else None
def _where(self):
token = self._peek()
return f"'{token[3]}' at position {token[2]}" if token else "the end"
def parse(self):
items = self._list(nested=False)
if not items:
raise EmptyExpressionError("the expression is empty; use 'all' for every instance")
return self._union(items)
@staticmethod
def _union(items):
tree = items[0]
for item in items[1:]:
tree = ("|", tree, item)
return tree
def _list(self, nested):
"""The items of a list up to the end (or the closing ')' when ``nested``); empty items are skipped."""
items = []
while True:
token = self._peek()
if token is not None and token[0] == "sep":
self.pos += 1
continue
if token is None or (nested and token[:2] == ("op", ")")):
return items
items.append(self._expr())
token = self._peek()
if token is not None and token[0] != "sep" and not (nested and token[:2] == ("op", ")")):
raise ExpressionError(f"unexpected {self._where()}")
def _expr(self):
tree = self._term()
while (token := self._peek()) is not None and token[0] == "op" and token[1] in ("|", "-"):
self.pos += 1
tree = (token[1], tree, self._term())
return tree
def _term(self):
tree = self._unary()
while (token := self._peek()) is not None and token[0] == "op" and token[1] == "&":
self.pos += 1
tree = ("&", tree, self._unary())
return tree
def _unary(self):
token = self._peek()
if token is not None and token[0] == "op" and token[1] == "~":
self.pos += 1
return ("~", self._unary())
return self._atom()
def _atom(self):
token = self._peek()
if token is None:
raise ExpressionError("the expression ends where a label, 'all', 'image', 'none' or '(' was expected")
kind, value, position, written = token
self.pos += 1
if kind == "label":
return ("label", value[0], written, value[1])
if kind == "keyword":
return ("keyword", value, written)
if value == "(":
items = self._list(nested=True)
closing = self._peek()
if closing is None or closing[:2] != ("op", ")"):
raise ExpressionError(f"'(' at position {position} is not closed (found {self._where()})")
if not items:
raise ExpressionError(f"'(' at position {position} holds nothing")
self.pos += 1
return self._union(items)
raise ExpressionError(f"unexpected '{written}' at position {position}")
@functools.lru_cache(maxsize=64)
def parse_expression(text, word_operators=True):
"""Parse a mask expression (see ``evaluate``) into a tree; raises ExpressionError with the position on mistakes.
``word_operators=False`` reads or/and/not as words, for the natural phrases SAM 3 takes ("black and white dog").
"""
return _Parser(str(text), word_operators).parse()
def parse_concepts(text):
"""``parse_expression`` for SAM3 Segment's concepts: or/and/not are plain words there."""
return parse_expression(text, word_operators=False)
def expression_labels(tree):
"""Label patterns (in English) used in a parsed expression, in order."""
if tree[0] == "label":
return [tree[1]]
if tree[0] == "keyword":
return []
return [label for child in tree[1:] for label in expression_labels(child)]
def expression_terms(tree):
"""(written, english, status) of each label and keyword of a parsed expression, in order; status is "english",
"translated" or "unknown" (Vietnamese not in vi_words) for labels and "keyword" for keywords."""
if tree[0] == "label":
return [(tree[2], tree[1], tree[3])]
if tree[0] == "keyword":
return [(tree[2], tree[1], "keyword")]
return [term for child in tree[1:] for term in expression_terms(child)]
def uses_keyword(tree, name):
return any(status == "keyword" and english == name for _, english, status in expression_terms(tree))
def evaluate(tree, frame):
"""Bool [H, W] mask of a parsed expression for one frame.
A label (wildcards allowed; quotes or '_' for spaces; a '-' inside a word is part of it; words in a row make one
label) is the union of the matching instances, ``all`` every instance, ``background`` everything outside them,
``image`` the whole frame, ``none`` nothing. ``a | b`` (also ``+``, ``or``, a comma or a new line) is the union,
``a - b`` the difference, ``a & b`` (``and``) the intersection, ``~a`` (``!``, ``not``) everything outside ``a``.
"""
kind = tree[0]
if kind == "label":
return frame_mask(frame, [seg for seg in frame.segments if label_matches(seg.label, tree[1])])
if kind == "keyword":
if tree[1] == "all":
return frame_mask(frame)
if tree[1] == "background":
return ~frame_mask(frame)
if tree[1] == "image":
return torch.ones((frame.height, frame.width), dtype=torch.bool)
return torch.zeros((frame.height, frame.width), dtype=torch.bool)
if kind == "~":
return ~evaluate(tree[1], frame)
left, right = evaluate(tree[1], frame), evaluate(tree[2], frame)
if kind == "|":
return left | right
if kind == "&":
return left & right
return left & ~right
def expression_masks(tree, segs):
"""MASK [B, H, W] float of a parsed expression, one mask per image of ``segs``."""
return torch.stack([evaluate(tree, frame) for frame in segs]).float()
# ---------------------------------------------------------------------------
# Impact Pack SEGS
# ---------------------------------------------------------------------------
# The field names of Impact Pack's SEG, so its detailers read these items; NH-Nodes does not import Impact Pack.
IMPACT_SEG = namedtuple("SEG", ["cropped_image", "cropped_mask", "confidence", "crop_region", "bbox", "label",
"control_net_wrapper"], defaults=[None])
def _centred_span(lo, hi, factor, limit):
"""[start, end) of ``factor`` times the length of [lo, hi), centred on it and slid inside [0, limit).
It is never shorter than [lo, hi) and only shorter than asked when the image is, so it always contains [lo, hi).
"""
size = min(limit, max(hi - lo, round((hi - lo) * factor)))
start = min(max(0, round((lo + hi - size) / 2)), limit - size)
return start, start + size
def detail_crop_region(width, height, box, crop_factor):
"""[x1, y1, x2, y2] a detailer works on around ``box``: ``crop_factor`` times its size, centred, inside the image."""
rx1, rx2 = _centred_span(box[0], box[2], crop_factor, width)
ry1, ry2 = _centred_span(box[1], box[3], crop_factor, height)
return [rx1, ry1, rx2, ry2]
def to_impact_segs(frame, crop_factor=3.0):
"""Impact Pack SEGS ``((height, width), [SEG, ...])`` for one frame; each mask is placed in its crop region."""
items = []
for seg in frame.segments:
region = detail_crop_region(frame.width, frame.height, seg.box, crop_factor)
rx1, ry1, rx2, ry2 = region
x1, y1, x2, y2 = seg.box
cropped = np.zeros((ry2 - ry1, rx2 - rx1), dtype=np.float32)
cropped[y1 - ry1:y2 - ry1, x1 - rx1:x2 - rx1] = seg.mask.numpy()
items.append(IMPACT_SEG(None, cropped, float(seg.score), region, [x1, y1, x2, y2], seg.label, None))
return (frame.height, frame.width), items
# ---------------------------------------------------------------------------
# Preview
# ---------------------------------------------------------------------------
_PALETTE = (
(230, 25, 75), (60, 180, 75), (255, 225, 25), (0, 130, 200), (245, 130, 48), (145, 30, 180),
(70, 240, 240), (240, 50, 230), (210, 245, 60), (250, 190, 212), (0, 128, 128), (170, 110, 40),
)
@functools.lru_cache(maxsize=16)
def _label_font(size):
for name in ("arial.ttf", "DejaVuSans.ttf"): # both cover Vietnamese
try:
return ImageFont.truetype(name, size)
except Exception:
continue
try:
return ImageFont.load_default(size=size)
except TypeError: # Pillow < 10.1
return ImageFont.load_default()
def _free_spot(x, y, width, height, placed, limit):
"""``y`` moved down (or up near the bottom ``limit``) until a ``width`` x ``height`` label at ``x`` overlaps none
of the ``placed`` (x1, y1, x2, y2) rectangles; labels of instances sharing a top edge (person, hair) then stack."""
for _ in range(len(placed)):
hit = next((r for r in placed if x < r[2] and r[0] < x + width and y < r[3] and r[1] < y + height), None)
if hit is None:
break
y = hit[3] + 1 if hit[3] + 1 + height <= limit else max(0, hit[1] - height - 1)
return y
def render_preview(image, frame, fill=0.45):
"""uint8 HxWx3 copy of ``image`` with each instance tinted, outlined and labelled "label score"."""
canvas = np.ascontiguousarray(image[:, :, :3]).copy()
height, width = canvas.shape[:2]
if not frame.segments:
return canvas
thickness = max(1, round(min(height, width) / 700))
kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (2 * thickness + 1, 2 * thickness + 1))
for index, seg in enumerate(frame.segments):
color = np.array(_PALETTE[index % len(_PALETTE)], dtype=np.float32)
x1, y1, x2, y2 = seg.box
region = canvas[y1:y2, x1:x2]
inside = seg.mask.numpy()
region[inside] = (region[inside] * (1.0 - fill) + color * fill).astype(np.uint8)
padded = np.pad(inside.astype(np.uint8), thickness) # so a box edge counts as an outline too
edge = (padded > 0) & (cv2.erode(padded, kernel) == 0)
region[edge[thickness:-thickness, thickness:-thickness]] = color.astype(np.uint8)
font = _label_font(max(12, round(min(height, width) / 45)))
picture = Image.fromarray(canvas)
draw = ImageDraw.Draw(picture)
placed = [] # label rectangles drawn so far, so a label never hides another one
for index, seg in enumerate(frame.segments):
color = _PALETTE[index % len(_PALETTE)]
text = f"{seg.label} {seg.score:.2f}"
left, top, right, bottom = draw.textbbox((0, 0), text, font=font)
pad = max(2, (bottom - top) // 6)
tw, th = right - left + 2 * pad, bottom - top + 2 * pad
# centred over the instance's topmost pixels (a head, a roof), not at a box corner that may be far from it
top_columns = torch.nonzero(seg.mask[0]).flatten()
anchor = seg.box[0] + int(top_columns[top_columns.numel() // 2]) if top_columns.numel() else seg.box[0]
x = min(max(0, anchor - tw // 2), max(0, width - tw))
y = _free_spot(x, seg.box[1] - th if seg.box[1] - th >= 0 else seg.box[1], tw, th, placed, height)
placed.append((x, y, x + tw, y + th))
draw.rectangle((x, y, x + tw, y + th), fill=color)
dark_text = (0.299 * color[0] + 0.587 * color[1] + 0.114 * color[2]) > 150
draw.text((x + pad - left, y + pad - top), text, font=font, fill=(0, 0, 0) if dark_text else (255, 255, 255))
return np.asarray(picture)
def scale_frame(frame, scale):
"""The frame of the picture made ``scale`` times smaller (for previews): boxes scaled, masks area-resampled.
An instance too small to keep a pixel keeps its centre pixel, so it still shows up with its label.
"""
height, width = max(1, round(frame.height * scale)), max(1, round(frame.width * scale))
segments = []
for seg in frame.segments:
x1, y1, x2, y2 = seg.box
nx1, ny1 = min(width - 1, int(x1 * scale)), min(height - 1, int(y1 * scale))
nx2 = max(nx1 + 1, min(width, int(np.ceil(x2 * scale))))
ny2 = max(ny1 + 1, min(height, int(np.ceil(y2 * scale))))
small = cv2.resize(seg.mask.numpy().astype(np.uint8) * 255, (nx2 - nx1, ny2 - ny1),
interpolation=cv2.INTER_AREA) >= 128
if not small.any():
small[small.shape[0] // 2, small.shape[1] // 2] = True
segments.append(make_segment(seg.label, seg.score, small, (nx1, ny1), seg.source))
return SegsFrame(height, width, tuple(segments))
def _caption(picture, notes):
"""``picture`` (uint8 RGB) with ``notes`` written on a white strip below it, one line each, long lines wrapped."""
width = picture.shape[1]
font = _label_font(max(13, round(width / 70)))
measure = ImageDraw.Draw(Image.new("RGB", (1, 1)))
lines = []
for note in notes:
words, line = str(note).split(), ""
for word in words:
candidate = f"{line} {word}".strip()
if line and measure.textlength(candidate, font=font) > width - 16:
lines.append(line)
line = word
else:
line = candidate
lines.append(line)
left, top, right, bottom = measure.textbbox((0, 0), "Ag", font=font)
step = bottom - top + 6
strip = Image.new("RGB", (width, step * len(lines) + 10), (255, 255, 255))
draw = ImageDraw.Draw(strip)
for index, line in enumerate(lines):
draw.text((8, 5 + index * step - top), line, font=font, fill=(20, 20, 20))
return np.concatenate([picture, np.asarray(strip)], axis=0)
def preview_picture(image, frame, notes=(), max_side=1536, mask=None):
"""uint8 RGB preview of one picture, at most ``max_side`` px, its instances drawn and ``notes`` written below.
``image`` is one ComfyUI image [H, W, C] (float, 0..1). With ``mask`` (the node's [H, W] output) everything
outside it is darkened, so the preview shows what the mask holds ("background" lights up the background) while the
instances keep their outline and label. The picture is shrunk first (area averaging), so an 8K image with a
hundred instances needs a few MB more, not the full-size float preview Segs Preview makes.
"""
rgb = image[..., :3] if image.shape[-1] >= 3 else image[..., :1].expand(-1, -1, 3)
scale = min(1.0, max_side / max(frame.height, frame.width))
size = (max(1, round(frame.height * scale)), max(1, round(frame.width * scale)))
if scale < 1.0: # shrink before anything else, so no full-size copy of an 8K picture is ever made
rgb = F.interpolate(rgb.permute(2, 0, 1)[None].float(), size=size, mode="area")[0].permute(1, 2, 0)
frame = scale_frame(frame, scale)
pixels = (rgb.float().clamp(0.0, 1.0) * 255.0).round().to(torch.uint8).cpu().numpy()
if mask is not None:
held = mask.float()
if scale < 1.0:
held = F.interpolate(held[None, None], size=size, mode="area")[0, 0]
outside = held.cpu().numpy() < 0.5
pixels = pixels.copy()
pixels[outside] = (pixels[outside] * _DIM_OUTSIDE).astype(np.uint8)
picture = render_preview(pixels, frame, fill=0.2 if mask is not None else 0.45)
return _caption(picture, notes) if notes else picture
_DIM_OUTSIDE = 0.35 # brightness kept outside the mask in previews
def describe_request(tree, segs, threshold=None, misses=None, added=()):
"""Lines telling what each label and keyword of a parsed request found in ``segs``, for reports and previews.
``misses`` maps a concept to the best score its candidates reached under ``threshold``; ``added`` names concepts
the node searched for on its own (person, for background).
"""
instances = [seg for frame in segs for seg in frame.segments]
lines, seen = [], set()
for written, english, status in expression_terms(tree):
key = (status == "keyword", normalize_label(english))
if key in seen:
continue
seen.add(key)
if status == "keyword":
if english == "background":
names = list(dict.fromkeys(seg.label for seg in instances))
outside = ", ".join(names) if names else "nothing (no instance was found)"
lines.append(f"background = everything outside {outside}")
continue
name = f"{written} -> {english}" if status == "translated" else written
found = [seg for seg in instances if label_matches(seg.label, english)]
if found:
lines.append(f"{name}: {len(found)} found (best {max(seg.score for seg in found):.2f})")
elif status == "unknown":
lines.append(f"{written}: not understood; write it in English (SAM 3 reads English only)")
elif misses and english in misses and threshold is not None:
lines.append(f"{name}: none at threshold {threshold:.2f}; the best candidate scored "
f"{misses[english]:.2f}, lower the threshold to keep it")
else:
lines.append(f"{name}: none found; try another English word")
for name in added:
found = [seg for seg in instances if label_matches(seg.label, name)]
lines.append(f"{name} (searched for background): {len(found)} found")
return lines
def preview_batch(image, segs):
"""IMAGE batch with every instance of ``segs`` drawn on its image (tinted, outlined, labelled)."""
if len(segs) != image.shape[0]:
raise ValueError(
f"[NH-Nodes] The image batch has {image.shape[0]} image(s) but the segs belong to {len(segs)}; "
"connect the image the segs were found in."
)
pictures = []
for picture, frame in zip(image, segs):
if (picture.shape[0], picture.shape[1]) != (frame.height, frame.width):
raise ValueError(
f"[NH-Nodes] The image is {picture.shape[1]}x{picture.shape[0]} but the segs were found on a "
f"{frame.width}x{frame.height} image; connect the image the segs were found in."
)
rgb = picture[..., :3] if picture.shape[-1] >= 3 else picture[..., :1].expand(-1, -1, 3)
pixels = (rgb.float().clamp(0.0, 1.0) * 255.0).round().to(torch.uint8).cpu().numpy()
pictures.append(torch.from_numpy(render_preview(pixels, frame).astype(np.float32) / 255.0))
return torch.stack(pictures)
+33
View File
@@ -0,0 +1,33 @@
"""Pictures a node shows on itself in the ComfyUI editor (its "ui" images), saved to ComfyUI's temp folder.
A node returns ``{"ui": {"images": save_temp_previews(...)}, "result": (...)}`` and the editor draws the pictures
on the node, so nobody has to wire a Preview Image node to see what it did.
"""
import os
import uuid
from PIL import Image
MAX_PREVIEWS = 16 # pictures per run; a long batch shows its first ones
def save_temp_previews(pictures, prefix):
"""Save uint8 RGB arrays as JPEG files in ComfyUI's temp folder; the "ui" image entries for them.
Outside ComfyUI (tests, scripts) there is no temp folder to serve them from, and nothing is saved.
"""
try:
import folder_paths
folder = folder_paths.get_temp_directory()
except Exception:
return []
os.makedirs(folder, exist_ok=True)
entries = []
run = uuid.uuid4().hex[:12]
for index, picture in enumerate(pictures[:MAX_PREVIEWS]):
name = f"{prefix}_{run}_{index}.jpg"
Image.fromarray(picture).save(os.path.join(folder, name), quality=90)
entries.append({"filename": name, "subfolder": "", "type": "temp"})
return entries
+146
View File
@@ -0,0 +1,146 @@
"""Vietnamese words for the things NH vision nodes look for, in the English SAM 3 and the parsers understand.
SAM 3 reads English only: "tóc" finds nothing where "hair" finds the hair. ``translate`` turns the Vietnamese words
people type most for photos (people, body, clothes, accessories, rooms, buildings, nature) into English, including a
thing followed by colours or sizes ("áo sơ mi trắng" -> "white shirt"). English passes through unchanged; Vietnamese
it does not know is reported, so the node can say what to write instead.
"""
import re
import unicodedata
# Things. Longer phrases win over their first word ("cửa sổ" over "cửa"), so both may be listed.
NOUNS = {
# people and body
"người": "person", "con người": "person", "người mẫu": "person", "mọi người": "person",
"phụ nữ": "woman", "người phụ nữ": "woman", "đàn bà": "woman", "người đàn bà": "woman", "cô gái": "girl",
"con gái": "girl", "bé gái": "girl", "đàn ông": "man", "người đàn ông": "man", "chàng trai": "man",
"con trai": "boy", "bé trai": "boy",
"trẻ em": "child", "em bé": "baby", "cô dâu": "bride", "chú rể": "groom",
"cơ thể": "body", "thân người": "body", "đầu": "head",
"mặt": "face", "khuôn mặt": "face", "gương mặt": "face",
"tóc": "hair", "mái tóc": "hair", "tóc vàng": "blonde hair",
"da": "skin", "làn da": "skin", "da mặt": "face skin", "da người": "skin",
"mắt": "eye", "đôi mắt": "eye", "lông mày": "eyebrow", "chân mày": "eyebrow", "lông mi": "eyelash",
"mũi": "nose", "miệng": "mouth", "môi": "lips", "đôi môi": "lips", "răng": "teeth", "lưỡi": "tongue",
"tai": "ear", "cổ": "neck", "vai": "shoulder", "ngực": "chest", "bụng": "belly",
"tay": "arm", "cánh tay": "arm", "bàn tay": "hand", "ngón tay": "finger", "móng tay": "fingernail",
"chân": "leg", "đôi chân": "leg", "bàn chân": "foot", "móng chân": "toenail",
"râu": "beard", "ria mép": "mustache",
# clothes and accessories
"quần áo": "clothes", "trang phục": "outfit", "áo": "top", "áo sơ mi": "shirt", "áo thun": "t-shirt",
"áo phông": "t-shirt", "áo khoác": "jacket", "áo vest": "suit jacket", "áo len": "sweater",
"áo dài": "ao dai", "áo choàng": "coat", "áo măng tô": "coat", "áo ngực": "bra",
"váy": "dress", "đầm": "dress", "váy cưới": "wedding dress", "chân váy": "skirt",
"quần": "pants", "quần dài": "pants", "quần âu": "trousers", "quần jean": "jeans", "quần bò": "jeans",
"quần short": "shorts", "quần đùi": "shorts",
"giày": "shoe", "giày cao gót": "high heels", "dép": "sandal", "tất": "sock", "vớ": "sock",
"mũ": "hat", "nón": "hat", "khăn": "scarf", "khăn quàng": "scarf", "cà vạt": "tie",
"thắt lưng": "belt", "dây lưng": "belt", "túi": "bag", "túi xách": "handbag", "ví": "wallet",
"ba lô": "backpack", "balo": "backpack", "kính": "glasses", "mắt kính": "glasses",
"kính mát": "sunglasses", "kính râm": "sunglasses", "đồng hồ": "watch", "nhẫn": "ring",
"vòng cổ": "necklace", "dây chuyền": "necklace", "vòng tay": "bracelet", "lắc tay": "bracelet",
"bông tai": "earring", "khuyên tai": "earring", "hoa tai": "earring", "trang sức": "jewelry",
"găng tay": "glove", "ô": "umbrella",
# nature and outdoors
"trời": "sky", "bầu trời": "sky", "mây": "cloud", "đám mây": "cloud", "mặt trời": "sun",
"cây": "tree", "cây cối": "tree", "lá": "leaf", "lá cây": "leaf", "cỏ": "grass", "bãi cỏ": "lawn",
"hoa": "flower", "bông hoa": "flower", "bụi cây": "bush", "núi": "mountain", "đồi": "hill",
"biển": "sea", "bãi biển": "beach", "cát": "sand", "nước": "water", "sông": "river", "hồ": "lake",
"đá": "rock", "đường": "road", "con đường": "road", "vỉa hè": "sidewalk", "cầu": "bridge",
# buildings and rooms
"nhà": "house", "ngôi nhà": "house", "tòa nhà": "building", "toà nhà": "building",
"mái": "roof", "mái nhà": "roof", "tường": "wall", "bức tường": "wall", "cửa": "door",
"cửa ra vào": "door", "cửa sổ": "window", "ban công": "balcony", "cầu thang": "stairs",
"cột": "pillar", "hàng rào": "fence", "sàn": "floor", "sàn nhà": "floor", "nền nhà": "floor",
"trần": "ceiling", "trần nhà": "ceiling", "đèn": "lamp", "đèn trần": "ceiling light",
"rèm": "curtain", "rèm cửa": "curtain", "thảm": "rug", "tranh": "painting", "bức tranh": "painting",
"gương": "mirror", "kệ": "shelf", "tủ": "cabinet", "bàn": "table", "ghế": "chair",
"ghế sofa": "sofa", "giường": "bed", "gối": "pillow", "chăn": "blanket", "cây cảnh": "potted plant",
"chậu cây": "potted plant", "lọ hoa": "vase", "bình hoa": "vase", "ti vi": "television",
"tivi": "television", "máy tính": "computer", "điện thoại": "phone", "sách": "book",
# vehicles, food, objects, animals
"xe": "vehicle", "ô tô": "car", "xe hơi": "car", "xe máy": "motorcycle", "xe đạp": "bicycle",
"xe buýt": "bus", "thuyền": "boat", "máy bay": "airplane",
"cốc": "cup", "tách": "cup", "ly": "glass", "chai": "bottle", "đĩa": "plate", "bát": "bowl",
"chén": "bowl", "bánh": "cake", "đồ ăn": "food", "thức ăn": "food", "hộp": "box",
"sản phẩm": "product", "chữ": "text", "biển hiệu": "sign", "bảng hiệu": "sign",
"chó": "dog", "mèo": "cat", "chim": "bird", "ngựa": "horse", "cá": "fish",
# mask formula keywords
"nền": "background", "phông": "background", "phông nền": "background", "hậu cảnh": "background",
"tất cả": "all", "toàn ảnh": "image", "cả ảnh": "image",
}
# Words that describe a thing and come after it in Vietnamese ("áo đỏ" = "red top"). "xanh" alone is left out on
# purpose: it means blue or green, so the node asks for "xanh dương" / "xanh lá" (or English) instead.
ADJECTIVES = {
"đỏ": "red", "xanh dương": "blue", "xanh da trời": "blue", "xanh lam": "blue", "xanh lá": "green",
"xanh lá cây": "green", "xanh lục": "green", "vàng": "yellow", "trắng": "white", "đen": "black",
"hồng": "pink", "tím": "purple", "cam": "orange", "nâu": "brown", "xám": "gray", "ghi": "gray",
"bạc": "silver", "be": "beige", "kem": "cream",
"lớn": "large", "to": "large", "nhỏ": "small", "dài": "long", "ngắn": "short",
}
# Words that only count or point ("các áo" = the tops) and the word for colour ("áo màu đỏ" = a red top).
_LEADING = ("các", "những", "mấy", "cái", "chiếc", "con")
_COLOUR_WORD = "màu"
_VIETNAMESE = re.compile("[àáạảãâầấậẩẫăằắặẳẵèéẹẻẽêềếệểễìíịỉĩòóọỏõôồốộổỗơờớợởỡùúụủũưừứựửữỳýỵỷỹđ]")
def normalize(text):
"""Lower case, composed accents (what IMEs and pasted text differ in), single spaces."""
return " ".join(unicodedata.normalize("NFC", str(text)).lower().split())
def is_vietnamese(text):
"""True when ``text`` has a letter only Vietnamese uses (an accent, đ)."""
return bool(_VIETNAMESE.search(normalize(text)))
def _longest_prefix(words, table):
"""(English, number of words) for the longest leading phrase of ``words`` found in ``table``, or (None, 0)."""
for size in range(len(words), 0, -1):
english = table.get(" ".join(words[:size]))
if english is not None:
return english, size
return None, 0
def translate(text):
"""(English words, status) for one thing someone typed; status is "english", "translated" or "unknown".
A whole phrase is looked up first, then a thing followed by colours or sizes. Text without Vietnamese letters is
taken as English and returned unchanged, as is Vietnamese that is not in the lists ("unknown").
"""
phrase = normalize(text)
if not is_vietnamese(phrase) and phrase not in NOUNS:
return str(text).strip(), "english"
words = phrase.split()
english = _phrase(words)
while english is None and len(words) > 1 and words[0] in _LEADING: # "những chiếc xe": drop the counting words
words = words[1:]
english = _phrase(words)
return (english, "translated") if english is not None else (str(text).strip(), "unknown")
def _phrase(words):
"""English for a thing followed by colours or sizes, or None."""
if " ".join(words) in NOUNS:
return NOUNS[" ".join(words)]
noun, used = _longest_prefix(words, NOUNS)
if noun is None:
return None
adjectives, rest = [], words[used:]
while rest:
if rest[0] == _COLOUR_WORD:
rest = rest[1:]
continue
english, size = _longest_prefix(rest, ADJECTIVES)
if english is None:
return None
adjectives.append(english)
rest = rest[size:]
if noun == "background":
return "background" # "nền trắng": the keyword, which needs no colour
return " ".join([*adjectives, noun])
+62
View File
@@ -3,6 +3,68 @@
All notable changes to NH-Nodes. Versions follow the Comfy Registry releases.
"Behaviour change" marks fixes that can change the output of an existing workflow.
## 1.8.0 — Vision: masks from one node, faces and clothes parsed close up (2026-10-18)
No node is added (75 in total). `SAM3 Segment (NH)`, `Segs Compose (NH)`, `Segs Select (NH)`, `Portrait Segment (NH)` and `Garment Segment (NH)` change. Entries marked "behaviour change" can change what a saved workflow produces, so read them before you update. Workflows saved with 1.7.x still load and run: the new input of `SAM3 Segment (NH)` is optional and its new output comes last.
### Added
- `SAM3 Segment (NH)` returns the mask of a formula, so one node gives "the person without the bag" or "the background". In `concepts`, commas or new lines list several things (their union, applied after everything else); `+` and `|` join, `-` removes (put spaces around it), `&` keeps the overlap, `~` is everything except, parentheses group, and `background` is everything outside all the instances that were found: `person - bag`, `hair, face`, `~person`, `background`. Several words in a row are one label (`red car`, no quotes), and `#` starts a comment where a word could start. Here `or`, `and` and `not` are plain words, so a phrase such as `black and white dog` reaches SAM 3 as written, and wildcards are refused (they work in Segs Select and Segs Compose). When a request uses `background` and does not name `person`, SAM 3 also looks for `person` on its own; those instances are part of `segs` and `count`.
- `SAM3 Segment (NH)` has a new output, `report` (STRING, last): for each concept the number found and the best score, the English a Vietnamese word became, a concept that was not found with the best score it reached under `threshold`, words that were not understood, and what `background` meant.
- `SAM3 Segment (NH)` shows its result on the node (new optional input `show_preview`, on by default): the mask bright and the rest darkened, the instances outlined and labelled, and the report lines below. The preview is at most 1536 px and shows up to 16 pictures of a batch.
- Common Vietnamese words work in `SAM3 Segment (NH)`, `Segs Compose (NH)` and `Segs Select (NH)`: people, body, clothes, accessories, nature, rooms, vehicles, food and animals. A thing followed by colours or sizes is reordered (`áo đỏ` is `red top`, `tóc dài` is `long hair`, `tóc vàng` is `blonde hair`). `nền`, `phông`, `phông nền` and `hậu cảnh` (also with a colour, as in `nền trắng`) mean `background`. `xanh` after a thing is ambiguous (blue or green) and is reported: write `xanh dương` or `xanh lá`. The labels of the instances are the English words. The word list is `core/vi_words.py`.
- `Segs Compose (NH)` reads the same formula language: commas and line breaks list several items, `background` (the same as `~all`) is new, several words in a row are one label (`red car`; quotes are still allowed and keep `all`, `image`, `none` and `background` as ordinary labels), `#` starts a comment, and `or`, `and`, `not` and the wildcards work as before. When `person hair` finds no label but `person` and `hair` both exist, the node stops and suggests `person + hair`.
- `Portrait Segment (NH)` and `Garment Segment (NH)` find people with YuNet, a face detector (about 230 KB, MIT licence, from OpenCV Zoo). It is downloaded once into `models/NH-Nodes/yunet` and runs on the CPU with the OpenCV that NH-Nodes already needs. If it cannot be loaded (offline on the first run, or an OpenCV that cannot run it), the console prints one warning and the nodes parse whole pictures.
- `Portrait Segment (NH)` parses each face on close-ups. The whole picture is parsed first (as a padded square, no longer squeezed). Faces are looked for on two sizes of the picture (at most 32; when none is found, the picture gets a grey margin and is searched once more, which finds a face cut by the frame), and each one is confirmed by the parser on a tight crop. A wide crop (hair, neck) and the tight crop (eyes, eyebrows, nose, lips, only inside an ellipse around the face) then refine the whole-picture parse. A face that already fills most of the frame keeps the whole-picture parse. Half-body and group photos now get eyes, eyebrows, nose and lips, and group photos get every face.
- `Garment Segment (NH)` parses groups of people on crops. The whole picture is parsed first and kept. Groups of people (found from that parse and from faces) get crops of the model's shape (a row of crops for a wide group) that are parsed at the model's full resolution, and a crop grows when people run past its edge. A crop's labels replace the first look only where they are surer, away from the crop's edges, and only for parts on people already found, so nothing is invented on an empty backdrop.
### Changed
- (behaviour change) `SAM3 Segment (NH)`: a concept list that contains a keyword (`all`, `image`, `none`, `background`) or an operator character (`+ - & ~ ! | ( )`) now acts as a formula. Lists of plain words give the same result as before. `person, background` is the whole picture (a list is a union); `salt & pepper shaker` is the overlap of `salt` and `pepper shaker` (put it in quotes to keep one label); `person (adult)` is an error; in `car #2` the `#2` is a comment, so it finds `car`.
- (behaviour change) `Segs Compose (NH)`: a line break between two labels now lists them (it used to count as a space, which made it an error). Next to an operator a line break still joins, so formulas laid out over several lines keep working. Words in a row are one label: `person hair` used to be an error and is now a label (with the hint above when both parts exist), and `red car` needs no quotes.
- (behaviour change) `Portrait Segment (NH)` and `Garment Segment (NH)` masks change on photos where the subject does not fill the frame; they are better (see Added). The whole-picture parse of Portrait Segment, which is also the whole result when no face is found, is now a padded square instead of the picture squeezed into the model's square. A picture with a side longer than twice the model's 512 px is shrunk with area averaging before the model sees it, which is faster and keeps fine detail clean, so masks of big photos can differ slightly. On a photo longer than 4096 px the label map is worked out on a 4096 px copy and enlarged at the end (the models see 512 px at a time, so nothing is lost). Once faces are found, face parts that the whole-picture parse saw far from all of them (a reflection, a pattern) are dropped.
- `SAM3 Segment (NH)` is an output node: ComfyUI runs it even when nothing is connected to it. A copy you no longer need keeps running SAM 3 whenever its inputs change, until you mute or delete it.
- The `mask` of `SAM3 Segment (NH)` is the result of the formula. A plain list of words still gives the union of every instance. When nothing is found, the mask is empty for a plain list and the whole picture for `background` or `~person`.
- `SAM3 Segment (NH)` asks SAM 3 for candidates down to a score of 0.1 and applies `threshold` itself, so the report can say how close a missed concept came. The instances kept are the same as before: a score exactly at `threshold` is dropped, as SAM 3 does. Instances with exactly equal scores keep the order SAM 3 gave them (the sort used to be unspecified).
### Fixed
- (behaviour change) SAM 3 masks of things that run off the picture now reach the image edge. SAM 3 fades its masks along the border of the square view it works in (the last 3 of 1008 rows and columns), which left a ragged gap along the edge of a person cut by the frame. That band is now lifted where the object reaches it from inside, also in the close-ups of `balanced`. A thing that ends within the band (about 0.3% of the picture's side) above the edge is drawn down to the edge.
- (behaviour change) `Portrait Segment (NH)` found few or no eyes, eyebrows, nose and lips on half-body shots, and missed faces in group photos, because the model saw a small face squeezed into 512 px. `Garment Segment (NH)` had the same problem with small people. See Added.
### Notes
- ComfyUI's Unload Models button does not release SAM 3's VRAM while `keep_model_loaded` is on. Turn it off and run the node once, or restart ComfyUI.
- Known limit: the Fashionpedia model behind `Garment Segment (NH)` misses some tops, for example a blouse under a big hat.
### Planned
Not part of this release, and the names may change: a better garment model.
## 1.7.1 — SAM3 Segment with other SAM 3 packs installed (2026-10-18)
### Fixed
- `SAM3 Segment (NH)` failed with `FileNotFoundError: ...bpe_simple_vocab_16e6.txt.gz` when another pack's bundled copy of `sam3` was imported first. ComfyUI-RMBG puts its copy ahead of the installed package, and that copy looks for its tokenizer vocabulary one folder too high when it is not told where it is. NH-Nodes now always finds the vocabulary inside the `sam3` package that was actually imported and passes it, so the node works with Meta's package and with the copies that SAM 3 packs bundle.
## 1.7.0 — SAM 3 segmentation and instance nodes (2026-10-18)
### Added
Six nodes under **NH-Nodes → Vision** (75 nodes in total) and a new data type, `NH_SEGS`. Nothing in the existing nodes changes.
- `SAM3 Segment (NH)`: finds every instance of the concepts you name (short English noun phrases, one per line or separated by commas) with Meta's SAM 3. Outputs the instances (`segs`), their union `mask` and the `count`; `Segs Preview (NH)` shows what it found. `detail_level` `balanced` (default) segments each instance a second time on a close-up cut from the full-resolution image, so small and medium objects get crisp edges even on 4K to 8K photos; `fast` keeps the whole-image pass only. Other inputs: `threshold`, `max_per_concept`, `checkpoint` and `keep_model_loaded`.
- `NH_SEGS`: for every image of a batch, a list of instances with a label, a score, a tight box and the mask inside that box only, so dozens of instances on an 8K photo stay light. Full-size masks are built only when a node outputs a `MASK`.
- `Segs Select (NH)`: keeps instances by label (`*` and `?` wildcards), score and size, sorts them (score, area, position, centre first, as found) and takes a slice, for example the second person from the left.
- `Segs Compose (NH)`: one mask from a formula over labels: `|` `+` `or` (union), `-` (minus), `&` `and` (overlap), `~` `!` `not` (everything except), parentheses, and `all`, `image`, `none`. `~person` is the background, `(wall | ceiling) - window` the walls without the windows. A hyphen inside a word belongs to the label (`t-shirt` needs no quotes), so put spaces around a minus: `person - hair`.
- `Segs To Masks (NH)`: one mask per instance, the union per image, the labels and the count. It stops with a message when the masks would need more than about half of the free RAM, for example hundreds of instances on an 8K photo.
- `Segs To Impact SEGS (NH)`: Impact Pack's `SEGS` for one image of the batch (`crop_factor` 3.0 like Impact Pack). Impact Pack is not needed for anything else in NH-Nodes.
- `Segs Preview (NH)`: draws the instances (tint, outline, label and score) on the picture they were found in.
### Notes
- SAM3 Segment needs an NVIDIA GPU with CUDA: the official sam3 code builds part of the model on the GPU, so there is no CPU mode.
- NH-Nodes does not install or download anything for it. You install Meta's `sam3` package yourself (the Vision guide gives the command, pinned to the commit NH-Nodes was tested with; newer commits with the same image API also work), and the weights are gated on Hugging Face: request access, accept Meta's SAM License and put `sam3.pt` in `models/sam3` (or a folder registered as `sam3` in `extra_model_paths.yaml`). The folder is shared with other SAM 3 packs, so an existing copy is found. The licence is in `THIRD_PARTY_NOTICES.md`.
- The model stays in system memory between runs and moves to the GPU only while the node runs, unless `keep_model_loaded` is on.
- `detail_level` has two levels for now, `balanced` and `fast`.
- SAM 3 describes whatever it looks at on a grid of about 288 cells. A subject that fills most of an 8K frame therefore gets edges at about 1/300 of its size: fine for the outline of a person, soft on hair, lace and fur. Hair-level edges are not part of this release.
- SAM 3 first sees the whole picture squeezed into a square, so on a picture wider or taller than 4:1 it can miss small things. The console says so; split such a picture into tiles with `Image Tile (NH)`.
- A file in the `sam3` folder that is not a SAM 3 checkpoint, or an incomplete download, is refused with the download advice before anything runs.
### Planned
Not part of this release, and the names may change: scene parsing that labels the whole picture (sky, wall, floor…), automatic concept tagging with ready presets and a report, matte refinement for hair and fur edges (it will add a `max` detail level), and parsing of each person's parts.
## 1.6.2 — Load images matching picks the right group
- `Load images matching` (behaviour change): when the source and the matching library share parent folders (for example
+101 -5
View File
@@ -1,14 +1,15 @@
"""Build the example pictures of the NH-Nodes guides by running the real node code on open-licensed sample photos.
Run with ComfyUI's Python from the ComfyUI root:
python custom_nodes/NH-Nodes/docs/_tools/build_examples.py [assets_dir] [--only a.jpg,b.jpg]
python custom_nodes/NH-Nodes/docs/_tools/build_examples.py [assets_dir] [--only a.jpg,b.jpg] [--gpu]
It writes <assets_dir>/examples/*.jpg and <assets_dir>/banner.jpg (default assets_dir: docs/assets of the pack).
Every picture is made from the live node classes, called the way ComfyUI calls them, so the examples cannot drift
from the code: rerun the script after a node changes. The photos are the public-domain / CC0 sample images that ship
with scikit-image (astronaut, coffee, chelsea, rocket). The script is deterministic and runs on the CPU only: the GPU
is hidden before torch loads, so a ComfyUI server that is running at the same time is never disturbed. An example whose
models are missing is skipped (and reported); nothing is downloaded.
is hidden before torch loads, so a ComfyUI server that is running at the same time is never disturbed. The one
exception is the SAM 3 example, which needs a CUDA GPU: it is built only with --gpu, which leaves the GPU visible (SAM 3
goes back to RAM after its run). An example whose models are missing is skipped (and reported); nothing is downloaded.
"""
import argparse
@@ -23,7 +24,9 @@ import warnings
from dataclasses import dataclass
from pathlib import Path
os.environ["CUDA_VISIBLE_DEVICES"] = "-1" # CPU only. (An empty value would be dropped by Windows, so use -1.)
GPU = "--gpu" in sys.argv[1:] # read before torch loads: only the SAM 3 example needs the GPU
if not GPU:
os.environ["CUDA_VISIBLE_DEVICES"] = "-1" # CPU only. (An empty value would be dropped by Windows, so use -1.)
warnings.filterwarnings("ignore")
if hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
@@ -46,7 +49,7 @@ from PIL import Image, ImageChops, ImageDraw, ImageFilter, ImageFont # noqa: E4
from comfy.cli_args import args # noqa: E402
args.cpu = True # before the pack imports comfy.model_management
args.cpu = not GPU # before the pack imports comfy.model_management
torch.set_num_threads(max(1, min(8, os.cpu_count() or 1))) # leave the rest of the machine to its other users
PACK_DIR = COMFY_ROOT / "custom_nodes" / "NH-Nodes"
@@ -1056,6 +1059,98 @@ def ex_vton_masks():
"pose, body parsing and the try-on mask of a person photo")
SAM3_CONCEPTS = "person, helmet, face"
def require_sam3():
"""Skip the SAM 3 examples unless they can run here: --gpu, the sam3 package and models/sam3/sam3.pt."""
if not GPU or not torch.cuda.is_available():
raise Skip("SAM 3 needs a CUDA GPU: run with --gpu")
if importlib.util.find_spec("sam3") is None:
raise Skip("the sam3 package is not installed")
try:
importlib.import_module("NH-Nodes.core.sam3_backend").resolve_checkpoint("sam3.pt")
except RuntimeError:
raise Skip("models/sam3/sam3.pt is missing") from None
@example("sam3-segment.jpg")
def ex_sam3_segment():
require_sam3()
image = photo("astronaut")
segs, _mask, count, _report = run("NH_SAM3Segment", image=to_tensor(image), concepts=SAM3_CONCEPTS,
threshold=0.5, detail_level="balanced", max_per_concept=0,
checkpoint="sam3.pt", keep_model_loaded=False, show_preview=False)
(preview,) = run("NH_SegsPreview", image=to_tensor(image), segs=segs)
def compose(expression):
(mask,) = run("NH_SegsCompose", segs=segs, expression=expression)
return mask_array(mask)
_biggest, biggest_mask, biggest_count = run("NH_SegsSelect", segs=segs, labels="helmet", min_score=0.0,
min_area=0.0, max_area=1.0, sort_by="area", start=0, count=1)
if biggest_count != 1:
raise RuntimeError("SAM 3 found no helmet in the astronaut photo")
top = Row(
[Tile(image, "image", "input photo"), ARROW,
Tile(to_pil(preview), "Segs Preview", f"concepts: {SAM3_CONCEPTS}")],
"SAM3 Segment (NH) → Segs Preview (NH)", f"{count} instances · threshold 0.5 · detail_level balanced",
max_h=470,
)
bottom = Row(
[Tile(masked(image, Layer(compose("person - face"), TEAL, 0.45, 2.0)), "person - face",
"Segs Compose: the person without the face", aspect=1.0),
Tile(masked(image, Layer(compose("~person"), TEAL, 0.45, 2.0)), "~person",
"Segs Compose: everything except the person", aspect=1.0),
Tile(masked(image, Layer(compose("helmet"), TEAL, 0.0, 1.6), Layer(mask_array(biggest_mask), AMBER, 0.5, 2.2)),
"Segs Select", "labels helmet · sort_by area · count 1: the biggest helmet", aspect=1.0)],
"Segs Compose and Segs Select (NH)", "masks from the labels", max_h=330,
)
return render_panel([top, bottom])
@example("sam3-formula.jpg")
def ex_sam3_formula():
"""What SAM3 Segment shows on the node itself: the picture its ui images hold, for three formulas."""
require_sam3()
import tempfile
import folder_paths
image = photo("astronaut")
formulas = ("background", "person - face", "hair, face", "~(person + helmet)")
shown = {}
with tempfile.TemporaryDirectory() as folder:
previous = folder_paths.get_temp_directory()
folder_paths.set_temp_directory(folder) # the node saves its preview into ComfyUI's temp folder: use ours
try:
for index, concepts in enumerate(formulas):
result = run("NH_SAM3Segment", image=to_tensor(image), concepts=concepts, threshold=0.5,
detail_level="balanced", max_per_concept=0, checkpoint="sam3.pt",
keep_model_loaded=index < len(formulas) - 1, show_preview=True)
with Image.open(Path(folder) / result["ui"]["images"][0]["filename"]) as picture:
shown[concepts] = picture.convert("RGB")
finally:
folder_paths.set_temp_directory(previous)
def picture_only(concepts): # the report is written on a strip under the picture: keep the picture
return shown[concepts].crop((0, 0, image.width, image.height))
top = Row(
[Tile(image, "image", "input photo"), ARROW,
Tile(shown["background"], "concepts: background", "everything outside the person; the report is under the picture")],
"SAM3 Segment (NH)", "what the node shows on itself: the mask is bright, the rest darkened", max_h=470,
)
bottom = Row(
[Tile(picture_only("person - face"), "person - face", "the person without the face", aspect=1.0),
Tile(picture_only("hair, face"), "hair, face", "a list: the hair and the face together", aspect=1.0),
Tile(picture_only("~(person + helmet)"), "~(person + helmet)", "everything except the person and the helmets",
aspect=1.0)],
"More formulas", "one SAM3 Segment (NH) each; the bright part is its mask output", max_h=330,
)
return render_panel([top, bottom])
# ---------------------------------------------------------------------------------------------------------------
# Banner
# ---------------------------------------------------------------------------------------------------------------
@@ -1204,6 +1299,7 @@ def main():
parser = argparse.ArgumentParser(description=__doc__.split("\n")[0])
parser.add_argument("assets_dir", nargs="?", default=str(PACK_DIR / "docs" / "assets"))
parser.add_argument("--only", default="", help="comma separated file names to build (default: all)")
parser.add_argument("--gpu", action="store_true", help="leave the GPU visible so the SAM 3 example is built too")
options = parser.parse_args()
assets = Path(options.assets_dir)
wanted = {name.strip() for name in options.only.split(",") if name.strip()}
Binary file not shown.

Before

Width:  |  Height:  |  Size: 114 KiB

After

Width:  |  Height:  |  Size: 114 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 81 KiB

After

Width:  |  Height:  |  Size: 81 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 170 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 193 KiB

+48
View File
@@ -0,0 +1,48 @@
<svg xmlns="http://www.w3.org/2000/svg" width="360" height="389" viewBox="0 0 360 389" font-family="Inter, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif" role="img" aria-label="SAM3 Segment (NH) node">
<rect x="0.5" y="0.5" width="359" height="388" rx="10" fill="#26282c" stroke="#45484f"/>
<path d="M0.5 10.5 a10 10 0 0 1 10 -10 h339 a10 10 0 0 1 10 10 v36 h-359 z" fill="#34373d"/>
<circle cx="17" cy="17" r="4.5" fill="#F6AD55"/>
<text x="28" y="22" font-size="14" font-weight="600" fill="#EEF0F3">SAM3 Segment (NH)</text>
<text x="28" y="38" font-size="10.5" fill="#9AA0A8" font-family="'JetBrains Mono', Consolas, 'Courier New', monospace">NH_SAM3Segment · Vision</text>
<circle cx="12" cy="67.0" r="5" fill="#64B5F6" stroke="#1b1c1f" stroke-width="1.5"/>
<text x="24" y="71.0" font-size="12" fill="#D7DADF">image</text>
<text x="336" y="71.0" font-size="12" fill="#D7DADF" text-anchor="end">segs</text>
<circle cx="348" cy="67.0" r="5" fill="#E0B272" stroke="#1b1c1f" stroke-width="1.5"/>
<text x="336" y="93.0" font-size="12" fill="#D7DADF" text-anchor="end">mask</text>
<circle cx="348" cy="89.0" r="5" fill="#81C784" stroke="#1b1c1f" stroke-width="1.5"/>
<text x="336" y="115.0" font-size="12" fill="#D7DADF" text-anchor="end">count</text>
<circle cx="348" cy="111.0" r="5" fill="#9CC2E5" stroke="#1b1c1f" stroke-width="1.5"/>
<text x="336" y="137.0" font-size="12" fill="#D7DADF" text-anchor="end">report</text>
<circle cx="348" cy="133.0" r="5" fill="#C5E1A5" stroke="#1b1c1f" stroke-width="1.5"/>
<rect x="12" y="152" width="336" height="46" rx="7" fill="#1d1f22" stroke="#3a3d43"/>
<text x="22" y="168" font-size="11.5" fill="#9AA0A8">concepts</text>
<text x="22" y="188" font-size="11.5" fill="#E6E8EB" font-family="'JetBrains Mono', Consolas, 'Courier New', monospace">person</text>
<rect x="12" y="203" width="336" height="24" rx="7" fill="#1d1f22" stroke="#3a3d43"/>
<text x="20" y="219" font-size="10" fill="#6f747c">◀</text>
<text x="332" y="219" font-size="10" fill="#6f747c">▶</text>
<text x="34" y="219" font-size="11.5" fill="#9AA0A8">threshold</text>
<text x="326" y="219" font-size="11.5" fill="#E6E8EB" text-anchor="end">0.50</text>
<rect x="12" y="232" width="336" height="24" rx="7" fill="#1d1f22" stroke="#3a3d43"/>
<text x="20" y="248" font-size="10" fill="#6f747c">◀</text>
<text x="332" y="248" font-size="10" fill="#6f747c">▶</text>
<text x="34" y="248" font-size="11.5" fill="#9AA0A8">detail_level</text>
<text x="326" y="248" font-size="11.5" fill="#E6E8EB" text-anchor="end">balanced</text>
<rect x="12" y="261" width="336" height="24" rx="7" fill="#1d1f22" stroke="#3a3d43"/>
<text x="20" y="277" font-size="10" fill="#6f747c">◀</text>
<text x="332" y="277" font-size="10" fill="#6f747c">▶</text>
<text x="34" y="277" font-size="11.5" fill="#9AA0A8">max_per_concept</text>
<text x="326" y="277" font-size="11.5" fill="#E6E8EB" text-anchor="end">0</text>
<rect x="12" y="290" width="336" height="24" rx="7" fill="#1d1f22" stroke="#3a3d43"/>
<text x="20" y="306" font-size="10" fill="#6f747c">◀</text>
<text x="332" y="306" font-size="10" fill="#6f747c">▶</text>
<text x="34" y="306" font-size="11.5" fill="#9AA0A8">checkpoint</text>
<text x="326" y="306" font-size="11.5" fill="#E6E8EB" text-anchor="end">(choose a file)</text>
<rect x="12" y="319" width="336" height="24" rx="7" fill="#1d1f22" stroke="#3a3d43"/>
<text x="22" y="335" font-size="11.5" fill="#9AA0A8">keep_model_loaded</text>
<rect x="308" y="325" width="30" height="13" rx="6.5" fill="#4a4d53"/>
<circle cx="315" cy="331.5" r="5" fill="#F5F6F7"/>
<rect x="12" y="348" width="336" height="24" rx="7" fill="#1d1f22" stroke="#3a3d43"/>
<text x="22" y="364" font-size="11.5" fill="#9AA0A8">show_preview <tspan fill="#6f747c" font-style="italic">opt</tspan></text>
<rect x="308" y="354" width="30" height="13" rx="6.5" fill="#4C8BF5"/>
<circle cx="331" cy="360.5" r="5" fill="#F5F6F7"/>
</svg>

After

Width:  |  Height:  |  Size: 3.8 KiB

+14
View File
@@ -0,0 +1,14 @@
<svg xmlns="http://www.w3.org/2000/svg" width="360" height="149" viewBox="0 0 360 149" font-family="Inter, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif" role="img" aria-label="Segs Compose (NH) node">
<rect x="0.5" y="0.5" width="359" height="148" rx="10" fill="#26282c" stroke="#45484f"/>
<path d="M0.5 10.5 a10 10 0 0 1 10 -10 h339 a10 10 0 0 1 10 10 v36 h-359 z" fill="#34373d"/>
<circle cx="17" cy="17" r="4.5" fill="#8FD18F"/>
<text x="28" y="22" font-size="14" font-weight="600" fill="#EEF0F3">Segs Compose (NH)</text>
<text x="28" y="38" font-size="10.5" fill="#9AA0A8" font-family="'JetBrains Mono', Consolas, 'Courier New', monospace">NH_SegsCompose · Vision</text>
<circle cx="12" cy="67.0" r="5" fill="#E0B272" stroke="#1b1c1f" stroke-width="1.5"/>
<text x="24" y="71.0" font-size="12" fill="#D7DADF">segs</text>
<text x="336" y="71.0" font-size="12" fill="#D7DADF" text-anchor="end">mask</text>
<circle cx="348" cy="67.0" r="5" fill="#81C784" stroke="#1b1c1f" stroke-width="1.5"/>
<rect x="12" y="86" width="336" height="46" rx="7" fill="#1d1f22" stroke="#3a3d43"/>
<text x="22" y="102" font-size="11.5" fill="#9AA0A8">expression</text>
<text x="22" y="122" font-size="11.5" fill="#E6E8EB" font-family="'JetBrains Mono', Consolas, 'Courier New', monospace">all</text>
</svg>

After

Width:  |  Height:  |  Size: 1.3 KiB

+13
View File
@@ -0,0 +1,13 @@
<svg xmlns="http://www.w3.org/2000/svg" width="360" height="112" viewBox="0 0 360 112" font-family="Inter, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif" role="img" aria-label="Segs Preview (NH) node">
<rect x="0.5" y="0.5" width="359" height="111" rx="10" fill="#26282c" stroke="#45484f"/>
<path d="M0.5 10.5 a10 10 0 0 1 10 -10 h339 a10 10 0 0 1 10 10 v36 h-359 z" fill="#34373d"/>
<circle cx="17" cy="17" r="4.5" fill="#8FD18F"/>
<text x="28" y="22" font-size="14" font-weight="600" fill="#EEF0F3">Segs Preview (NH)</text>
<text x="28" y="38" font-size="10.5" fill="#9AA0A8" font-family="'JetBrains Mono', Consolas, 'Courier New', monospace">NH_SegsPreview · Vision</text>
<circle cx="12" cy="67.0" r="5" fill="#64B5F6" stroke="#1b1c1f" stroke-width="1.5"/>
<text x="24" y="71.0" font-size="12" fill="#D7DADF">image</text>
<text x="336" y="71.0" font-size="12" fill="#D7DADF" text-anchor="end">preview</text>
<circle cx="348" cy="67.0" r="5" fill="#64B5F6" stroke="#1b1c1f" stroke-width="1.5"/>
<circle cx="12" cy="89.0" r="5" fill="#E0B272" stroke="#1b1c1f" stroke-width="1.5"/>
<text x="24" y="93.0" font-size="12" fill="#D7DADF">segs</text>
</svg>

After

Width:  |  Height:  |  Size: 1.1 KiB

+48
View File
@@ -0,0 +1,48 @@
<svg xmlns="http://www.w3.org/2000/svg" width="360" height="345" viewBox="0 0 360 345" font-family="Inter, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif" role="img" aria-label="Segs Select (NH) node">
<rect x="0.5" y="0.5" width="359" height="344" rx="10" fill="#26282c" stroke="#45484f"/>
<path d="M0.5 10.5 a10 10 0 0 1 10 -10 h339 a10 10 0 0 1 10 10 v36 h-359 z" fill="#34373d"/>
<circle cx="17" cy="17" r="4.5" fill="#8FD18F"/>
<text x="28" y="22" font-size="14" font-weight="600" fill="#EEF0F3">Segs Select (NH)</text>
<text x="28" y="38" font-size="10.5" fill="#9AA0A8" font-family="'JetBrains Mono', Consolas, 'Courier New', monospace">NH_SegsSelect · Vision</text>
<circle cx="12" cy="67.0" r="5" fill="#E0B272" stroke="#1b1c1f" stroke-width="1.5"/>
<text x="24" y="71.0" font-size="12" fill="#D7DADF">segs</text>
<text x="336" y="71.0" font-size="12" fill="#D7DADF" text-anchor="end">segs</text>
<circle cx="348" cy="67.0" r="5" fill="#E0B272" stroke="#1b1c1f" stroke-width="1.5"/>
<text x="336" y="93.0" font-size="12" fill="#D7DADF" text-anchor="end">mask</text>
<circle cx="348" cy="89.0" r="5" fill="#81C784" stroke="#1b1c1f" stroke-width="1.5"/>
<text x="336" y="115.0" font-size="12" fill="#D7DADF" text-anchor="end">count</text>
<circle cx="348" cy="111.0" r="5" fill="#9CC2E5" stroke="#1b1c1f" stroke-width="1.5"/>
<rect x="12" y="130" width="336" height="24" rx="7" fill="#1d1f22" stroke="#3a3d43"/>
<text x="22" y="146" font-size="11.5" fill="#9AA0A8">labels</text>
<text x="338" y="146" font-size="11.5" fill="#E6E8EB" text-anchor="end"></text>
<rect x="12" y="159" width="336" height="24" rx="7" fill="#1d1f22" stroke="#3a3d43"/>
<text x="20" y="175" font-size="10" fill="#6f747c">◀</text>
<text x="332" y="175" font-size="10" fill="#6f747c">▶</text>
<text x="34" y="175" font-size="11.5" fill="#9AA0A8">min_score</text>
<text x="326" y="175" font-size="11.5" fill="#E6E8EB" text-anchor="end">0.00</text>
<rect x="12" y="188" width="336" height="24" rx="7" fill="#1d1f22" stroke="#3a3d43"/>
<text x="20" y="204" font-size="10" fill="#6f747c">◀</text>
<text x="332" y="204" font-size="10" fill="#6f747c">▶</text>
<text x="34" y="204" font-size="11.5" fill="#9AA0A8">min_area</text>
<text x="326" y="204" font-size="11.5" fill="#E6E8EB" text-anchor="end">0.0000</text>
<rect x="12" y="217" width="336" height="24" rx="7" fill="#1d1f22" stroke="#3a3d43"/>
<text x="20" y="233" font-size="10" fill="#6f747c">◀</text>
<text x="332" y="233" font-size="10" fill="#6f747c">▶</text>
<text x="34" y="233" font-size="11.5" fill="#9AA0A8">max_area</text>
<text x="326" y="233" font-size="11.5" fill="#E6E8EB" text-anchor="end">1.0000</text>
<rect x="12" y="246" width="336" height="24" rx="7" fill="#1d1f22" stroke="#3a3d43"/>
<text x="20" y="262" font-size="10" fill="#6f747c">◀</text>
<text x="332" y="262" font-size="10" fill="#6f747c">▶</text>
<text x="34" y="262" font-size="11.5" fill="#9AA0A8">sort_by</text>
<text x="326" y="262" font-size="11.5" fill="#E6E8EB" text-anchor="end">score</text>
<rect x="12" y="275" width="336" height="24" rx="7" fill="#1d1f22" stroke="#3a3d43"/>
<text x="20" y="291" font-size="10" fill="#6f747c">◀</text>
<text x="332" y="291" font-size="10" fill="#6f747c">▶</text>
<text x="34" y="291" font-size="11.5" fill="#9AA0A8">start</text>
<text x="326" y="291" font-size="11.5" fill="#E6E8EB" text-anchor="end">0</text>
<rect x="12" y="304" width="336" height="24" rx="7" fill="#1d1f22" stroke="#3a3d43"/>
<text x="20" y="320" font-size="10" fill="#6f747c">◀</text>
<text x="332" y="320" font-size="10" fill="#6f747c">▶</text>
<text x="34" y="320" font-size="11.5" fill="#9AA0A8">count</text>
<text x="326" y="320" font-size="11.5" fill="#E6E8EB" text-anchor="end">0</text>
</svg>

After

Width:  |  Height:  |  Size: 3.7 KiB

+21
View File
@@ -0,0 +1,21 @@
<svg xmlns="http://www.w3.org/2000/svg" width="360" height="156" viewBox="0 0 360 156" font-family="Inter, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif" role="img" aria-label="Segs To Impact SEGS (NH) node">
<rect x="0.5" y="0.5" width="359" height="155" rx="10" fill="#26282c" stroke="#45484f"/>
<path d="M0.5 10.5 a10 10 0 0 1 10 -10 h339 a10 10 0 0 1 10 10 v36 h-359 z" fill="#34373d"/>
<circle cx="17" cy="17" r="4.5" fill="#8FD18F"/>
<text x="28" y="22" font-size="14" font-weight="600" fill="#EEF0F3">Segs To Impact SEGS (NH)</text>
<text x="28" y="38" font-size="10.5" fill="#9AA0A8" font-family="'JetBrains Mono', Consolas, 'Courier New', monospace">NH_SegsToImpactSEGS · Vision</text>
<circle cx="12" cy="67.0" r="5" fill="#E0B272" stroke="#1b1c1f" stroke-width="1.5"/>
<text x="24" y="71.0" font-size="12" fill="#D7DADF">segs</text>
<text x="336" y="71.0" font-size="12" fill="#D7DADF" text-anchor="end">segs</text>
<circle cx="348" cy="67.0" r="5" fill="#E0B272" stroke="#1b1c1f" stroke-width="1.5"/>
<rect x="12" y="86" width="336" height="24" rx="7" fill="#1d1f22" stroke="#3a3d43"/>
<text x="20" y="102" font-size="10" fill="#6f747c">◀</text>
<text x="332" y="102" font-size="10" fill="#6f747c">▶</text>
<text x="34" y="102" font-size="11.5" fill="#9AA0A8">crop_factor</text>
<text x="326" y="102" font-size="11.5" fill="#E6E8EB" text-anchor="end">3.0</text>
<rect x="12" y="115" width="336" height="24" rx="7" fill="#1d1f22" stroke="#3a3d43"/>
<text x="20" y="131" font-size="10" fill="#6f747c">◀</text>
<text x="332" y="131" font-size="10" fill="#6f747c">▶</text>
<text x="34" y="131" font-size="11.5" fill="#9AA0A8">batch_index</text>
<text x="326" y="131" font-size="11.5" fill="#E6E8EB" text-anchor="end">0</text>
</svg>

After

Width:  |  Height:  |  Size: 1.7 KiB

+17
View File
@@ -0,0 +1,17 @@
<svg xmlns="http://www.w3.org/2000/svg" width="360" height="156" viewBox="0 0 360 156" font-family="Inter, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif" role="img" aria-label="Segs To Masks (NH) node">
<rect x="0.5" y="0.5" width="359" height="155" rx="10" fill="#26282c" stroke="#45484f"/>
<path d="M0.5 10.5 a10 10 0 0 1 10 -10 h339 a10 10 0 0 1 10 10 v36 h-359 z" fill="#34373d"/>
<circle cx="17" cy="17" r="4.5" fill="#8FD18F"/>
<text x="28" y="22" font-size="14" font-weight="600" fill="#EEF0F3">Segs To Masks (NH)</text>
<text x="28" y="38" font-size="10.5" fill="#9AA0A8" font-family="'JetBrains Mono', Consolas, 'Courier New', monospace">NH_SegsToMasks · Vision</text>
<circle cx="12" cy="67.0" r="5" fill="#E0B272" stroke="#1b1c1f" stroke-width="1.5"/>
<text x="24" y="71.0" font-size="12" fill="#D7DADF">segs</text>
<text x="336" y="71.0" font-size="12" fill="#D7DADF" text-anchor="end">masks</text>
<circle cx="348" cy="67.0" r="5" fill="#81C784" stroke="#1b1c1f" stroke-width="1.5"/>
<text x="336" y="93.0" font-size="12" fill="#D7DADF" text-anchor="end">union</text>
<circle cx="348" cy="89.0" r="5" fill="#81C784" stroke="#1b1c1f" stroke-width="1.5"/>
<text x="336" y="115.0" font-size="12" fill="#D7DADF" text-anchor="end">labels</text>
<circle cx="348" cy="111.0" r="5" fill="#C5E1A5" stroke="#1b1c1f" stroke-width="1.5"/>
<text x="336" y="137.0" font-size="12" fill="#D7DADF" text-anchor="end">count</text>
<circle cx="348" cy="133.0" r="5" fill="#9CC2E5" stroke="#1b1c1f" stroke-width="1.5"/>
</svg>

After

Width:  |  Height:  |  Size: 1.5 KiB

+36
View File
@@ -311,6 +311,12 @@
"category": "NH-Nodes/Resolution",
"deprecated": false
},
{
"id": "NH_SAM3Segment",
"name": "SAM3 Segment (NH)",
"category": "NH-Nodes/Vision",
"deprecated": false
},
{
"id": "NH_SaveImagePath",
"name": "Save Image Path (NH)",
@@ -323,6 +329,36 @@
"category": "NH-Nodes/Image",
"deprecated": false
},
{
"id": "NH_SegsCompose",
"name": "Segs Compose (NH)",
"category": "NH-Nodes/Vision",
"deprecated": false
},
{
"id": "NH_SegsPreview",
"name": "Segs Preview (NH)",
"category": "NH-Nodes/Vision",
"deprecated": false
},
{
"id": "NH_SegsSelect",
"name": "Segs Select (NH)",
"category": "NH-Nodes/Vision",
"deprecated": false
},
{
"id": "NH_SegsToImpactSEGS",
"name": "Segs To Impact SEGS (NH)",
"category": "NH-Nodes/Vision",
"deprecated": false
},
{
"id": "NH_SegsToMasks",
"name": "Segs To Masks (NH)",
"category": "NH-Nodes/Vision",
"deprecated": false
},
{
"id": "NH_SimpleFacePaste",
"name": "NH Simple Face Paste",
+15 -7
View File
@@ -4,13 +4,17 @@
# NH-Nodes guide
NH-Nodes adds **69 nodes** to ComfyUI for photo and production work: loading and saving image sets, tiling and compositing,
resolution management for current models, mask tools, garment and portrait segmentation, VTON masks, logic routing,
prompt and text tools, batch helpers and indexed model loaders.
NH-Nodes adds **75 nodes** to ComfyUI for photo and production work: loading and saving image sets, tiling and compositing,
resolution management for current models, mask tools, garment and portrait segmentation, SAM 3 masks for anything you
can name from a short formula, VTON masks, logic routing, prompt and text tools, batch helpers and indexed model loaders.
This guide explains every node — what it is for, each input and output, and how it behaves in the details that matter.
Installation and a quick start are on the [project page](../../README.md#installation).
> **Requirements for SAM 3:** SAM3 Segment needs an NVIDIA GPU with CUDA, Meta's `sam3` package and the `sam3.pt`
> checkpoint, which you install and download yourself. Every other node works without them. Setup:
> [Before you start with SAM 3](vision.md#before-you-start-with-sam-3).
## Reading a node card
Every node section starts with a card drawn from the node's real definition:
@@ -20,7 +24,7 @@ Every node section starts with a card drawn from the node's real definition:
- **Title bar** — the display name, then the node id (type it in the node search) and its menu group under `NH-Nodes`.
- **Dots on the left** are inputs you connect, **dots on the right** are outputs. The colour shows the data type:
IMAGE blue, MASK green, LATENT pink, MODEL purple, CLIP yellow, VAE red, numbers (INT, FLOAT) light blue,
text (STRING) light green, BOOLEAN rose, any type (`*`) grey, and NH-specific data (tile data, resize data…) amber.
text (STRING) light green, BOOLEAN rose, any type (`*`) grey, and NH-specific data (tile data, resize data, instances…) amber.
- **Rounded rows** are widgets with their default value. `opt` marks an optional input; `≡` after an output means it
returns a list.
- A **green** dot in the title is an ordinary node; **orange** marks an output node, which runs even when nothing is
@@ -36,7 +40,7 @@ Every node section starts with a card drawn from the node's real definition:
| Image — mask-based retouch | [image-retouch](image-retouch.md) | NH Mask-Aware Resize · NH Agnostic Image Generator · NH Simple Face Paste |
| Resolution | [resolution](resolution.md) | NH Smart Resolution Picker · NH Smart Ratio Image Resize · NH Ratio Preset Image Resize · NH Remove Resize Padding · NH Restore Image Geometry |
| Mask | [mask](mask.md) | Mask Morphology · Mask Properties · NH Create Box Mask · NH Mask Aspect Ratio Match |
| Vision | [vision](vision.md) | Garment Segment · Portrait Segment |
| Vision | [vision](vision.md) | Garment Segment · Portrait Segment · SAM3 Segment · Segs Select · Segs Compose · Segs To Masks · Segs To Impact SEGS · Segs Preview |
| VTON | [vton](vton.md) | VTON Ultimate Processor |
| Logic | [logic](logic.md) | Compare · Logic Gate · If/Else · Switch N · Any Switch · Any Branch Switch · Gate Switch · Value Match Index · Math Eval · Random Choice |
| Text | [text](text.md) | Prompt Template · Prompt Scheduler · String Operations · Prompt Join · Text Split · Regex Extract · Text Index · Text Concatenate · Text Split Lines · Text Random Line |
@@ -58,13 +62,17 @@ Every node section starts with a card drawn from the node's real definition:
[image-io](image-io.md#paths-and-file-types).
- **Resize data.** The resolution nodes return a `resize_data` record of how an image was cropped or padded, so a result
can be put back into the original frame with [NH Restore Image Geometry](resolution.md#nh-restore-image-geometry).
- **Instances.** SAM3 Segment returns every thing it finds as its own instance, with a label, a score and a mask, in an
`NH_SEGS` value. The Segs nodes pick instances, combine them with a formula and turn them into masks; see
[Instances: the NH_SEGS type](vision.md#instances-the-nh_segs-type).
- **Licences.** All nodes are MIT except the parts listed in [THIRD_PARTY_NOTICES](../../THIRD_PARTY_NOTICES.md):
VTON Ultimate Processor and Portrait Segment are for **non-commercial** use, and Garment Segment's model declares no
licence.
licence. SAM3 Segment uses Meta's SAM 3 under the SAM License, which has no non-commercial clause but bars military,
nuclear, espionage, and gun or illegal-weapon uses; you install the package and download the checkpoint yourself.
## More
- [Changelog](../CHANGELOG.md) — what changed in each release, including behaviour changes.
- [Logic gate diagram](../logic_gate_diagram.html) — a visual truth-table sheet for the Logic Gate node (download and open
it in a browser).
- [Third-party notices](../../THIRD_PARTY_NOTICES.md) — licences of bundled code and downloaded models.
- [Third-party notices](../../THIRD_PARTY_NOTICES.md) — licences of bundled code, downloaded models and SAM 3.
+2 -1
View File
@@ -12,7 +12,7 @@ Three nodes that work with a mask. One resizes an image to an exact size and kee
| [NH Agnostic Image Generator](#nh-agnostic-image-generator) | Erase the masked area with grey, noise or blur |
| [NH Simple Face Paste](#nh-simple-face-paste) | Paste the masked face of one image onto the masked area of another |
A `MASK` is white (`1`) where the node should work and black (`0`) elsewhere. Get one from a segmentation node such as [Garment Segment (NH)](./vision.md#garment-segment-nh) or [Portrait Segment (NH)](./vision.md#portrait-segment-nh), or from any mask tool in ComfyUI.
A `MASK` is white (`1`) where the node should work and black (`0`) elsewhere. Get one from a segmentation node such as [SAM3 Segment (NH)](./vision.md#sam3-segment-nh) (any object you can name, or a formula such as `person - bag` or `background`), [Garment Segment (NH)](./vision.md#garment-segment-nh) or [Portrait Segment (NH)](./vision.md#portrait-segment-nh), or from any mask tool in ComfyUI.
---
@@ -172,6 +172,7 @@ Set `feathering` to `20`. Because a single source is reused, every image of the
- Using masks whose boxes have very different proportions. The face looks squashed or stretched.
- Giving a mask that has another size than its image.
- Using a [Portrait Segment (NH)](./vision.md#portrait-segment-nh) mask of a photo with several faces. The mask holds every face, so the box around it covers them all.
- Setting `feathering` very high for a small face. The blur can eat the whole shape.
> **Tip:** Make `dest_mask` slightly larger than the face you replace, and keep `source_mask` tight. The soft edge of the source mask then blends into the destination.
+468 -14
View File
@@ -2,17 +2,45 @@
# Vision
Two nodes that cut parts out of a photo of a person with SegFormer models: clothing with Garment Segment, and face, hair and neck with Portrait Segment. Each returns a mask at the size of the input image, a cutout with transparency and a mask preview. Find them under **NH-Nodes → Vision**.
Eight nodes that cut parts out of a photo. Two use SegFormer models that know people: clothing with Garment Segment, and face, hair and neck with Portrait Segment. Each returns a mask at the size of the input image, a cutout with transparency and a mask preview. The other six work with SAM 3, Meta's segmentation model: you type what you want into SAM3 Segment (people, a wine glass, windows, the sky, or a formula such as `person - bag` or `background`) and it returns the mask and shows it on the node. It also keeps every instance it found, and five Segs nodes pick, combine, preview and convert them. Find them under **NH-Nodes → Vision**.
**On this page**
- [Before you start](#before-you-start) — models, first run, licences, memory
- [Refining the mask](#refining-the-mask) — the options both nodes share
- [Garment Segment (NH)](#garment-segment-nh) — mask upper garments, lower garments, footwear or headwear
- [Portrait Segment (NH)](#portrait-segment-nh) — mask face, hair, neck and facial features
- [Which node do I need?](#which-node-do-i-need) — the SegFormer nodes or SAM 3
- **SegFormer nodes: garments and portraits**
- [Before you start](#before-you-start) — models, the face detector, first run, licences, memory
- [How the SegFormer nodes look at a photo](#how-the-segformer-nodes-look-at-a-photo) — why small faces and people get close-ups
- [Refining the mask](#refining-the-mask) — the options both nodes share
- [Garment Segment (NH)](#garment-segment-nh) — mask upper garments, lower garments, footwear or headwear
- [Portrait Segment (NH)](#portrait-segment-nh) — mask face, hair, neck and facial features
- **SAM 3 nodes: anything you can name**
- [Before you start with SAM 3](#before-you-start-with-sam-3) — GPU, package, checkpoint, memory, licence
- [How the two passes work](#how-the-two-passes-work) — what `balanced` does on 4K to 8K photos, and where it stops
- [Instances: the NH_SEGS type](#instances-the-nh_segs-type) — what flows between the nodes
- [Writing a formula](#writing-a-formula) — what `concepts` and `expression` accept: lists, `-`, `&`, `~`, `background`, Vietnamese words
- [SAM3 Segment (NH)](#sam3-segment-nh) — type what you want, get its mask and see it on the node
- [Segs Select (NH)](#segs-select-nh) — keep instances by label, score, size and position
- [Segs Compose (NH)](#segs-compose-nh) — build a mask from labels with `|`, `-`, `&` and `~`
- [Segs To Masks (NH)](#segs-to-masks-nh) — one mask per instance, the union and the labels
- [Segs To Impact SEGS (NH)](#segs-to-impact-segs-nh) — hand the instances to Impact Pack's detailers
- [Segs Preview (NH)](#segs-preview-nh) — draw instances on their picture
- [SAM 3 workflows](#sam-3-workflows) — the background behind people, the biggest object, a facade without windows
- [If something goes wrong with SAM 3](#if-something-goes-wrong-with-sam-3)
## Which node do I need?
| You want | Use | What it needs |
|---|---|---|
| The clothing of one person or of a group: shirt, trousers, shoes, hat | [Garment Segment](#garment-segment-nh) | A SegFormer model and a small face detector that download themselves on the first run. Runs on the CPU or the GPU. |
| The face, hair, neck and facial features of a portrait, a half-body shot or every face of a group photo | [Portrait Segment](#portrait-segment-nh) | The same. |
| Anything you can name (people, objects, sky, walls, windows) as a mask from a short formula (`person - bag`, `background`), or every instance on its own, also on 4K to 8K photos | [SAM3 Segment](#sam3-segment-nh) and the Segs nodes | An NVIDIA GPU, Meta's `sam3` package and a checkpoint that you install and download yourself. See [Before you start with SAM 3](#before-you-start-with-sam-3). |
You can ask SAM3 Segment for `dress` or `hair` too. The SegFormer nodes stay the easier choice for their fixed sets of clothing and face parts: they set themselves up and also run without a GPU.
## Before you start
This section is about Garment Segment and Portrait Segment. SAM3 Segment and the Segs nodes have their own: [Before you start with SAM 3](#before-you-start-with-sam-3).
| | Garment Segment (NH) | Portrait Segment (NH) |
|---|---|---|
| Model | SegFormer trained for Fashionpedia clothing classes | SegFormer fine-tuned on CelebAMask-HQ face parts |
@@ -20,17 +48,31 @@ Two nodes that cut parts out of a photo of a person with SegFormer models: cloth
| Download | about 190 MB | about 340 MB |
| Licence | The uploader declares none | Non-commercial research and education only |
**First run.** The first time a node runs, it downloads its model from Hugging Face. Each download is pinned to a fixed revision of the repository, so a later change upstream cannot alter what you get. The files go to `ComfyUI/models/NH-Nodes/`, one sub-folder per model. A complete copy that already exists is used instead: in `models/RMBG/segformer_fashion` for the garment model, in `models/face_parsing` or `models/RMBG/segformer_face` for the face model, or in any folder you add under the `nh_nodes` key in `extra_model_paths.yaml` (see [Installation](../../README.md#installation) for an example). If a download is interrupted, run the node again: finished files are kept.
**First run.** The first time a node runs, it downloads its model from Hugging Face, together with the face detector described below. Each download is pinned to a fixed revision of the repository, so a later change upstream cannot alter what you get. The files go to `ComfyUI/models/NH-Nodes/`, one sub-folder per model. A complete copy that already exists is used instead: in `models/RMBG/segformer_fashion` for the garment model, in `models/face_parsing` or `models/RMBG/segformer_face` for the face model, or in any folder you add under the `nh_nodes` key in `extra_model_paths.yaml` (see [Installation](../../README.md#installation) for an example). If a download is interrupted, run the node again: finished files are kept.
**Face detector.** Both nodes also use YuNet, a small face detector (about 230 KB, MIT licence, from OpenCV Zoo), to find where the people are before they parse a picture. It is downloaded once, pinned to a fixed revision of `opencv/face_detection_yunet`, into `ComfyUI/models/NH-Nodes/yunet/`. It runs on the CPU with the OpenCV that NH-Nodes already needs, so it takes no VRAM. A copy that already sits in a `yunet` sub-folder of any `nh_nodes` folder (`models/NH-Nodes/`, or one you add in `extra_model_paths.yaml`) is used instead; to run offline, put `face_detection_yunet_2023mar.onnx` there yourself. If the detector cannot be loaded (offline on the first run, or an OpenCV that cannot run it), the node prints one warning in the console and parses whole pictures, without the close-ups described in [How the SegFormer nodes look at a photo](#how-the-segformer-nodes-look-at-a-photo). Restart ComfyUI to try again.
**Memory.** A model is read from disk once and kept in system memory. It moves to the GPU only while a node runs, and goes back to system memory afterwards, so it does not hold VRAM between runs. Before it moves to the GPU, the node asks ComfyUI to unload some of its own models if VRAM is tight.
> **Note:** If a node reports unexpected labels, a different checkpoint sits in one of the folders it searches. Remove that copy and the right model is downloaded.
> **Licence:** The model behind Portrait Segment may only be used for non-commercial research and education. The model behind Garment Segment declares no licence, so its terms are unclear. NH-Nodes' own code is MIT. Read [THIRD_PARTY_NOTICES.md](../../THIRD_PARTY_NOTICES.md) before you use either node commercially.
> **Licence:** The model behind Portrait Segment may only be used for non-commercial research and education. The model behind Garment Segment declares no licence, so its terms are unclear. NH-Nodes' own code is MIT. The YuNet face detector is MIT-licensed and changes none of this. Read [THIRD_PARTY_NOTICES.md](../../THIRD_PARTY_NOTICES.md) before you use either node commercially.
## How the SegFormer nodes look at a photo
Both models look at a 512 × 512 picture. On a 6000 px photo, a face that is 600 px wide is only about 50 px wide to the model, so the nodes do not hand it the photo in one piece.
1. **The whole picture first.** A picture with a side longer than twice the model's 512 px is shrunk with area averaging, which is faster than the model's own resize and keeps fine detail clean, and then parsed. This first answer is kept wherever nothing better is found, so whatever the later steps miss keeps at least this result.
2. **People are found.** The face detector looks for faces on two sizes of the picture. Garment Segment also takes the people that the first parse found.
3. **Close-ups refine the answer.** Each face (Portrait Segment) or group of people (Garment Segment) is parsed again on a crop that the model sees at its full 512 px. The crop's labels go into the first answer, but not along the crop's cut edges, where the model sees things cut off, so no straight edge shows in hair or clothes. How far a crop is trusted differs by node, as the Garment Segment and Portrait Segment sections describe.
On a photo longer than 4096 px, all of this is worked out on a copy 4096 px long and the label map is enlarged to the photo at the end: the model sees 512 px at a time, so its labels hold no finer detail, and merging crops at full 8K size would take seconds.
What each node does with this is in its own section: [Garment Segment](#garment-segment-nh) and [Portrait Segment](#portrait-segment-nh).
## Refining the mask
Both nodes turn the model's label map into a mask, then refine it in this order: `fill_holes`, `mask_expand`, `mask_blur`. The mask always has the size of the input image.
Garment Segment and Portrait Segment turn the model's label map into a mask, then refine it in this order: `fill_holes`, `mask_expand`, `mask_blur`. The mask always has the size of the input image.
| Option | Range | Garment Segment | Portrait Segment | What it does |
|---|---|---|---|---|
@@ -55,13 +97,13 @@ Batches work: every image is segmented in turn and the outputs keep the batch or
<p align="center"><img src="../assets/nodes/NH_GarmentSegment.svg" alt="Garment Segment (NH) node" width="360"></p>
Masks one group of clothing, or all groups together, on a person in the image.
Masks one group of clothing, or all groups together, on the people in the image.
**Use it when** you want a clean mask of a shirt, trousers, shoes or a hat for inpainting, recolouring or a try-on workflow.
| Input | Type | Default | Description |
|---|---|---|---|
| `image` | IMAGE | — | A photo of a person. |
| `image` | IMAGE | — | A photo of a person or of a group of people. |
| `part` | COMBO | `upper_garment` | `upper_garment`, `lower_garment`, `footwear`, `headwear` or `all_garments`. See the table below. |
| `attach_parts` | BOOLEAN | `true` | Merges pockets, zippers, bows, ruffles and similar decorations that touch the selected garment. |
| `mask_expand` | INT | `0` | Range -64 to 64. Grows (+) or shrinks (-) the mask by N pixels. |
@@ -90,6 +132,8 @@ Masks one group of clothing, or all groups together, on a person in the image.
- **Dresses and jumpsuits** belong to `upper_garment`. There is no separate dress option. For a shirt and trousers together, or a dress and shoes, choose `all_garments`.
- **Attached parts** are pockets, zippers, appliqués, beads, bows, flowers, fringe, ribbons, rivets, ruffles, sequins and tassels. With `attach_parts` on, a piece is added only when it touches the selected garment (within a few pixels). A pocket on a jacket joins an `upper_garment` mask, but a bow on a bag does not.
- **Never selected:** glasses, gloves, watches, bags, wallets and umbrellas have no `part`.
- **Close-ups of people.** The node parses the whole picture first and keeps that answer. It then finds the people (in that parse and with the face detector) and cuts square crops, the model's own input shape, around each group of them: a row of overlapping crops for a wide group, one crop for a person or a tight group. Each crop is parsed at the model's full 512 px, and a crop grows when people run past one of its edges. This helps people who are small in a large photo. See [How the SegFormer nodes look at a photo](#how-the-segformer-nodes-look-at-a-photo).
- **A crop only refines.** Its labels replace the first answer only where the model is surer, away from the crop's cut edges, and only for parts on people already found. Nothing is invented on an empty backdrop, and no straight crop edge cuts through a garment.
<p align="center"><img src="../assets/examples/garment-segment.jpg" alt="Garment Segment masks overlaid on a sample photo" width="720"><br><em>Garment Segment on a sample photo: the coloured areas are the masks of the selected parts.</em></p>
@@ -101,6 +145,8 @@ Load Image → Garment Segment (NH, upper_garment) → mask → Mask Morphology
Grow the mask a few pixels with [Mask Morphology (NH)](./mask.md#mask-morphology-nh) so the inpainted garment can overlap the old outline.
> **Note:** The Fashionpedia model misses some tops, for example a blouse under a big hat. A better garment model is planned.
> **Licence:** the garment model declares no licence. See [Before you start](#before-you-start).
---
@@ -109,13 +155,13 @@ Grow the mask a few pixels with [Mask Morphology (NH)](./mask.md#mask-morphology
<p align="center"><img src="../assets/nodes/NH_PortraitSegment.svg" alt="Portrait Segment (NH) node" width="360"></p>
Masks the face, hair, neck and facial features of a portrait. You tick the regions you want.
Masks the face, hair, neck and facial features of a portrait, or of every face in a group photo. You tick the regions you want.
**Use it when** you want to retouch or recolour hair, work on skin and facial features, or protect the face while you edit the rest of the picture.
| Input | Type | Default | Description |
|---|---|---|---|
| `image` | IMAGE | — | A portrait photo. |
| `image` | IMAGE | — | A portrait, a half-body shot or a group photo. |
| `face` | BOOLEAN | `true` | The solid face region: skin, nose, eyes, eyebrows, eyeglasses, lips and the inside of the mouth. |
| `neck` | BOOLEAN | `true` | The neck. |
| `hair` | BOOLEAN | `true` | The hair. |
@@ -132,7 +178,7 @@ Masks the face, hair, neck and facial features of a portrait. You tick the regio
| Output | Type | Description |
|---|---|---|
| `mask` | MASK | The combined mask of every ticked region. |
| `mask` | MASK | The combined mask of every ticked region, of every face found. |
| `cutout_rgba` | IMAGE | The input with the mask as alpha. |
| `mask_image` | IMAGE | The mask as a grey RGB preview. |
@@ -141,6 +187,10 @@ Masks the face, hair, neck and facial features of a portrait. You tick the regio
- The ticked regions are combined into one mask. With nothing ticked, the mask is empty.
- `face` already contains eyes, eyebrows, nose, lips and the inside of the mouth. Those options change the result only when `face` is off.
- Hats, earrings, necklaces and clothing exist in the model, but the node does not offer them.
- **The whole picture first.** It is parsed as a padded square (it keeps its proportions, grey fills the rest) instead of being squeezed into the model's square. This answer stays wherever the crops below do not reach, so a picture in which no face is found, or in which the detector is not available, keeps it as the result. Once faces are found, the face parts (skin, eyes, eyebrows, nose, lips, ears) that this answer shows far from all of them are dropped as mistakes, such as a reflection or a pattern; its hair, neck and clothes stay.
- **Faces.** The face detector finds the faces (at most 32, the largest ones). When it finds none, the picture gets a grey margin and is searched once more, which finds a face that the frame cuts. The parser then checks each face on a tight crop: a hand or a pattern that the detector took for a face is turned down and changes nothing.
- **Two crops per face.** A wide crop shows hair and neck whole. A tight crop, framed like the photos the model learnt on, gives eyes, eyebrows, nose and lips. The wide crop refines what is inside it away from its cut edges, and the tight crop decides only inside an ellipse around the face, so no straight crop edge shows in long hair. Where the crops of two faces overlap, each pixel goes to the nearer face. In a group photo the mask therefore holds the ticked regions of every person.
- **Large faces.** A face that already fills a large part of the frame is left to the whole-picture parse, which shows it large enough. See [How the SegFormer nodes look at a photo](#how-the-segformer-nodes-look-at-a-photo).
<p align="center"><img src="../assets/examples/portrait-segment.jpg" alt="Portrait Segment masks overlaid on a sample photo" width="720"><br><em>Portrait Segment on a sample photo: the coloured areas are the masks of the ticked regions.</em></p>
@@ -152,10 +202,414 @@ Load Image → Portrait Segment (NH) → mask → recolour or inpaint
Switch off every region except `hair`. With the defaults you get face, ears, neck and hair together in one mask.
> **Tip:** The model was fine-tuned on CelebAMask-HQ, a set of close-up face photos. It works best when the face fills a good part of the frame. For a small face in a large photo, crop around the face first.
> **Tip:** The model was fine-tuned on CelebAMask-HQ, a set of close-up face photos, so the node parses each face it finds on a close-up crop. A small face in a large photo, a half-body shot or a group photo needs no cropping first. A face the detector misses (very small, hidden or turned away) keeps the whole-picture parse, which is coarser: crop around it yourself when you need its finest detail.
> **Licence:** the face-parsing model is for non-commercial research and education only. See [Before you start](#before-you-start).
---
## Before you start with SAM 3
SAM 3 is Meta's model for finding things by name: you write what you want (`person`, `window`, `wine glass`) and it returns every instance of it, each with a score and its own mask. NH-Nodes does not include the model and never downloads it, so SAM3 Segment needs three things from you, once. If another SAM 3 pack already works in your ComfyUI, steps 2 and 3 are usually done: that pack has installed the package and put `sam3.pt` where this node looks.
**1. An NVIDIA GPU with CUDA.** SAM 3 runs only on CUDA: the official sam3 code builds part of the model on the GPU, so there is no CPU mode and no `device` input. Plan for about 5 GB of VRAM at the peak.
**2. Meta's `sam3` package.** NH-Nodes does not install it for you. In the Python that runs ComfyUI, run these two lines, then restart ComfyUI:
```text
python -m pip install --no-deps git+https://github.com/facebookresearch/sam3@b26a5f330e05d321afb39d01d3d4881f258f65ff
python -m pip install timm ftfy regex iopath
```
The first line installs the sam3 commit that NH-Nodes was tested with; newer commits with the same image API also work. It needs Git on your computer. `--no-deps` keeps your numpy: sam3 asks for numpy below 2, which ComfyUI does not need. Meta's page for the package lists Python 3.12 or newer, PyTorch 2.7 or newer and CUDA 12.6 or newer. If another SAM 3 pack already provides `sam3`, skip the first line: installing it again could replace the copy that pack uses.
**3. The checkpoint `sam3.pt`.** Meta hands out the weights only after you accept its SAM License, so NH-Nodes cannot download them for you. Request access at [huggingface.co/facebook/sam3](https://huggingface.co/facebook/sam3), download `sam3.pt` from that page and put it in `ComfyUI/models/sam3/`. Press R in ComfyUI to refresh the lists, then pick the file in `checkpoint`. To keep it somewhere else, add the folder under the `sam3` key in `extra_model_paths.yaml` (see [Installation](../../README.md#installation) for the file):
```yaml
my_models:
base_path: D:/ai-models/
sam3: sam3/ # SAM 3 checkpoints, shared with other SAM 3 packs
```
Only the PyTorch checkpoint `sam3.pt` works. The safetensors export on Hugging Face is made for the transformers library and uses other weight names.
**Memory.** The weights are read from disk once and kept in system memory (about 3.6 GB). They move to the GPU while SAM3 Segment runs and go back afterwards, so ComfyUI gets its VRAM back. Before they move, the node asks ComfyUI to unload some of its own models if VRAM is tight. Turn on `keep_model_loaded` to leave them on the GPU when you run the node again and again. While it is on, ComfyUI's **Unload Models** button does not release them: turn `keep_model_loaded` off and run the node once, or restart ComfyUI.
> **Licence:** SAM 3 is released under Meta's SAM License. It is royalty-free and has no non-commercial clause, but it forbids uses that fall under ITAR or that trade controls prohibit, including military or warfare, nuclear, espionage, and guns or illegal weapons. NH-Nodes' own code is MIT, and it contains neither the model nor the `sam3` package. Read [THIRD_PARTY_NOTICES.md](../../THIRD_PARTY_NOTICES.md) before you use the node commercially.
---
## How the two passes work
SAM 3 looks at a picture of about 1000 px. On a 6000 px photo that is a sixth of the width, so small objects and thin edges would come out soft. SAM3 Segment therefore looks twice.
1. **First pass: the whole image.** The picture is shrunk to the model's square input (1008 px) and encoded once. Every concept you listed is asked against that one encoding, which is cheap. This gives each instance a score, a box and a first mask, which is resampled to full resolution inside its own box only.
2. **Second pass: a close-up of each instance** (`balanced` only). The instance is cut out of the full-resolution image with a margin (20% of its longer side, at least 16 px) and segmented again with SAM 3's box-prompt decoder, with the first mask as a hint. Its edges now come from the original pixels.
3. **A safety net.** A close-up is kept when it matches the first-pass instance to within the blur of the first pass, so thin things such as poles and cables survive. When it does not match, the first-pass mask stays, and the console says for how many instances. Specks of the new mask that do not touch the first-pass mask are dropped.
| `detail_level` | Passes | Use it for |
|---|---|---|
| `balanced` (default) | Both | Crisp edges on small and medium objects, also on 4K to 8K photos. About 0.1 s per instance on a fast GPU. |
| `fast` | The first only | Small pictures, and quick tries while you look for the right words for the concepts. |
**What `balanced` can and cannot do.** Small and medium objects (a window, a glass, one person in a group photo) get crisp edges even on a 4K to 8K photo, because each one is judged on a close-up of the original pixels. The limit is the model: SAM 3 describes whatever it looks at on a grid of about 288 cells. A subject that fills most of an 8K frame is its own close-up, so its edges are accurate to about 1/300 of its size, around 25 px for a subject 7000 px across. That is fine for the outline of a person or a building, and soft on hair, lace and fur. Hair-level edges need a matting step that NH-Nodes does not have yet; a Matte Refine node is planned.
**Measured on one machine** (an NVIDIA RTX 5090): the first run loads the model in about 9 s. A 7360×4912 photo with the concepts `person, wine glass, dress` gave 13 instances in about 3.6 s with `balanced`. A 6720×4480 interior with 8 concepts gave 29 instances in about 6 s. The VRAM peak was about 5 GB. Other GPUs will differ.
---
## Instances: the NH_SEGS type
`NH_SEGS` is the data that SAM3 Segment hands to the Segs nodes. For every image of a batch it holds a list of instances. Each instance keeps its **label** (the English words SAM 3 was asked for: `áo đỏ` is stored as `red top`), its **score**, a tight **box** and the mask **only inside that box**. A full-size mask is built only when a node outputs a `MASK`, so dozens of instances on an 8K photo stay light.
- Instances come in the order the labels appear in your concepts, and by score within a label.
- The Segs nodes treat every image of a batch separately.
- ComfyUI's own nodes do not read `NH_SEGS`. To get masks out, use the `mask` outputs, [Segs Compose](#segs-compose-nh) or [Segs To Masks](#segs-to-masks-nh). To use Impact Pack's detailers, convert with [Segs To Impact SEGS](#segs-to-impact-segs-nh).
---
## Writing a formula
SAM3 Segment's `concepts` and Segs Compose's `expression` read the same small language. You name the things you want, combine them with operators, and the result is one mask per image. In SAM3 Segment the words you write are also what SAM 3 looks for. In Segs Compose they pick instances that were found already.
| Write | Meaning |
|---|---|
| `person` | A label: the instances with that name. Case does not matter. |
| `red car`, `women's dress` | Several words in a row are one label, with no quotes. A hyphen between two letters or digits belongs to the label (`t-shirt`, `x-ray`), and so does an apostrophe inside a word. |
| `person, hair` | A list: the union of its items. A new line works like a comma. A list is read after every other operator, so `person - hair, dress` is `person - hair` plus `dress`. Empty items are skipped. |
| `a + b`, `a \| b` | Union: everything in a or in b. |
| `a - b` | a without b. Put spaces around the minus: `person-hair` is read as one label. |
| `a & b` | Overlap: only where a and b both are. |
| `~a`, `!a` | Everything except a: the whole picture minus a. |
| `( … )` | Grouping. |
| `background` | Everything outside every instance that was found. The same as `~all`. |
| `all` | Every instance that was found. |
| `image` | The whole picture. |
| `none` | An empty mask. |
| `# …` | A comment, to the end of the line. It starts where a word could start; inside a word, as in `car#2`, the `#` belongs to the word. |
| `"salt & pepper"` | Quotes keep together a label that contains operator characters. A quoted `all`, `image`, `none` or `background` is an ordinary label. |
`~` binds first, then `&`, then `+`, `|` and `-` together, left to right, and last the list: `a - b + c` is `(a - b) + c`. When in doubt, add parentheses.
**Line breaks.** A line break lists, like a comma. It joins instead when it is next to an operator (the line before ends with one, other than `)`, or the next line starts with `+`, `|`, `-`, `&` or `)`), so a long formula can be laid out over several lines:
```text
(wall + ceiling)
- window
```
**Vietnamese words.** SAM 3 reads English only, so the nodes translate the common Vietnamese words you type: `tóc` is `hair`, `váy` is `dress`, `cửa sổ` is `window`. The list covers people, body, clothes, accessories, nature, rooms and buildings, vehicles, food and animals. The full list is in `core/vi_words.py`.
- A thing followed by colours or sizes is put into English order: `áo đỏ` is `red top`, `tóc dài` is `long hair`, `ghế sofa lớn` is `large sofa`. `màu` (colour) may stand between them: `áo màu đỏ`. `tóc vàng` is `blonde hair`.
- `xanh` after a thing means blue or green, so it is not guessed. Write `xanh dương` (blue) or `xanh lá` (green), as in `áo xanh dương`, or write the English word.
- `các`, `những`, `mấy`, `cái`, `chiếc` and `con` in front of a thing are skipped when the phrase is not in the list as written: `những chiếc xe` is `vehicle`.
- `nền`, `phông`, `phông nền` and `hậu cảnh` are the keyword `background`, also with a colour (`nền trắng`). `tất cả` is `all`, and `toàn ảnh` and `cả ảnh` are `image`.
- Write Vietnamese with its accents. Text without Vietnamese accents is taken as English (a few short words, such as `xe`, are in the list as written).
- A Vietnamese phrase that is not in the list is searched for as written, and SAM 3 usually finds nothing. The report then says that it was not understood. The labels of the instances are always the English words.
- Quotes do not stop the translation. A quoted `nền` is the label `background`, not the keyword.
**Where the two nodes differ**
| | SAM3 Segment, `concepts` | Segs Compose, `expression` |
|---|---|---|
| `or`, `and`, `not` | Plain words, so a phrase such as `black and white dog` reaches SAM 3 as written. | Operators: `a or b`, `a and b`, `not a`. |
| Wildcards `*` and `?` | Refused: SAM 3 needs plain words. | Match labels (`*shirt`). |
| What a label does | It is what SAM 3 is asked to find. | It picks the instances that have that label. A label that no instance has gives an empty part. |
| An empty field | Stops with a message to write at least one concept. A field with only comments, or only `all`, `image` or `none`, counts as empty. | Stops with a message to use `all` for every instance. |
---
## SAM3 Segment (NH)
<p align="center"><img src="../assets/nodes/NH_SAM3Segment.svg" alt="SAM3 Segment (NH) node" width="360"></p>
Type what you want to cut out and get its mask: one word (`person`), a list (`hair, face`) or a formula (`person - bag`, `background`). SAM 3 finds every instance of the things you name. The node returns the mask of your formula, the instances as `NH_SEGS`, a count and a report of what was and was not found, and it shows what it found on itself.
**Use it when** you want masks of "all the people", "every window", "the sky", "the background" or "the person without the bag" in a photo, also a 4K to 8K one, without painting anything. For finer picks (the second person from the left, the biggest object), or to make several masks from one run, connect `segs` to the [Segs nodes](#segs-select-nh).
Combo inputs without a stated default start on their first option.
| Input | Type | Default | Description |
|---|---|---|---|
| `image` | IMAGE | — | The picture. Colour (RGB) is used and any alpha is dropped. Every image of a batch is processed in turn. |
| `concepts` | STRING | `person` | What to find and which mask to return, written as a formula (see [Writing a formula](#writing-a-formula)): short English words (`person`, `red car`, `window`), commas or new lines to list several, and `+`, `-`, `&`, `~`, parentheses and `background` to combine them (`person - bag`). Common Vietnamese words work too (`tóc`, `áo đỏ`, `nền`). Write at least one word. |
| `threshold` | FLOAT | `0.5` | Range 0.05 to 0.95. The minimum confidence. Lower finds more instances and more mistakes; higher keeps only the sure ones. A score exactly at the threshold is not kept. When a concept finds nothing, the report says how close the best candidate came. |
| `detail_level` | COMBO | `balanced` | `balanced` or `fast`. `balanced` segments every instance a second time on a close-up of the full-resolution image, for crisp edges on small and medium objects. `fast` keeps the first, whole-image pass. See [How the two passes work](#how-the-two-passes-work). |
| `max_per_concept` | INT | `0` | Range 0 to 200. Keep at most this many instances of each concept, best score first. `0` keeps all. |
| `checkpoint` | COMBO | — | A SAM 3 `.pt` or `.pth` file from `models/sam3` or another folder registered as `sam3`. Other SAM 3 packs use the same folder. When it holds no file, the list shows `sam3.pt` and the node tells you where to get it when you run it. |
| `keep_model_loaded` | BOOLEAN | `false` | Off: after the run the weights go back to system memory and ComfyUI gets the VRAM back. On: they stay on the GPU (about 3.6 GB), which is faster when the node runs again, but ComfyUI cannot use that VRAM meanwhile, and its **Unload Models** button does not release it (turn this off and run the node once, or restart ComfyUI). |
| `show_preview` | BOOLEAN | `true` | Optional. Shows the result on the node: the mask bright and the rest darkened, the instances outlined and labelled, and the report lines below. At most 1536 px on the longer side, and the first 16 pictures of a batch. Off: nothing is drawn, and the node still runs (see below). |
| Output | Type | Description |
|---|---|---|
| `segs` | NH_SEGS | The instances of every image, each with its label, score, box and mask. The labels are the English words SAM 3 was asked for. When the formula uses `background`, they include the `person` instances that were searched for it. Connect it to the Segs nodes. |
| `mask` | MASK | The mask of your formula, one per image, at the size of the image. A plain list of words gives the union of every instance. |
| `count` | INT | The number of instances found, over the whole batch, including the `person` instances that `background` searched for. |
| `report` | STRING | What the request found, one line per word: counts and best scores, translations, near misses, words that were not understood, and what `background` meant. See below. |
<p align="center"><img src="../assets/examples/sam3-formula.jpg" alt="What SAM3 Segment shows on the node for four formulas on a sample photo: background, person - face, hair and face, and everything except the person and the helmets" width="720"><br><em>SAM3 Segment on a sample photo, as the node shows it. Top: "background" lights up everything outside the person, and the report is under the picture. Bottom: three more formulas; the bright part is each one's mask output.</em></p>
**The report.** The first line of `report` gives the totals. Each line after it (they are also written under the preview) starts with what you wrote:
| Line | Meaning |
|---|---|
| `person: 2 found (best 0.91)` | The number of instances found and the best score. |
| `áo đỏ -> red top: 1 found (best 0.64)` | What a Vietnamese phrase was translated to. |
| `hair: none at threshold 0.50; the best candidate scored 0.31, lower the threshold to keep it` | Nothing passed `threshold`, but a candidate came close. Lower `threshold` below its score to keep it. |
| `bag: none found; try another English word` | No candidate scored even 0.1. |
| `áo xanh: not understood; write it in English (SAM 3 reads English only)` | A Vietnamese phrase that is not in the list and found nothing. |
| `background = everything outside person, red top` | What `background` meant: everything outside the instances of these labels. |
| `person (searched for background): 2 found` | The `person` search that the node added because the formula uses `background` and does not name `person`. |
**How it works**
- **Labels.** The English words SAM 3 was asked for are the labels of the instances: `áo đỏ` is looked for as `red top`, and its instances are labelled `red top`. The Segs nodes ignore case when they match labels, so spelling is not a problem.
- **The mask.** `mask` is the result of the formula. For `person - bag`, SAM 3 looks for `person` and for `bag`, `mask` is the people without the bags, and `segs` still holds both kinds of instance.
- **`background`.** It is everything outside every instance that was found, so the node needs to know where the subject is. When the formula uses it and does not name `person`, SAM 3 also looks for `person`; those instances are part of `segs` and `count` and are outlined in the preview. To keep other things out of the background too, name them: `~(person + wine glass + chair)`. A list is a union, so `person, background` is the whole picture.
- **Order and duplicates.** Instances come in the order the labels appear in your formula (the `person` that `background` added comes last), best score first within a label. Two instances of the same label whose masks overlap almost completely (90% or more) count as one, and the better score stays. `max_per_concept` then keeps the best of what is left. Instances with exactly equal scores keep the order SAM 3 gave them.
- **Seeing the result.** With `show_preview` on, the node draws the preview itself: no other node is needed. To see the picture at full size (the preview is at most 1536 px), or to see what Segs Select works with, connect `segs` and the same picture to [Segs Preview](#segs-preview-nh).
- **It runs on its own.** SAM3 Segment is an output node (the dot on its card is orange): ComfyUI runs it even when nothing is connected to its outputs. A copy that you no longer need therefore keeps running SAM 3 whenever its inputs change, and turning `show_preview` off does not stop it. Mute or delete it. While its inputs stay the same, ComfyUI reuses the last result.
- **Nothing found** is not an error: you get no instances and a `count` of 0. The mask is empty for a plain list, and the whole picture for a formula such as `background` or `~person`, because nothing was found to leave out.
- **Batches.** Every image is processed in turn and has its own list of instances. A progress bar counts the concepts, and you can cancel the run between concepts and between close-ups.
- **Things cut by the frame.** SAM 3 fades its masks over a band along the border of the square view it works in, about 0.3% of the picture's side wide. The mask of a thing that runs off the picture still reaches the image edge: the band is lifted where the object reaches it from inside. The price is that a thing which ends within that band just above the edge is drawn down to the edge.
- **Very wide or tall pictures.** SAM 3 first sees the whole picture squeezed into a square. On a picture wider or taller than 4:1 (a panorama), it can miss small things, and the console says so. Split such a picture into tiles with [Image Tile (NH)](./image-compose.md#image-tile-nh) and run the tiles.
- **Setup problems** stop the node with a message that says what to do: a missing package, a missing checkpoint, no CUDA GPU. See [If something goes wrong with SAM 3](#if-something-goes-wrong-with-sam-3).
> **Tip:** Write simple English nouns: `window`, `wine glass`, `red car`. If a concept finds nothing, try another word (an unusual building may answer to `structure` or `monument` rather than `building`) or lower `threshold`. The report says which case it is: nothing found, a candidate that came close, or a word that was not understood.
> **Tip:** Ask for a whole and its parts in one formula: `person - hair`. The node looks for both and `mask` is the difference. `segs` still holds both, so [Segs Compose](#segs-compose-nh) can make other masks from the same run.
> **Licence:** SAM 3 is under Meta's SAM License. See [Before you start with SAM 3](#before-you-start-with-sam-3).
---
## Segs Select (NH)
<p align="center"><img src="../assets/nodes/NH_SegsSelect.svg" alt="Segs Select (NH) node" width="360"></p>
Keeps the instances you want, by label, score, size and position, in the order you choose.
**Use it when** you need only some of what SAM3 Segment found: the biggest object, the second person from the left, only the sure detections, or only the large windows.
| Input | Type | Default | Description |
|---|---|---|---|
| `segs` | NH_SEGS | — | The instances, from SAM3 Segment or another Segs node. |
| `labels` | STRING | empty | The labels to keep, separated by commas. `*` stands for any run of characters and `?` for one character (`*shirt`). Case does not matter, and `_` counts as a space (`red_car` finds `red car`). Labels with spaces need no quotes here: `wine glass, red car`. Common Vietnamese words are translated (`tóc` finds `hair`); the formula words `nền` and `tất cả` are not labels here. Empty keeps every label. |
| `min_score` | FLOAT | `0.0` | Range 0 to 1. Drops instances with a lower score. |
| `min_area` | FLOAT | `0.0` | Range 0 to 1. Drops instances smaller than this fraction of the image (`0.01` is 1%). |
| `max_area` | FLOAT | `1.0` | Range 0 to 1. Drops instances larger than this fraction of the image. |
| `sort_by` | COMBO | `score` | The order of the kept instances. See the table below. |
| `start` | INT | `0` | Range 0 to 4096. Skips this many instances of the sorted list. `0` starts at the first. |
| `count` | INT | `0` | Range 0 to 4096. Keeps this many instances from `start` on. `0` keeps all of them. |
| Output | Type | Description |
|---|---|---|
| `segs` | NH_SEGS | The kept instances, in the sorted order. |
| `mask` | MASK | The union of the kept instances, one mask per image. |
| `count` | INT | The number of instances kept, over the whole batch. |
| `sort_by` | Order |
|---|---|
| `score` | Highest score first. |
| `area` | Largest first. |
| `left_to_right`, `right_to_left` | By the horizontal centre of the box. |
| `top_to_bottom`, `bottom_to_top` | By the vertical centre of the box. |
| `center_first` | The instance whose box centre is nearest the centre of the image first. |
| `as_found` | The order SAM3 Segment found them in: your concepts in turn, best score first within each. |
**How it works**
- **Three steps, in this order:** the filters (`labels`, `min_score`, `min_area`, `max_area`), then the sorting, then the slice (`start` and `count`).
- **Per image.** Every image of a batch is handled on its own, so `start` `1` skips the first instance of each image.
- **Area** is the share of the image's pixels that the mask covers, not the size of its box. Positions use the centre of the box. Ties keep the order the instances were found in.
- The kept instances stay sorted, so [Segs To Masks](#segs-to-masks-nh) after this node returns its masks in that order.
| To get | Set |
|---|---|
| The second person from the left | `labels` `person`, `sort_by` `left_to_right`, `start` `1`, `count` `1` |
| The biggest object | `sort_by` `area`, `count` `1` |
| The three surest instances | `sort_by` `score`, `count` `3` |
| Nothing under 1% of the picture | `min_area` `0.01` |
| The instance nearest the middle | `sort_by` `center_first`, `count` `1` |
---
## Segs Compose (NH)
<p align="center"><img src="../assets/nodes/NH_SegsCompose.svg" alt="Segs Compose (NH) node" width="360"></p>
Builds one mask from the labels of the instances, with the same small formula as SAM3 Segment's `concepts`: union, minus, overlap and "everything except".
**Use it when** you already have instances and want "the background", "the person without the hair", "the wall without the windows", or another combination of what SAM3 Segment found: after [Segs Select](#segs-select-nh), or to make several masks from one SAM 3 run. For a single mask you can also write the same formula straight into SAM3 Segment's `concepts`.
| Input | Type | Default | Description |
|---|---|---|---|
| `segs` | NH_SEGS | — | The instances, from SAM3 Segment or Segs Select. |
| `expression` | STRING | `all` | The formula. See [Writing a formula](#writing-a-formula). The field has several lines: a line break lists, like a comma, and joins when it is next to an operator. |
| Output | Type | Description |
|---|---|---|
| `mask` | MASK | The result of the formula, one mask per image, at the size of the image. |
**What is different here.** The language is the one in [Writing a formula](#writing-a-formula), with these differences from SAM3 Segment:
- `or`, `and` and `not` are operators: `a or b` is `a | b`, `a and b` is `a & b`, `not a` is `~a`.
- A label picks the instances that have it. Case does not matter, `_` counts as a space (`red_car` finds `red car`), and `*` and `?` are wildcards (`*shirt`).
- Put a label that contains an operator character in quotes, `"` or `'` (`"salt & pepper"`), and use double quotes when it also has an apostrophe. Several words in a row need no quotes.
- Vietnamese labels are translated (`tóc` finds `hair`), also inside quotes. `nền` is the keyword `background`.
| Expression | You get |
|---|---|
| `person - hair` | The people without their hair (ask SAM3 Segment for `person, hair`). |
| `~person` | Everything except the people. |
| `background` | Everything outside every instance in the segs. Unlike `~person`, it also leaves out windows, chairs or anything else that was found. |
| `(wall \| ceiling) - window` | The walls and the ceiling with the window openings cut out. |
| `wine glass, person` | The glasses and the people together (a list, the same as `wine glass \| person`). |
| `all - person` | Everything that was found except the people. Unlike `~person`, it leaves out the background that nothing found. |
**How it works**
- **A label that no instance has** gives an empty part, and the console says so (`no instance is labelled 'x'; those parts are empty`). A typo in `~persn` therefore gives the whole picture: check the console when a mask looks wrong.
- **Words in a row are one label.** When no instance has that label but each word is the label of an instance (`person hair`), the node stops with an error that suggests `person + hair`. The same goes for a hyphen: when both `person` and `hair` were found, `person-hair` stops with an error that shows `person - hair`.
- **Every image** of a batch gets its own mask, built from its own instances.
- **A formula that cannot be read** stops the node with a message that gives the position of the problem, for example `unexpected '(' at position 8` for `person (adult)`.
<p align="center"><img src="../assets/examples/sam3-segment.jpg" alt="SAM3 Segment instances drawn by Segs Preview, and three masks built from them with Segs Compose and Segs Select on a sample photo" width="720"><br><em>The instances of "person, helmet, face" on a sample photo, drawn by Segs Preview, then three masks made from them with Segs Compose and Segs Select.</em></p>
---
## Segs To Masks (NH)
<p align="center"><img src="../assets/nodes/NH_SegsToMasks.svg" alt="Segs To Masks (NH) node" width="360"></p>
Turns the instances into ordinary masks: one mask per instance, the union per image, and the list of labels.
**Use it when** you need each instance as a mask of its own: to process them one by one, to save them, or to read their labels.
| Input | Type | Default | Description |
|---|---|---|---|
| `segs` | NH_SEGS | — | The instances, from SAM3 Segment or Segs Select. |
| Output | Type | Description |
|---|---|---|
| `masks` | MASK | One mask per instance, image after image, in the order of `labels`. When there is no instance at all, a single empty mask, so that nodes after it do not fail. |
| `union` | MASK | One mask per image: the union of its instances. |
| `labels` | STRING | The label of every instance, one per line, in the order of `masks`. |
| `count` | INT | The number of instances. It is 0 when there are none, although `masks` then holds one empty mask. |
> **Note:** Every mask has the size of the picture and 4 bytes per pixel: about 145 MB on a 7360×4912 photo. When the masks would need more than about half of your free RAM (for example hundreds of instances on an 8K photo), the node stops with a message instead of filling the memory. Use [Segs Select](#segs-select-nh) first to keep only the instances you need, or [Segs Compose](#segs-compose-nh) to get one mask.
---
## Segs To Impact SEGS (NH)
<p align="center"><img src="../assets/nodes/NH_SegsToImpactSEGS.svg" alt="Segs To Impact SEGS (NH) node" width="360"></p>
Converts the instances of one image to Impact Pack's `SEGS`, so that its detailers and SEGS tools can work on them.
**Use it when** you already use Impact Pack and want its detailers to refine what SAM3 Segment found. NH-Nodes needs Impact Pack for nothing else, so skip this node if you do not use it.
| Input | Type | Default | Description |
|---|---|---|---|
| `segs` | NH_SEGS | — | The instances, from SAM3 Segment or Segs Select. |
| `crop_factor` | FLOAT | `3.0` | Range 1 to 10. The size of the region a detailer works on around each instance, as a multiple of its box (Impact Pack's `crop_factor`). The region is moved to stay inside the image and always contains the box. |
| `batch_index` | INT | `0` | Range 0 to 4095. Impact Pack's SEGS hold one image, so this picks which image of the batch to convert. A number past the last image stops the node with a message. |
| Output | Type | Description |
|---|---|---|
| `segs` | SEGS | Impact Pack's `SEGS` for that image. Each instance has its box, crop region, mask, label and score (as `confidence`). |
**How it works**
- The output is Impact Pack's own type, not `NH_SEGS`. NH-Nodes builds it with the same fields and does not import Impact Pack, so the conversion needs no Impact Pack. Using the result does.
- To convert several images of a batch, use one node per `batch_index`.
---
## Segs Preview (NH)
<p align="center"><img src="../assets/nodes/NH_SegsPreview.svg" alt="Segs Preview (NH) node" width="360"></p>
Draws the instances on their picture: each one tinted in its own colour, outlined and labelled with its label and score.
**Use it when** you want to see the instances at the full size of your picture, or to see what Segs Select picked. SAM3 Segment already shows a small preview on itself (at most 1536 px, with its report). Put this node after SAM3 Segment to see everything, or after Segs Select to see only the selection.
| Input | Type | Default | Description |
|---|---|---|---|
| `image` | IMAGE | — | The picture the segs were found in: the same number of images and the same size. Otherwise the node stops and says so. |
| `segs` | NH_SEGS | — | The instances to draw. |
| Output | Type | Description |
|---|---|---|
| `preview` | IMAGE | The picture with the instances drawn on it, at the same size. |
The colours follow the order of the instances, each label sits above the highest point of its instance, and the size of the labels follows the size of the picture. An 8K picture gives an 8K preview.
---
## SAM 3 workflows
In each of these, check what SAM3 Segment found before you edit anything. Its preview on the node shows every instance with an outline and a label, and its report says what was not found. If an instance you want is missing, lower `threshold` (for example to 0.3) or try another word for the concept.
### The background behind people
```text
Load Image → SAM3 Segment (concepts: background)
→ Mask Morphology (NH)
→ edit the background
```
`background` is everything outside the people that SAM 3 found: the node looks for `person` on its own. When people hold or sit on things, name those too, so that they stay out of the background, for example the concepts `~(person + wine glass + chair)`. Move the edge of the mask a few pixels with [Mask Morphology (NH)](./mask.md#mask-morphology-nh): grow it so that the edit covers the old outline, or shrink it if the edit eats into the people. Then use the mask to blur, relight, recolour or inpaint the background.
### The biggest object, inpainted
```text
Load Image → SAM3 Segment (concepts: person, dog, car)
→ Segs Select (sort_by: area, count: 1)
→ Mask Morphology (NH, grow)
→ inpaint
```
With `labels` empty, Segs Select picks the biggest of everything you asked for. Set `labels` (for example `car`) to pick the biggest of one kind, `start` `1` for the second biggest, and raise `min_score` so that a doubtful detection cannot be picked. Grow the mask a few pixels so that the inpainted area covers the old outline.
### Architecture: a facade without windows
```text
Load Image → SAM3 Segment (concepts: building - (window + door))
→ recolour, restyle or inpaint the wall
```
`building` covers the whole facade with its windows and doors; the formula cuts them out and leaves the wall. For a room, use `(wall + ceiling) - window`. If `building` finds nothing on an unusual building, try `structure` or `monument` and change the formula to match. On a 4K to 8K photo, keep `balanced`: every window is segmented again on a close-up of the original pixels. The outline of a facade that fills the whole frame is accurate to about 1/300 of its width (see [How the two passes work](#how-the-two-passes-work)), so check the roofline in the preview.
---
## If something goes wrong with SAM 3
| What you see | What to do |
|---|---|
| An error that SAM 3 needs Meta's sam3 package | Install it as in [Before you start with SAM 3](#before-you-start-with-sam-3), then restart ComfyUI. |
| An error that the sam3 package is installed but failed to load, or lacks something | The installed version is not one NH-Nodes supports, or one of its extras is missing. Reinstall with the two lines in [Before you start with SAM 3](#before-you-start-with-sam-3). The message shows the cause. |
| An error that the checkpoint was not found | Download `sam3.pt` as in [Before you start with SAM 3](#before-you-start-with-sam-3), press R in ComfyUI and pick the file. The message lists the folders that were searched. |
| An error that a file is not a SAM 3 `.pt` checkpoint | Use `sam3.pt`. The safetensors files on Hugging Face do not work with this node. |
| An error that a file cannot be read as a SAM 3 checkpoint, or is not one | The file in the `sam3` folder is an incomplete download or another model's `.pt`. Download `sam3.pt` again as in [Before you start with SAM 3](#before-you-start-with-sam-3). |
| An error that SAM 3 needs an NVIDIA GPU with CUDA | ComfyUI is running on the CPU or on another kind of GPU. SAM3 Segment has no CPU mode. |
| An error that you must write at least one concept | `concepts` is empty, holds only comments, or holds only the keywords `all`, `image` or `none`. Write a word such as `person`. |
| An error that a concept has a wildcard | Remove the `*` or `?`: SAM 3 needs plain words. Wildcards work in Segs Select and Segs Compose. |
| An error from SAM3 Segment that gives a position, such as `unexpected '(' at position 8` | Check the formula at that position. A bracket in a label is read as grouping. See [Writing a formula](#writing-a-formula). |
| `count` is 0, or an instance is missing | Read the report: it says whether the word found nothing, a candidate came close (lower `threshold` to its score), or the word was not understood (write it in English). The preview on the node shows what was found. |
| The report says a word was not understood | It is a Vietnamese phrase that is not in the list. Write it in English. After a thing, `xanh` is ambiguous: write `xanh dương` (blue) or `xanh lá` (green). |
| The mask is the whole picture, or not what you expected | A list is a union: `person, background` is the person plus everything outside it. Write `background` alone, or a formula such as `person - bag`, and look at the preview: the bright part is the mask. |
| SAM3 Segment runs although nothing is connected to it | It is an output node. Mute or delete it when you do not need it; turning `show_preview` off does not stop it. |
| The console says it kept the first-pass mask for some instances | Their close-up did not match the first pass, so those instances keep the first pass's edges. Nothing is wrong. |
| An error from Segs Preview about sizes or the number of images | Connect the picture that the segs were found in. |
| An error from Segs Compose that gives a position | Check the formula at that position. See [Writing a formula](#writing-a-formula). |
| An error from Segs Compose that no instance is labelled `person hair`, but `person` and `hair` are | Words in a row are one label. Join them: `person + hair`. |
| A Segs Compose mask is empty, and the console says no instance has a label | Check the spelling of the label. Put spaces around a minus: `person - hair`, not `person-hair`. |
| An error from Segs To Masks that the full-size instance masks would need more memory than is free | There are too many instances for the size of the picture. Use Segs Select to keep only the ones you need, or Segs Compose to get one mask. |
| The console says that SAM 3 sees the picture squeezed into a square and can miss small things | The picture is wider or taller than 4:1. Split it into tiles with Image Tile (NH) and run the tiles. |
| ComfyUI runs out of VRAM after the node | Leave `keep_model_loaded` off, so the weights leave the GPU after each run. |
| ComfyUI's Unload Models button does not free the VRAM of SAM 3 | `keep_model_loaded` is on. Turn it off and run the node once, or restart ComfyUI. |
---
[← Back to guide](./README.md)
+16 -7
View File
@@ -4,13 +4,18 @@
# Hướng dẫn NH-Nodes
NH-Nodes bổ sung **69 node** cho ComfyUI, phục vụ công việc ảnh và sản xuất: nạp và lưu bộ ảnh, chia ô (tile) và ghép ảnh,
quản lý độ phân giải cho các model hiện nay, công cụ mask, phân đoạn trang phục và khuôn mặt, tạo mask VTON, rẽ nhánh
logic, xử lý prompt và văn bản, tiện ích batch và bộ nạp model theo chỉ số.
NH-Nodes bổ sung **75 node** cho ComfyUI, phục vụ công việc ảnh và sản xuất: nạp và lưu bộ ảnh, chia ô (tile) và ghép ảnh,
quản lý độ phân giải cho các model hiện nay, công cụ mask, phân đoạn trang phục và khuôn mặt, mask của bất cứ thứ gì bạn gọi tên
bằng SAM 3 qua một công thức ngắn, tạo mask VTON, rẽ nhánh logic, xử lý prompt và văn bản, tiện ích batch và bộ nạp model theo
chỉ số.
Tài liệu này giải thích từng node: dùng để làm gì, từng đầu vào và đầu ra, và cách node xử lý ở những chi tiết quan trọng.
Phần cài đặt và hướng dẫn nhanh nằm ở [trang dự án](../../README.md#installation).
> **Yêu cầu cho SAM 3:** SAM3 Segment cần GPU NVIDIA có CUDA, gói `sam3` của Meta và checkpoint `sam3.pt`, những thứ bạn
> tự cài và tự tải. Mọi node khác chạy được mà không cần chúng. Cách thiết lập:
> [Trước khi bắt đầu với SAM 3](vision.md#trước-khi-bắt-đầu-với-sam-3).
## Cách đọc thẻ node
Mỗi phần node bắt đầu bằng một thẻ được vẽ trực tiếp từ định nghĩa thật của node:
@@ -20,7 +25,7 @@ Mỗi phần node bắt đầu bằng một thẻ được vẽ trực tiếp t
- **Thanh tiêu đề** — tên hiển thị, bên dưới là mã node (gõ mã này vào ô tìm node) và nhóm menu trong `NH-Nodes`.
- **Chấm bên trái** là đầu vào để nối dây, **chấm bên phải** là đầu ra. Màu chấm cho biết kiểu dữ liệu: IMAGE xanh dương,
MASK xanh lá, LATENT hồng, MODEL tím, CLIP vàng, VAE đỏ, số (INT, FLOAT) xanh dương nhạt, chữ (STRING) xanh lá nhạt,
BOOLEAN hồng phấn, kiểu bất kỳ (`*`) màu xám, và dữ liệu riêng của NH (thông tin tile, thông tin resize…) màu hổ phách.
BOOLEAN hồng phấn, kiểu bất kỳ (`*`) màu xám, và dữ liệu riêng của NH (thông tin tile, thông tin resize, instance…) màu hổ phách.
- **Các hàng bo tròn** là widget kèm giá trị mặc định. `opt` đánh dấu đầu vào tùy chọn; `≡` sau một đầu ra nghĩa là đầu ra
đó trả về danh sách.
- Chấm **xanh lá** trên tiêu đề là node thường; chấm **cam** là node đầu ra (output node), luôn chạy kể cả khi không có gì
@@ -36,7 +41,7 @@ Mỗi phần node bắt đầu bằng một thẻ được vẽ trực tiếp t
| Ảnh — chỉnh sửa theo mask | [image-retouch](image-retouch.md) | NH Mask-Aware Resize · NH Agnostic Image Generator · NH Simple Face Paste |
| Độ phân giải | [resolution](resolution.md) | NH Smart Resolution Picker · NH Smart Ratio Image Resize · NH Ratio Preset Image Resize · NH Remove Resize Padding · NH Restore Image Geometry |
| Mask | [mask](mask.md) | Mask Morphology · Mask Properties · NH Create Box Mask · NH Mask Aspect Ratio Match |
| Vision | [vision](vision.md) | Garment Segment · Portrait Segment |
| Vision | [vision](vision.md) | Garment Segment · Portrait Segment · SAM3 Segment · Segs Select · Segs Compose · Segs To Masks · Segs To Impact SEGS · Segs Preview |
| VTON | [vton](vton.md) | VTON Ultimate Processor |
| Logic | [logic](logic.md) | Compare · Logic Gate · If/Else · Switch N · Any Switch · Any Branch Switch · Gate Switch · Value Match Index · Math Eval · Random Choice |
| Văn bản | [text](text.md) | Prompt Template · Prompt Scheduler · String Operations · Prompt Join · Text Split · Regex Extract · Text Index · Text Concatenate · Text Split Lines · Text Random Line |
@@ -58,13 +63,17 @@ Mỗi phần node bắt đầu bằng một thẻ được vẽ trực tiếp t
ComfyUI; chi tiết ở [image-io](image-io.md#đường-dẫn-và-loại-tệp).
- **Resize data.** Các node độ phân giải trả về `resize_data` ghi lại ảnh đã được cắt hay thêm viền thế nào, để đưa kết quả
về đúng khung ảnh gốc bằng [NH Restore Image Geometry](resolution.md#nh-restore-image-geometry).
- **Instance.** SAM3 Segment trả về mỗi thứ nó tìm thấy như một instance riêng, có nhãn, điểm và mask, trong một giá trị
`NH_SEGS`. Các node Segs chọn instance, ghép chúng bằng công thức và đổi chúng thành mask; xem
[Các instance: kiểu NH_SEGS](vision.md#các-instance-kiểu-nh_segs).
- **Giấy phép.** Mọi node theo giấy phép MIT, trừ các phần ghi trong [THIRD_PARTY_NOTICES](../../THIRD_PARTY_NOTICES.md):
VTON Ultimate Processor và Portrait Segment chỉ dùng cho mục đích **phi thương mại**, còn model của Garment Segment
không công bố giấy phép.
không công bố giấy phép. SAM3 Segment dùng SAM 3 của Meta theo SAM License, giấy phép không có điều khoản cấm thương mại
nhưng cấm các mục đích quân sự, hạt nhân, gián điệp, súng và vũ khí bất hợp pháp; bạn tự cài gói và tự tải checkpoint.
## Xem thêm
- [Changelog](../CHANGELOG.md) — thay đổi của từng phiên bản, gồm cả các thay đổi về hành vi.
- [Sơ đồ cổng logic](../logic_gate_diagram.html) — bảng chân trị trực quan cho node Logic Gate (tải về và mở bằng trình
duyệt).
- [Third-party notices](../../THIRD_PARTY_NOTICES.md) — giấy phép của mã nguồn đi kèm và các model được tải về.
- [Third-party notices](../../THIRD_PARTY_NOTICES.md) — giấy phép của mã nguồn đi kèm, các model được tải về và SAM 3.
+2 -1
View File
@@ -12,7 +12,7 @@ Ba node làm việc với mask. Một node đổi kích thước ảnh về đú
| [NH Agnostic Image Generator](#nh-agnostic-image-generator) | Xóa vùng mask bằng màu xám, nhiễu hoặc làm mờ |
| [NH Simple Face Paste](#nh-simple-face-paste) | Dán khuôn mặt trong mask của ảnh này lên vùng mask của ảnh khác |
Một `MASK` có màu trắng (`1`) ở nơi node cần làm việc và màu đen (`0`) ở những chỗ còn lại. Bạn lấy mask từ một node phân đoạn như [Garment Segment (NH)](./vision.md#garment-segment-nh) hoặc [Portrait Segment (NH)](./vision.md#portrait-segment-nh), hoặc từ bất kỳ công cụ mask nào trong ComfyUI.
Một `MASK` có màu trắng (`1`) ở nơi node cần làm việc và màu đen (`0`) ở những chỗ còn lại. Bạn lấy mask từ một node phân đoạn như [SAM3 Segment (NH)](./vision.md#sam3-segment-nh) (mọi vật thể gọi được tên, hoặc một công thức như `person - bag` hay `background`), [Garment Segment (NH)](./vision.md#garment-segment-nh) hoặc [Portrait Segment (NH)](./vision.md#portrait-segment-nh), hoặc từ bất kỳ công cụ mask nào trong ComfyUI.
---
@@ -172,6 +172,7 @@ Portrait Segment (NH) trên ảnh đích → NH Simple Face Paste.dest_mask
- Dùng các mask mà hộp bao quanh có tỉ lệ rất khác nhau. Khuôn mặt trông bị bẹp hoặc bị kéo dài.
- Đưa vào mask có kích thước khác ảnh của nó.
- Dùng mask của [Portrait Segment (NH)](./vision.md#portrait-segment-nh) trên một ảnh có nhiều khuôn mặt. Mask chứa mọi khuôn mặt, nên hộp bao quanh nó phủ cả chúng.
- Đặt `feathering` rất cao cho một khuôn mặt nhỏ. Phép làm mờ có thể nuốt mất cả hình dạng.
> **Mẹo:** Hãy làm `dest_mask` lớn hơn một chút so với khuôn mặt bạn thay, và giữ `source_mask` sát khuôn mặt. Khi đó biên mềm của mask nguồn hòa vào ảnh đích.
+468 -14
View File
@@ -2,17 +2,45 @@
# Vision (phân đoạn ảnh)
Hai node cắt từng phần trên ảnh chụp một người bằng các model SegFormer: trang phục với Garment Segment, và khuôn mặt, tóc, cổ với Portrait Segment. Mỗi node trả về một mask có cùng kích thước với ảnh đầu vào, một ảnh cắt nền có độ trong suốt và một ảnh xem trước mask. Bạn tìm thấy chúng trong **NH-Nodes → Vision**.
Tám node cắt từng phần trên ảnh chụp. Hai node dùng các model SegFormer hiểu về con người: trang phục với Garment Segment, và khuôn mặt, tóc, cổ với Portrait Segment. Mỗi node trả về một mask có cùng kích thước với ảnh đầu vào, một ảnh cắt nền có độ trong suốt và một ảnh xem trước mask. Sáu node còn lại dùng SAM 3, model phân đoạn của Meta: bạn gõ thứ mình muốn vào SAM3 Segment (người, ly rượu, cửa sổ, bầu trời, hoặc một công thức như `person - bag` hay `background`) và nó trả về mask, đồng thời hiển thị ngay trên node. Nó cũng giữ lại mọi instance (từng đối tượng riêng lẻ) tìm được, và năm node Segs giúp chọn, ghép, xem trước và chuyển đổi chúng. Bạn tìm thấy chúng trong **NH-Nodes → Vision**.
**Trên trang này**
- [Trước khi bắt đầu](#trước-khi-bắt-đầu) — model, lần chạy đầu tiên, giấy phép, bộ nhớ
- [Tinh chỉnh mask](#tinh-chỉnh-mask) — các tùy chọn dùng chung của hai node
- [Garment Segment (NH)](#garment-segment-nh) — mask trang phục thân trên, thân dưới, giày dép hoặc mũ nón
- [Portrait Segment (NH)](#portrait-segment-nh) — mask khuôn mặt, tóc, cổ và các bộ phận trên mặt
- [Nên dùng node nào?](#nên-dùng-node-nào) — các node SegFormer hay SAM 3
- **Các node SegFormer: trang phục và chân dung**
- [Trước khi bắt đầu](#trước-khi-bắt-đầu) — model, bộ nhận diện khuôn mặt, lần chạy đầu tiên, giấy phép, bộ nhớ
- [Cách các node SegFormer nhìn một bức ảnh](#cách-các-node-segformer-nhìn-một-bức-ảnh) — vì sao khuôn mặt và người nhỏ được xem cận cảnh
- [Tinh chỉnh mask](#tinh-chỉnh-mask) — các tùy chọn dùng chung của hai node
- [Garment Segment (NH)](#garment-segment-nh) — mask trang phục thân trên, thân dưới, giày dép hoặc mũ nón
- [Portrait Segment (NH)](#portrait-segment-nh) — mask khuôn mặt, tóc, cổ và các bộ phận trên mặt
- **Các node SAM 3: bất cứ thứ gì bạn gọi tên được**
- [Trước khi bắt đầu với SAM 3](#trước-khi-bắt-đầu-với-sam-3) — GPU, gói cài đặt, checkpoint, bộ nhớ, giấy phép
- [Cách hai lượt phân tích hoạt động](#cách-hai-lượt-phân-tích-hoạt-động) — `balanced` làm được gì trên ảnh 4K đến 8K, và giới hạn ở đâu
- [Các instance: kiểu NH_SEGS](#các-instance-kiểu-nh_segs) — thứ được chuyển qua lại giữa các node
- [Viết công thức](#viết-công-thức) — những gì `concepts` và `expression` nhận: danh sách, `-`, `&`, `~`, `background`, từ tiếng Việt
- [SAM3 Segment (NH)](#sam3-segment-nh) — gõ thứ bạn muốn, nhận mask của nó và xem ngay trên node
- [Segs Select (NH)](#segs-select-nh) — giữ instance theo nhãn, điểm, kích thước và vị trí
- [Segs Compose (NH)](#segs-compose-nh) — dựng mask từ nhãn bằng `|`, `-`, `&` và `~`
- [Segs To Masks (NH)](#segs-to-masks-nh) — mỗi instance một mask, cùng mask hợp và danh sách nhãn
- [Segs To Impact SEGS (NH)](#segs-to-impact-segs-nh) — đưa các instance cho detailer của Impact Pack
- [Segs Preview (NH)](#segs-preview-nh) — vẽ các instance lên ảnh của chúng
- [Workflow với SAM 3](#workflow-với-sam-3) — nền phía sau người, vật lớn nhất, mặt tiền không có cửa sổ
- [Khi có sự cố với SAM 3](#khi-có-sự-cố-với-sam-3)
## Nên dùng node nào?
| Bạn muốn | Dùng | Cần gì |
|---|---|---|
| Trang phục của một người hoặc cả nhóm: áo, quần, giày, mũ | [Garment Segment](#garment-segment-nh) | Một model SegFormer và một bộ nhận diện khuôn mặt nhỏ, đều tự tải ở lần chạy đầu. Chạy được trên CPU hoặc GPU. |
| Khuôn mặt, tóc, cổ và các bộ phận trên mặt của một ảnh chân dung, ảnh nửa người hoặc mọi khuôn mặt trong ảnh nhóm | [Portrait Segment](#portrait-segment-nh) | Giống như trên. |
| Bất cứ thứ gì bạn gọi tên được (người, đồ vật, bầu trời, tường, cửa sổ) dưới dạng mask từ một công thức ngắn (`person - bag`, `background`), hoặc từng instance riêng, kể cả trên ảnh 4K đến 8K | [SAM3 Segment](#sam3-segment-nh) và các node Segs | GPU NVIDIA, gói `sam3` của Meta và một checkpoint mà bạn tự cài và tự tải. Xem [Trước khi bắt đầu với SAM 3](#trước-khi-bắt-đầu-với-sam-3). |
Bạn cũng có thể yêu cầu SAM3 Segment tìm `dress` hay `hair`. Các node SegFormer vẫn là lựa chọn dễ hơn cho các bộ phận trang phục và khuôn mặt cố định của chúng: chúng tự thiết lập và cũng chạy được không cần GPU.
## Trước khi bắt đầu
Phần này nói về Garment Segment và Portrait Segment. SAM3 Segment và các node Segs có phần riêng: [Trước khi bắt đầu với SAM 3](#trước-khi-bắt-đầu-với-sam-3).
| | Garment Segment (NH) | Portrait Segment (NH) |
|---|---|---|
| Model | SegFormer huấn luyện cho các lớp trang phục của Fashionpedia | SegFormer tinh chỉnh (fine-tune) trên CelebAMask-HQ cho các bộ phận khuôn mặt |
@@ -20,17 +48,31 @@ Hai node cắt từng phần trên ảnh chụp một người bằng các model
| Dung lượng tải | khoảng 190 MB | khoảng 340 MB |
| Giấy phép | Người đăng tải không khai báo | Chỉ dành cho nghiên cứu và giáo dục phi thương mại |
**Lần chạy đầu tiên.** Khi chạy lần đầu, mỗi node tải model của nó từ Hugging Face. Mỗi lần tải được cố định (pinned) vào một phiên bản xác định của kho, nên thay đổi về sau ở nguồn không thể làm đổi thứ bạn nhận được. Các tệp được lưu vào `ComfyUI/models/NH-Nodes/`, mỗi model một thư mục con. Nếu đã có sẵn một bản đầy đủ thì node dùng luôn bản đó: trong `models/RMBG/segformer_fashion` với model trang phục, trong `models/face_parsing` hoặc `models/RMBG/segformer_face` với model khuôn mặt, hoặc trong bất kỳ thư mục nào bạn thêm bằng khóa `nh_nodes` trong `extra_model_paths.yaml` (xem ví dụ ở mục [Installation](../../README.md#installation)). Nếu quá trình tải bị gián đoạn, hãy chạy lại node: các tệp đã tải xong được giữ lại.
**Lần chạy đầu tiên.** Khi chạy lần đầu, mỗi node tải model của nó từ Hugging Face, cùng với bộ nhận diện khuôn mặt mô tả bên dưới. Mỗi lần tải được cố định (pinned) vào một phiên bản xác định của kho, nên thay đổi về sau ở nguồn không thể làm đổi thứ bạn nhận được. Các tệp được lưu vào `ComfyUI/models/NH-Nodes/`, mỗi model một thư mục con. Nếu đã có sẵn một bản đầy đủ thì node dùng luôn bản đó: trong `models/RMBG/segformer_fashion` với model trang phục, trong `models/face_parsing` hoặc `models/RMBG/segformer_face` với model khuôn mặt, hoặc trong bất kỳ thư mục nào bạn thêm bằng khóa `nh_nodes` trong `extra_model_paths.yaml` (xem ví dụ ở mục [Installation](../../README.md#installation)). Nếu quá trình tải bị gián đoạn, hãy chạy lại node: các tệp đã tải xong được giữ lại.
**Bộ nhận diện khuôn mặt.** Cả hai node còn dùng YuNet, một bộ nhận diện khuôn mặt nhỏ (khoảng 230 KB, giấy phép MIT, từ OpenCV Zoo), để tìm xem người đang ở đâu trước khi phân tích một bức ảnh. Nó được tải một lần, cố định vào một phiên bản xác định của `opencv/face_detection_yunet`, vào `ComfyUI/models/NH-Nodes/yunet/`. Nó chạy trên CPU bằng OpenCV mà NH-Nodes vốn đã cần, nên không tốn VRAM. Nếu đã có sẵn một bản trong thư mục con `yunet` của bất kỳ thư mục `nh_nodes` nào (`models/NH-Nodes/`, hoặc thư mục bạn thêm trong `extra_model_paths.yaml`) thì node dùng luôn bản đó; muốn chạy offline, hãy tự đặt `face_detection_yunet_2023mar.onnx` vào đó. Nếu không nạp được bộ nhận diện (offline ở lần chạy đầu, hoặc OpenCV quá cũ không chạy được nó), node in một cảnh báo trong console và phân tích cả bức ảnh, không có các ảnh cận cảnh nói ở [Cách các node SegFormer nhìn một bức ảnh](#cách-các-node-segformer-nhìn-một-bức-ảnh). Hãy khởi động lại ComfyUI để thử lại.
**Bộ nhớ.** Model được đọc từ đĩa một lần và giữ trong bộ nhớ hệ thống. Nó chỉ chuyển sang GPU trong lúc node chạy, rồi trở về bộ nhớ hệ thống, nên không chiếm VRAM giữa các lần chạy. Trước khi chuyển sang GPU, node nhờ ComfyUI gỡ bớt các model của chính nó nếu VRAM đang eo hẹp.
> **Lưu ý:** Nếu node báo nhãn không như mong đợi (unexpected labels), nghĩa là có một checkpoint khác nằm trong một thư mục mà node tìm. Hãy xóa bản đó, model đúng sẽ được tải về.
> **Giấy phép:** Model đứng sau Portrait Segment chỉ được dùng cho nghiên cứu và giáo dục phi thương mại. Model đứng sau Garment Segment không khai báo giấy phép, nên điều khoản của nó chưa rõ. Mã nguồn của NH-Nodes thuộc giấy phép MIT. Hãy đọc [THIRD_PARTY_NOTICES.md](../../THIRD_PARTY_NOTICES.md) trước khi dùng một trong hai node cho mục đích thương mại.
> **Giấy phép:** Model đứng sau Portrait Segment chỉ được dùng cho nghiên cứu và giáo dục phi thương mại. Model đứng sau Garment Segment không khai báo giấy phép, nên điều khoản của nó chưa rõ. Mã nguồn của NH-Nodes thuộc giấy phép MIT. Bộ nhận diện khuôn mặt YuNet theo giấy phép MIT và không thay đổi điều nào ở trên. Hãy đọc [THIRD_PARTY_NOTICES.md](../../THIRD_PARTY_NOTICES.md) trước khi dùng một trong hai node cho mục đích thương mại.
## Cách các node SegFormer nhìn một bức ảnh
Cả hai model đều nhìn một bức ảnh 512 × 512. Trên ảnh 6000 px, một khuôn mặt rộng 600 px chỉ còn rộng khoảng 50 px trong mắt model, nên các node không đưa cho model cả bức ảnh nguyên khối.
1. **Cả bức ảnh trước.** Ảnh có một cạnh dài hơn gấp đôi cỡ 512 px của model được thu nhỏ bằng cách lấy trung bình theo diện tích (area averaging), nhanh hơn phép resize riêng của model và giữ chi tiết nhỏ sạch sẽ, rồi mới được phân tích. Kết quả đầu tiên này được giữ lại ở mọi chỗ chưa có gì tốt hơn, nên những gì các bước sau bỏ sót vẫn giữ ít nhất kết quả này.
2. **Tìm người.** Bộ nhận diện khuôn mặt tìm khuôn mặt trên hai kích cỡ của bức ảnh. Garment Segment còn lấy thêm những người mà lượt phân tích đầu đã tìm thấy.
3. **Ảnh cận cảnh tinh chỉnh kết quả.** Mỗi khuôn mặt (Portrait Segment) hoặc mỗi nhóm người (Garment Segment) được phân tích lại trên một vùng cắt mà model thấy ở đủ 512 px. Nhãn của vùng cắt được đưa vào kết quả đầu, nhưng không đưa dọc các cạnh cắt của vùng, nơi model thấy vật bị cắt cụt, nên không có cạnh thẳng nào lộ ra trong tóc hay quần áo. Mức độ tin vào một vùng cắt khác nhau tùy node, như các mục Garment Segment và Portrait Segment mô tả.
Với ảnh dài hơn 4096 px, toàn bộ việc trên được làm trên một bản sao dài 4096 px rồi bản đồ nhãn được phóng lên đúng kích thước ảnh ở cuối: model chỉ nhìn 512 px mỗi lần nên nhãn của nó không chứa chi tiết mịn hơn, còn ghép các vùng cắt ở đủ kích thước 8K sẽ mất nhiều giây.
Mỗi node dùng điều này thế nào được nêu trong mục riêng của nó: [Garment Segment](#garment-segment-nh) và [Portrait Segment](#portrait-segment-nh).
## Tinh chỉnh mask
Cả hai node biến bản đồ nhãn của model thành mask, rồi tinh chỉnh theo thứ tự: `fill_holes`, `mask_expand`, `mask_blur`. Mask luôn có kích thước bằng ảnh đầu vào.
Garment Segment và Portrait Segment biến bản đồ nhãn của model thành mask, rồi tinh chỉnh theo thứ tự: `fill_holes`, `mask_expand`, `mask_blur`. Mask luôn có kích thước bằng ảnh đầu vào.
| Tùy chọn | Khoảng | Garment Segment | Portrait Segment | Tác dụng |
|---|---|---|---|---|
@@ -55,13 +97,13 @@ Batch (lô ảnh) đều dùng được: từng ảnh được phân đoạn l
<p align="center"><img src="../assets/nodes/NH_GarmentSegment.svg" alt="Garment Segment (NH) node" width="360"></p>
Tạo mask cho một nhóm trang phục, hoặc tất cả các nhóm cùng lúc, trên người trong ảnh.
Tạo mask cho một nhóm trang phục, hoặc tất cả các nhóm cùng lúc, trên những người trong ảnh.
**Dùng khi** bạn muốn có mask sạch của áo, quần, giày hoặc mũ để inpaint, đổi màu hoặc dùng trong workflow thử đồ.
| Đầu vào | Kiểu | Mặc định | Mô tả |
|---|---|---|---|
| `image` | IMAGE | — | Ảnh chụp một người. |
| `image` | IMAGE | — | Ảnh chụp một người hoặc một nhóm người. |
| `part` | COMBO | `upper_garment` | `upper_garment`, `lower_garment`, `footwear`, `headwear` hoặc `all_garments`. Xem bảng bên dưới. |
| `attach_parts` | BOOLEAN | `true` | Gộp túi, khóa kéo, nơ, bèo nhún và các chi tiết tương tự chạm vào trang phục đã chọn. |
| `mask_expand` | INT | `0` | Khoảng -64 đến 64. Mở rộng (+) hoặc thu hẹp (-) mask N pixel. |
@@ -90,6 +132,8 @@ Tạo mask cho một nhóm trang phục, hoặc tất cả các nhóm cùng lúc
- **Đầm và jumpsuit** thuộc `upper_garment`. Không có tùy chọn đầm riêng. Muốn mask áo và quần cùng lúc, hoặc đầm và giày, hãy chọn `all_garments`.
- **Chi tiết đi kèm (attached parts)** gồm túi (pocket), khóa kéo, họa tiết đính (appliqué), hạt cườm, nơ, hoa, tua rua, ruy băng, đinh tán, bèo nhún, kim sa (sequin) và tua (tassel). Khi bật `attach_parts`, một chi tiết chỉ được thêm vào khi nó chạm vào trang phục đã chọn (trong vòng vài pixel). Túi trên áo khoác sẽ nhập vào mask `upper_garment`, còn chiếc nơ trên một chiếc túi xách thì không.
- **Không bao giờ được chọn:** kính, găng tay, đồng hồ, túi xách, ví và ô không thuộc `part` nào.
- **Ảnh cận cảnh của người.** Node phân tích cả bức ảnh trước và giữ kết quả đó. Sau đó nó tìm những người (trong lượt phân tích ấy và bằng bộ nhận diện khuôn mặt) và cắt các vùng vuông, đúng hình dạng đầu vào của model, quanh từng nhóm người: một hàng các vùng cắt chồng lên nhau cho nhóm rộng, một vùng cho một người hoặc một nhóm gọn. Mỗi vùng được phân tích ở đủ 512 px của model, và vùng được nới rộng khi người bị tràn qua một cạnh của nó. Việc này giúp những người nhỏ trong một bức ảnh lớn. Xem [Cách các node SegFormer nhìn một bức ảnh](#cách-các-node-segformer-nhìn-một-bức-ảnh).
- **Vùng cắt chỉ tinh chỉnh.** Nhãn của nó chỉ thay kết quả đầu ở những chỗ model chắc hơn, cách xa cạnh cắt của vùng, và chỉ với các phần nằm trên những người đã tìm thấy. Không có gì bị bịa ra trên một phông trống, và không có cạnh thẳng nào của vùng cắt xẻ qua một món trang phục.
<p align="center"><img src="../assets/examples/garment-segment.jpg" alt="Mask của Garment Segment phủ lên một ảnh mẫu" width="720"><br><em>Garment Segment trên một ảnh mẫu: các vùng tô màu là mask của những phần đã chọn.</em></p>
@@ -101,6 +145,8 @@ Load Image → Garment Segment (NH, upper_garment) → mask → Mask Morphology
Hãy mở rộng mask thêm vài pixel bằng [Mask Morphology (NH)](./mask.md#mask-morphology-nh) để trang phục được inpaint có thể phủ lên đường viền cũ.
> **Lưu ý:** Model Fashionpedia bỏ sót một số áo, ví dụ chiếc áo cánh nằm dưới một chiếc mũ lớn. Một model trang phục tốt hơn đang được dự kiến.
> **Giấy phép:** model trang phục không khai báo giấy phép. Xem [Trước khi bắt đầu](#trước-khi-bắt-đầu).
---
@@ -109,13 +155,13 @@ Hãy mở rộng mask thêm vài pixel bằng [Mask Morphology (NH)](./mask.md#m
<p align="center"><img src="../assets/nodes/NH_PortraitSegment.svg" alt="Portrait Segment (NH) node" width="360"></p>
Tạo mask cho khuôn mặt, tóc, cổ và các bộ phận trên mặt của một ảnh chân dung. Bạn bật những vùng muốn lấy.
Tạo mask cho khuôn mặt, tóc, cổ và các bộ phận trên mặt của một ảnh chân dung, hoặc của mọi khuôn mặt trong ảnh nhóm. Bạn bật những vùng muốn lấy.
**Dùng khi** bạn muốn chỉnh sửa hoặc đổi màu tóc, xử lý da và các bộ phận trên mặt, hoặc bảo vệ khuôn mặt trong khi chỉnh phần còn lại của ảnh.
| Đầu vào | Kiểu | Mặc định | Mô tả |
|---|---|---|---|
| `image` | IMAGE | — | Một ảnh chân dung. |
| `image` | IMAGE | — | Một ảnh chân dung, ảnh nửa người hoặc ảnh nhóm. |
| `face` | BOOLEAN | `true` | Vùng khuôn mặt đặc: da, mũi, mắt, lông mày, kính mắt, môi và bên trong miệng. |
| `neck` | BOOLEAN | `true` | Cổ. |
| `hair` | BOOLEAN | `true` | Tóc. |
@@ -132,7 +178,7 @@ Tạo mask cho khuôn mặt, tóc, cổ và các bộ phận trên mặt của m
| Đầu ra | Kiểu | Mô tả |
|---|---|---|
| `mask` | MASK | Mask gộp của mọi vùng đã bật. |
| `mask` | MASK | Mask gộp của mọi vùng đã bật, trên mọi khuôn mặt tìm thấy. |
| `cutout_rgba` | IMAGE | Ảnh đầu vào với mask làm kênh alpha. |
| `mask_image` | IMAGE | Mask dưới dạng ảnh xem trước RGB màu xám. |
@@ -141,6 +187,10 @@ Tạo mask cho khuôn mặt, tóc, cổ và các bộ phận trên mặt của m
- Các vùng đã bật được gộp thành một mask. Không bật vùng nào thì mask rỗng.
- `face` đã bao gồm mắt, lông mày, mũi, môi và bên trong miệng. Các tùy chọn đó chỉ làm kết quả thay đổi khi `face` tắt.
- Mũ, hoa tai, dây chuyền và quần áo có trong model nhưng node không cung cấp.
- **Cả bức ảnh trước.** Bức ảnh được phân tích dưới dạng một hình vuông có thêm viền (giữ nguyên tỉ lệ, phần còn lại được lấp bằng màu xám) thay vì bị nén vào hình vuông của model. Kết quả này được giữ lại ở mọi chỗ mà các vùng cắt bên dưới không chạm tới, nên một bức ảnh không tìm thấy khuôn mặt nào, hoặc không có bộ nhận diện, giữ nó làm kết quả. Khi đã tìm thấy khuôn mặt, các phần của mặt (da, mắt, lông mày, mũi, môi, tai) mà kết quả này cho thấy ở xa mọi khuôn mặt được bỏ đi vì là nhầm lẫn, chẳng hạn một vệt phản chiếu hay một hoa văn; tóc, cổ và quần áo của nó vẫn được giữ.
- **Khuôn mặt.** Bộ nhận diện khuôn mặt tìm các khuôn mặt (tối đa 32, lấy những khuôn mặt lớn nhất). Khi không tìm thấy khuôn mặt nào, bức ảnh được thêm một lề xám và tìm thêm một lần nữa, cách này tìm được khuôn mặt bị khung hình cắt. Sau đó model phân tích (parser) kiểm tra từng khuôn mặt trên một vùng cắt sát: bàn tay hay một họa tiết mà bộ nhận diện tưởng là khuôn mặt bị loại và không làm thay đổi gì.
- **Hai vùng cắt cho mỗi khuôn mặt.** Một vùng cắt rộng cho thấy trọn tóc và cổ. Một vùng cắt sát, đóng khung giống những bức ảnh model đã học, cho mắt, lông mày, mũi và môi. Vùng rộng tinh chỉnh phần nằm trong nó, cách xa các cạnh cắt của nó, còn vùng sát chỉ quyết định bên trong một hình elip quanh khuôn mặt, nên không có cạnh cắt thẳng nào lộ ra trong mái tóc dài. Nơi các vùng cắt của hai khuôn mặt chồng nhau, mỗi điểm ảnh thuộc về khuôn mặt gần hơn. Vì vậy trong ảnh nhóm, mask chứa các vùng đã bật của mọi người.
- **Khuôn mặt lớn.** Khuôn mặt đã chiếm một phần lớn khung hình thì được giao cho lượt phân tích cả bức ảnh, vì lượt đó đã thấy nó đủ lớn. Xem [Cách các node SegFormer nhìn một bức ảnh](#cách-các-node-segformer-nhìn-một-bức-ảnh).
<p align="center"><img src="../assets/examples/portrait-segment.jpg" alt="Mask của Portrait Segment phủ lên một ảnh mẫu" width="720"><br><em>Portrait Segment trên một ảnh mẫu: các vùng tô màu là mask của những vùng đã bật.</em></p>
@@ -152,10 +202,414 @@ Load Image → Portrait Segment (NH) → mask → đổi màu hoặc inpaint
Tắt mọi vùng trừ `hair`. Với các giá trị mặc định, bạn có khuôn mặt, tai, cổ và tóc cùng trong một mask.
> **Mẹo:** Model được tinh chỉnh trên CelebAMask-HQ, một bộ ảnh chụp cận mặt. Nó hoạt động tốt nhất khi khuôn mặt chiếm một phần đáng kể của khung hình. Với khuôn mặt nhỏ trong ảnh lớn, hãy cắt (crop) quanh khuôn mặt trước.
> **Mẹo:** Model được tinh chỉnh trên CelebAMask-HQ, một bộ ảnh chụp cận mặt, nên node phân tích mỗi khuôn mặt tìm được trên một vùng cắt cận cảnh. Một khuôn mặt nhỏ trong ảnh lớn, một ảnh nửa người hay một ảnh nhóm không cần cắt (crop) trước. Khuôn mặt mà bộ nhận diện bỏ sót (rất nhỏ, bị che hoặc quay đi) giữ kết quả của lượt phân tích cả bức ảnh, thô hơn: hãy tự cắt quanh nó khi cần chi tiết mịn nhất.
> **Giấy phép:** model face-parsing chỉ dành cho nghiên cứu và giáo dục phi thương mại. Xem [Trước khi bắt đầu](#trước-khi-bắt-đầu).
---
## Trước khi bắt đầu với SAM 3
SAM 3 là model của Meta để tìm vật theo tên: bạn viết thứ mình muốn (`person`, `window`, `wine glass`) và nó trả về mọi instance của thứ đó, mỗi instance có điểm tin cậy (score) và mask riêng. NH-Nodes không kèm model và không bao giờ tự tải nó, nên SAM3 Segment cần bạn chuẩn bị ba thứ, chỉ một lần. Nếu một pack SAM 3 khác đã chạy tốt trong ComfyUI của bạn, bước 2 và 3 thường đã xong: pack đó đã cài gói và đặt `sam3.pt` đúng nơi node này tìm.
**1. GPU NVIDIA có CUDA.** SAM 3 chỉ chạy trên CUDA: mã sam3 chính thức dựng một phần model trên GPU, nên không có chế độ CPU và không có đầu vào `device`. Hãy dự trù khoảng 5 GB VRAM ở mức cao nhất.
**2. Gói `sam3` của Meta.** NH-Nodes không tự cài nó cho bạn. Trong Python đang chạy ComfyUI, hãy chạy hai dòng sau, rồi khởi động lại ComfyUI:
```text
python -m pip install --no-deps git+https://github.com/facebookresearch/sam3@b26a5f330e05d321afb39d01d3d4881f258f65ff
python -m pip install timm ftfy regex iopath
```
Dòng đầu cài đúng commit sam3 mà NH-Nodes đã được thử nghiệm; các commit mới hơn có cùng API ảnh cũng dùng được. Dòng này cần máy bạn có Git. `--no-deps` giữ nguyên numpy của bạn: sam3 yêu cầu numpy dưới 2, điều mà ComfyUI không cần. Trang của Meta cho gói này ghi Python 3.12 trở lên, PyTorch 2.7 trở lên và CUDA 12.6 trở lên. Nếu một pack SAM 3 khác đã cung cấp `sam3`, hãy bỏ qua dòng đầu: cài lại có thể thay thế bản mà pack đó đang dùng.
**3. Checkpoint `sam3.pt`.** Meta chỉ phát trọng số sau khi bạn chấp nhận SAM License của họ, nên NH-Nodes không thể tải giúp bạn. Hãy xin quyền truy cập tại [huggingface.co/facebook/sam3](https://huggingface.co/facebook/sam3), tải `sam3.pt` từ trang đó và đặt vào `ComfyUI/models/sam3/`. Nhấn R trong ComfyUI để làm mới danh sách, rồi chọn tệp trong `checkpoint`. Muốn để ở nơi khác, hãy thêm thư mục đó vào khóa `sam3` trong `extra_model_paths.yaml` (xem tệp này ở mục [Installation](../../README.md#installation)):
```yaml
my_models:
base_path: D:/ai-models/
sam3: sam3/ # checkpoint SAM 3, dùng chung với các pack SAM 3 khác
```
Chỉ checkpoint PyTorch `sam3.pt` dùng được. Bản safetensors trên Hugging Face dành cho thư viện transformers và dùng tên trọng số khác.
**Bộ nhớ.** Trọng số được đọc từ đĩa một lần và giữ trong bộ nhớ hệ thống (khoảng 3.6 GB). Chúng chuyển lên GPU khi SAM3 Segment chạy và trở về sau đó, nên ComfyUI nhận lại VRAM. Trước khi chuyển, node nhờ ComfyUI gỡ bớt các model của chính nó nếu VRAM đang eo hẹp. Hãy bật `keep_model_loaded` để trọng số ở lại trên GPU nếu bạn chạy node liên tục. Khi nó bật, nút **Unload Models** của ComfyUI không giải phóng chúng: hãy tắt `keep_model_loaded` và chạy node một lần, hoặc khởi động lại ComfyUI.
> **Giấy phép:** SAM 3 được phát hành theo SAM License của Meta. Giấy phép này miễn phí bản quyền (royalty-free) và không có điều khoản cấm thương mại, nhưng cấm những mục đích thuộc ITAR hoặc bị kiểm soát thương mại (trade controls) cấm, gồm quân sự hay chiến tranh, hạt nhân, gián điệp, súng và vũ khí bất hợp pháp. Mã nguồn của NH-Nodes thuộc giấy phép MIT và không chứa model lẫn gói `sam3`. Hãy đọc [THIRD_PARTY_NOTICES.md](../../THIRD_PARTY_NOTICES.md) trước khi dùng node cho mục đích thương mại.
---
## Cách hai lượt phân tích hoạt động
SAM 3 nhìn một bức ảnh cỡ 1000 px. Trên ảnh 6000 px, đó chỉ bằng một phần sáu bề ngang, nên vật nhỏ và các cạnh mảnh sẽ bị nhòe. Vì vậy SAM3 Segment nhìn hai lượt.
1. **Lượt đầu: cả bức ảnh.** Ảnh được thu nhỏ về đầu vào vuông của model (1008 px) và mã hóa một lần. Mọi concept bạn liệt kê được hỏi trên đúng lần mã hóa đó, rất nhẹ. Kết quả cho mỗi instance một điểm, một khung (box) và một mask đầu tiên, được lấy mẫu lại lên độ phân giải đầy đủ, chỉ trong phạm vi khung của chính nó.
2. **Lượt hai: ảnh cận cảnh của từng instance** (chỉ với `balanced`). Instance được cắt khỏi ảnh độ phân giải đầy đủ kèm một lề (20% cạnh dài của nó, tối thiểu 16 px) và được phân đoạn lại bằng bộ giải mã dùng khung gợi ý (box-prompt decoder) của SAM 3, lấy mask lượt đầu làm gợi ý. Lúc này cạnh của nó đến từ các điểm ảnh gốc.
3. **Cơ chế dự phòng.** Mask cận cảnh được giữ khi nó khớp với instance của lượt đầu trong phạm vi độ nhòe của lượt đầu, nên những thứ mảnh như cột và dây cáp vẫn còn. Khi không khớp, mask lượt đầu được giữ lại, và console cho biết số instance bị như vậy. Các mảnh vụn của mask mới không chạm vào mask lượt đầu bị bỏ đi.
| `detail_level` | Lượt | Dùng khi |
|---|---|---|
| `balanced` (mặc định) | Cả hai | Cạnh sắc nét cho vật nhỏ và vừa, kể cả trên ảnh 4K đến 8K. Khoảng 0.1 giây mỗi instance trên GPU nhanh. |
| `fast` | Chỉ lượt đầu | Ảnh nhỏ, và khi thử nhanh để tìm từ đúng cho concept. |
**`balanced` làm được và không làm được gì.** Vật nhỏ và vừa (một ô cửa sổ, một chiếc ly, một người trong ảnh nhóm) có cạnh sắc nét ngay cả trên ảnh 4K đến 8K, vì mỗi vật được xét trên một ảnh cận cảnh của các điểm ảnh gốc. Giới hạn nằm ở model: SAM 3 mô tả bất cứ thứ gì nó nhìn bằng một lưới khoảng 288 ô. Một đối tượng chiếm gần hết khung hình 8K tự nó là ảnh cận cảnh, nên cạnh của nó chỉ chính xác đến khoảng 1/300 kích thước của nó, tức cỡ 25 px với đối tượng rộng 7000 px. Mức đó ổn cho đường viền của một người hay một tòa nhà, và còn mềm ở tóc, ren và lông. Cạnh ở mức sợi tóc cần một bước tách nền mịn (matting) mà NH-Nodes chưa có; một node Matte Refine đang được dự kiến.
**Đo trên một máy** (NVIDIA RTX 5090): lần chạy đầu nạp model mất khoảng 9 giây. Một ảnh 7360×4912 với các concept `person, wine glass, dress` cho 13 instance trong khoảng 3.6 giây với `balanced`. Một ảnh nội thất 6720×4480 với 8 concept cho 29 instance trong khoảng 6 giây. Mức VRAM cao nhất khoảng 5 GB. GPU khác sẽ cho số khác.
---
## Các instance: kiểu NH_SEGS
`NH_SEGS` là dữ liệu mà SAM3 Segment chuyển cho các node Segs. Với mỗi ảnh trong batch (lô ảnh), nó chứa một danh sách instance. Mỗi instance giữ **nhãn** (các từ tiếng Anh mà SAM 3 được yêu cầu tìm: `áo đỏ` được lưu thành `red top`), **điểm tin cậy**, một **khung** ôm sát và mask **chỉ trong phạm vi khung đó**. Mask kích thước đầy đủ chỉ được dựng khi một node xuất `MASK`, nên hàng chục instance trên ảnh 8K vẫn nhẹ.
- Các instance xếp theo thứ tự các nhãn xuất hiện trong concepts của bạn, và theo điểm trong cùng một nhãn.
- Các node Segs xử lý từng ảnh trong batch riêng biệt.
- Các node gốc của ComfyUI không đọc được `NH_SEGS`. Để lấy mask ra, hãy dùng các đầu ra `mask`, [Segs Compose](#segs-compose-nh) hoặc [Segs To Masks](#segs-to-masks-nh). Để dùng detailer của Impact Pack, hãy chuyển đổi bằng [Segs To Impact SEGS](#segs-to-impact-segs-nh).
---
## Viết công thức
`concepts` của SAM3 Segment và `expression` của Segs Compose đọc cùng một ngôn ngữ nhỏ. Bạn gọi tên những thứ mình muốn, ghép chúng bằng các toán tử, và kết quả là một mask cho mỗi ảnh. Trong SAM3 Segment, các từ bạn viết cũng là thứ SAM 3 đi tìm. Trong Segs Compose, chúng chọn các instance đã được tìm sẵn.
| Viết | Nghĩa |
|---|---|
| `person` | Một nhãn: các instance mang tên đó. Không phân biệt chữ hoa/thường. |
| `red car`, `women's dress` | Nhiều từ liền nhau là một nhãn, không cần ngoặc kép. Dấu gạch nối nằm giữa hai chữ cái hoặc chữ số thuộc về nhãn (`t-shirt`, `x-ray`), dấu nháy đơn nằm giữa một từ cũng vậy. |
| `person, hair` | Một danh sách: hợp của các mục trong đó. Xuống dòng có tác dụng như dấu phẩy. Danh sách được đọc sau mọi toán tử khác, nên `person - hair, dress` là `person - hair` cộng với `dress`. Mục trống bị bỏ qua. |
| `a + b`, `a \| b` | Hợp: mọi thứ thuộc a hoặc thuộc b. |
| `a - b` | a bỏ đi phần của b. Hãy đặt dấu cách hai bên dấu trừ: `person-hair` được đọc là một nhãn duy nhất. |
| `a & b` | Giao: chỉ nơi cả a và b cùng có. |
| `~a`, `!a` | Mọi thứ trừ a: cả bức ảnh bỏ đi a. |
| `( … )` | Nhóm. |
| `background` | Mọi thứ nằm ngoài mọi instance đã tìm được. Giống `~all`. |
| `all` | Mọi instance đã tìm được. |
| `image` | Cả bức ảnh. |
| `none` | Một mask rỗng. |
| `# …` | Ghi chú, đến hết dòng. Nó bắt đầu ở nơi một từ có thể bắt đầu; nằm giữa một từ, như `car#2`, thì `#` thuộc về từ đó. |
| `"salt & pepper"` | Ngoặc kép giữ chung một nhãn có chứa ký tự toán tử. Một từ `all`, `image`, `none` hay `background` đặt trong ngoặc kép là nhãn thông thường. |
`~` được tính trước, rồi `&`, rồi `+`, `|` và `-` cùng cấp, từ trái sang phải, cuối cùng là danh sách: `a - b + c` nghĩa là `(a - b) + c`. Khi còn phân vân, hãy thêm dấu ngoặc.
**Xuống dòng.** Xuống dòng có tác dụng liệt kê, như dấu phẩy. Nó nối liền khi nằm cạnh một toán tử (dòng trước kết thúc bằng một toán tử, trừ `)`, hoặc dòng sau bắt đầu bằng `+`, `|`, `-`, `&` hay `)`), nên một công thức dài có thể được trình bày trên nhiều dòng:
```text
(wall + ceiling)
- window
```
**Từ tiếng Việt.** SAM 3 chỉ đọc tiếng Anh, nên các node dịch những từ tiếng Việt thông dụng bạn gõ: `tóc` là `hair`, `váy` là `dress`, `cửa sổ` là `window`. Danh sách gồm con người, cơ thể, quần áo, phụ kiện, thiên nhiên, phòng và công trình, phương tiện, đồ ăn và động vật. Danh sách đầy đủ nằm trong `core/vi_words.py`.
- Một thứ đi kèm màu sắc hoặc kích cỡ được đảo sang thứ tự của tiếng Anh: `áo đỏ` là `red top`, `tóc dài` là `long hair`, `ghế sofa lớn` là `large sofa`. `màu` có thể nằm giữa chúng: `áo màu đỏ`. `tóc vàng` là `blonde hair`.
- `xanh` đứng sau một thứ có thể là xanh dương hoặc xanh lá nên không được đoán. Hãy viết `xanh dương` (blue) hoặc `xanh lá` (green), như `áo xanh dương`, hoặc viết từ tiếng Anh.
- `các`, `những`, `mấy`, `cái`, `chiếc` và `con` đứng trước một thứ bị bỏ qua khi cụm từ không có trong danh sách đúng như đã viết: `những chiếc xe` là `vehicle`.
- `nền`, `phông`, `phông nền` và `hậu cảnh` là từ khóa `background`, cả khi có kèm màu (`nền trắng`). `tất cả` là `all`, còn `toàn ảnh` và `cả ảnh` là `image`.
- Hãy viết tiếng Việt có dấu. Văn bản không có dấu tiếng Việt được coi là tiếng Anh (một vài từ ngắn như `xe` có trong danh sách đúng như đã viết).
- Một cụm tiếng Việt không có trong danh sách sẽ được tìm đúng như đã viết, và SAM 3 thường không tìm thấy gì. Khi đó báo cáo ghi là không hiểu. Nhãn của các instance luôn là từ tiếng Anh.
- Ngoặc kép không chặn việc dịch. `nền` trong ngoặc kép là nhãn `background`, không phải từ khóa.
**Hai node khác nhau ở đâu**
| | SAM3 Segment, `concepts` | Segs Compose, `expression` |
|---|---|---|
| `or`, `and`, `not` | Là từ thông thường, nên một cụm như `black and white dog` đến với SAM 3 đúng như đã viết. | Là toán tử: `a or b`, `a and b`, `not a`. |
| Ký tự đại diện `*` và `?` | Bị từ chối: SAM 3 cần từ thông thường. | Khớp với nhãn (`*shirt`). |
| Nhãn làm gì | Là thứ SAM 3 được yêu cầu tìm. | Chọn các instance có nhãn đó. Nhãn mà không instance nào có cho một phần rỗng. |
| Ô trống | Node dừng lại với thông báo yêu cầu viết ít nhất một concept. Ô chỉ có ghi chú, hoặc chỉ có `all`, `image` hay `none`, cũng được tính là trống. | Node dừng lại với thông báo dùng `all` để lấy mọi instance. |
---
## SAM3 Segment (NH)
<p align="center"><img src="../assets/nodes/NH_SAM3Segment.svg" alt="SAM3 Segment (NH) node" width="360"></p>
Gõ thứ bạn muốn cắt ra và nhận mask của nó: một từ (`person`), một danh sách (`hair, face`) hoặc một công thức (`person - bag`, `background`). SAM 3 tìm mọi instance của những thứ bạn gọi tên. Node trả về mask của công thức, các instance dưới dạng `NH_SEGS`, một con số đếm và một báo cáo cho biết đã tìm thấy và chưa tìm thấy gì, đồng thời hiển thị những gì nó tìm được ngay trên node.
**Dùng khi** bạn muốn có mask của "mọi người", "mọi cửa sổ", "bầu trời", "phần nền" hay "người nhưng không có túi" trong một bức ảnh, kể cả ảnh 4K đến 8K, mà không phải vẽ gì. Với việc chọn tinh hơn (người thứ hai từ trái sang, vật lớn nhất), hoặc để tạo nhiều mask từ một lần chạy, hãy nối `segs` vào các [node Segs](#segs-select-nh).
Các đầu vào COMBO không nêu mặc định sẽ bắt đầu ở lựa chọn đầu tiên.
| Đầu vào | Kiểu | Mặc định | Mô tả |
|---|---|---|---|
| `image` | IMAGE | — | Bức ảnh. Node dùng màu (RGB), kênh alpha nếu có sẽ bị bỏ. Mọi ảnh trong batch được xử lý lần lượt. |
| `concepts` | STRING | `person` | Thứ cần tìm và mask cần trả về, viết dưới dạng công thức (xem [Viết công thức](#viết-công-thức)): các từ tiếng Anh ngắn (`person`, `red car`, `window`), dấu phẩy hoặc xuống dòng để liệt kê nhiều thứ, và `+`, `-`, `&`, `~`, dấu ngoặc cùng `background` để kết hợp chúng (`person - bag`). Các từ tiếng Việt thông dụng cũng dùng được (`tóc`, `áo đỏ`, `nền`). Hãy viết ít nhất một từ. |
| `threshold` | FLOAT | `0.5` | Khoảng 0.05 đến 0.95. Độ tin cậy tối thiểu. Thấp hơn thì tìm được nhiều instance hơn và cũng nhiều nhầm lẫn hơn; cao hơn chỉ giữ những cái chắc chắn. Điểm đúng bằng ngưỡng thì không được giữ. Khi một concept không tìm thấy gì, báo cáo cho biết ứng viên tốt nhất đã đến gần mức nào. |
| `detail_level` | COMBO | `balanced` | `balanced` hoặc `fast`. `balanced` phân đoạn lại mỗi instance lần thứ hai trên ảnh cận cảnh cắt từ ảnh độ phân giải đầy đủ, để vật nhỏ và vừa có cạnh sắc nét. `fast` giữ lượt đầu trên cả bức ảnh. Xem [Cách hai lượt phân tích hoạt động](#cách-hai-lượt-phân-tích-hoạt-động). |
| `max_per_concept` | INT | `0` | Khoảng 0 đến 200. Giữ tối đa bấy nhiêu instance cho mỗi concept, điểm cao nhất trước. `0` giữ tất cả. |
| `checkpoint` | COMBO | — | Một tệp `.pt` hoặc `.pth` của SAM 3 trong `models/sam3` hoặc thư mục khác đã đăng ký với tên `sam3`. Các pack SAM 3 khác dùng chung thư mục này. Khi thư mục chưa có tệp nào, danh sách hiện `sam3.pt` và node cho bạn biết lấy nó ở đâu khi bạn chạy. |
| `keep_model_loaded` | BOOLEAN | `false` | Tắt: sau lần chạy, trọng số trở về bộ nhớ hệ thống và ComfyUI nhận lại VRAM. Bật: chúng ở lại trên GPU (khoảng 3.6 GB), giúp lần chạy sau nhanh hơn, nhưng ComfyUI không dùng được phần VRAM đó trong lúc ấy, và nút **Unload Models** của nó không giải phóng được (hãy tắt tùy chọn này rồi chạy node một lần, hoặc khởi động lại ComfyUI). |
| `show_preview` | BOOLEAN | `true` | Tùy chọn (optional). Hiển thị kết quả ngay trên node: mask sáng và phần còn lại bị làm tối, các instance có viền và gắn nhãn, và các dòng báo cáo bên dưới. Tối đa 1536 px ở cạnh dài, và chỉ 16 ảnh đầu của một batch. Tắt: không vẽ gì, và node vẫn chạy (xem bên dưới). |
| Đầu ra | Kiểu | Mô tả |
|---|---|---|
| `segs` | NH_SEGS | Các instance của từng ảnh, mỗi instance có nhãn, điểm, khung và mask. Nhãn là các từ tiếng Anh mà SAM 3 được yêu cầu tìm. Khi công thức dùng `background`, chúng bao gồm cả các instance `person` đã được tìm cho nó. Hãy nối nó vào các node Segs. |
| `mask` | MASK | Mask của công thức, một mask cho mỗi ảnh, có kích thước bằng ảnh. Một danh sách từ đơn thuần cho hợp của mọi instance. |
| `count` | INT | Số instance tìm được, tính trên cả batch, gồm cả các instance `person` mà `background` đã tìm. |
| `report` | STRING | Những gì yêu cầu đã tìm được, mỗi từ một dòng: số lượng và điểm cao nhất, bản dịch, các trường hợp suýt đạt, các từ không hiểu, và ý nghĩa của `background`. Xem bên dưới. |
<p align="center"><img src="../assets/examples/sam3-formula.jpg" alt="Thứ SAM3 Segment hiển thị trên node với bốn công thức trên một ảnh mẫu: background, person - face, hair và face, và mọi thứ trừ người và các mũ bảo hiểm" width="720"><br><em>SAM3 Segment trên một ảnh mẫu, đúng như node hiển thị. Hàng trên: "background" làm sáng mọi thứ nằm ngoài người, và báo cáo nằm bên dưới bức ảnh. Hàng dưới: ba công thức khác; phần sáng là đầu ra mask của mỗi công thức.</em></p>
**Báo cáo.** Dòng đầu của `report` cho tổng số. Mỗi dòng sau đó (chúng cũng được viết bên dưới bản xem trước) bắt đầu bằng thứ bạn đã viết:
| Dòng | Nghĩa |
|---|---|
| `person: 2 found (best 0.91)` | Số instance tìm được và điểm cao nhất. |
| `áo đỏ -> red top: 1 found (best 0.64)` | Cụm tiếng Việt đã được dịch thành gì. |
| `hair: none at threshold 0.50; the best candidate scored 0.31, lower the threshold to keep it` | Không có gì vượt `threshold`, nhưng có một ứng viên suýt đạt. Hãy hạ `threshold` xuống dưới điểm của nó để giữ lại. |
| `bag: none found; try another English word` | Không ứng viên nào đạt nổi 0.1. |
| `áo xanh: not understood; write it in English (SAM 3 reads English only)` | Một cụm tiếng Việt không có trong danh sách và không tìm thấy gì. |
| `background = everything outside person, red top` | Ý nghĩa của `background`: mọi thứ nằm ngoài các instance của những nhãn này. |
| `person (searched for background): 2 found` | Lần tìm `person` mà node thêm vào vì công thức dùng `background` mà không nêu `person`. |
**Cách hoạt động**
- **Nhãn.** Các từ tiếng Anh mà SAM 3 được yêu cầu tìm là nhãn của các instance: `áo đỏ` được tìm là `red top`, và các instance của nó mang nhãn `red top`. Các node Segs bỏ qua chữ hoa/thường khi so khớp nhãn, nên cách viết hoa không thành vấn đề.
- **Mask.** `mask` là kết quả của công thức. Với `person - bag`, SAM 3 tìm `person` và `bag`, `mask` là người trừ đi các túi, và `segs` vẫn chứa cả hai loại instance.
- **`background`.** Là mọi thứ nằm ngoài mọi instance đã tìm được, nên node cần biết chủ thể nằm ở đâu. Khi công thức dùng nó mà không nêu `person`, SAM 3 cũng tìm `person`; các instance đó thuộc `segs` và `count` và có viền trong bản xem trước. Muốn loại cả những thứ khác ra khỏi phần nền, hãy nêu tên chúng: `~(person + wine glass + chair)`. Danh sách là phép hợp, nên `person, background` là cả bức ảnh.
- **Thứ tự và bản trùng.** Các instance xếp theo thứ tự các nhãn xuất hiện trong công thức (`person` do `background` thêm vào đứng cuối), điểm cao nhất trước trong mỗi nhãn. Hai instance của cùng một nhãn mà mask gần như trùng hoàn toàn (từ 90% trở lên) được tính là một, giữ lại cái điểm cao hơn. Sau đó `max_per_concept` giữ những cái tốt nhất trong phần còn lại. Các instance có điểm đúng bằng nhau giữ thứ tự mà SAM 3 đưa ra.
- **Xem kết quả.** Khi `show_preview` bật, node tự vẽ bản xem trước: không cần node nào khác. Muốn xem ảnh ở kích thước đầy đủ (bản xem trước tối đa 1536 px), hoặc xem Segs Select đang làm việc với gì, hãy nối `segs` và đúng bức ảnh đó vào [Segs Preview](#segs-preview-nh).
- **Tự chạy.** SAM3 Segment là một node đầu ra (output node; chấm trên thẻ của nó màu cam): ComfyUI chạy nó kể cả khi không có gì nối vào các đầu ra của nó. Vì vậy một node bạn không còn cần vẫn tiếp tục chạy SAM 3 mỗi khi đầu vào của nó thay đổi, và tắt `show_preview` không làm nó ngừng chạy. Hãy vô hiệu hóa nó (Mute) hoặc xóa nó. Chừng nào đầu vào giữ nguyên, ComfyUI dùng lại kết quả lần trước.
- **Không tìm thấy gì** không phải lỗi: bạn nhận được không instance nào và `count` bằng 0. Mask rỗng với danh sách từ đơn thuần, và là cả bức ảnh với một công thức như `background` hay `~person`, vì không có gì được tìm thấy để loại ra.
- **Batch.** Mỗi ảnh được xử lý lần lượt và có danh sách instance riêng. Thanh tiến trình đếm theo concept, và bạn có thể hủy lần chạy giữa các concept và giữa các ảnh cận cảnh.
- **Vật bị khung hình cắt.** SAM 3 làm mờ dần mask của nó trong một dải dọc biên của hình vuông nó làm việc, rộng khoảng 0.3% cạnh của bức ảnh. Mask của một vật tràn ra khỏi ảnh vẫn chạm tới mép ảnh: dải đó được nâng lên ở nơi vật chạm tới nó từ bên trong. Cái giá là một vật kết thúc trong dải đó, ngay phía trên mép ảnh, bị vẽ kéo xuống tận mép.
- **Ảnh rất rộng hoặc rất cao.** Trước hết SAM 3 nhìn cả bức ảnh bị nén vào một hình vuông. Với ảnh rộng hoặc cao hơn 4:1 (ảnh toàn cảnh), nó có thể bỏ sót vật nhỏ, và console cho biết điều đó. Hãy chia ảnh như vậy thành các ô bằng [Image Tile (NH)](./image-compose.md#image-tile-nh) rồi chạy từng ô.
- **Lỗi cài đặt** làm node dừng lại với thông báo nói rõ cần làm gì: thiếu gói, thiếu checkpoint, không có GPU CUDA. Xem [Khi có sự cố với SAM 3](#khi-có-sự-cố-với-sam-3).
> **Mẹo:** Hãy viết danh từ tiếng Anh đơn giản: `window`, `wine glass`, `red car`. Nếu một concept không tìm thấy gì, hãy thử từ khác (một tòa nhà có hình dạng lạ có thể đáp ứng `structure` hoặc `monument` thay vì `building`) hoặc hạ `threshold`. Báo cáo cho biết đó là trường hợp nào: không tìm thấy gì, một ứng viên suýt đạt, hay một từ không được hiểu.
> **Mẹo:** Hãy yêu cầu cả tổng thể lẫn các phần của nó trong cùng một công thức: `person - hair`. Node tìm cả hai và `mask` là phần chênh lệch. `segs` vẫn chứa cả hai, nên [Segs Compose](#segs-compose-nh) có thể tạo các mask khác từ cùng lần chạy đó.
> **Giấy phép:** SAM 3 thuộc SAM License của Meta. Xem [Trước khi bắt đầu với SAM 3](#trước-khi-bắt-đầu-với-sam-3).
---
## Segs Select (NH)
<p align="center"><img src="../assets/nodes/NH_SegsSelect.svg" alt="Segs Select (NH) node" width="360"></p>
Giữ lại những instance bạn muốn, theo nhãn, điểm, kích thước và vị trí, theo thứ tự bạn chọn.
**Dùng khi** bạn chỉ cần một phần những gì SAM3 Segment tìm được: vật lớn nhất, người thứ hai từ trái sang, chỉ những kết quả chắc chắn, hay chỉ những ô cửa lớn.
| Đầu vào | Kiểu | Mặc định | Mô tả |
|---|---|---|---|
| `segs` | NH_SEGS | — | Các instance, từ SAM3 Segment hoặc một node Segs khác. |
| `labels` | STRING | rỗng | Các nhãn cần giữ, cách nhau bằng dấu phẩy. `*` thay cho một chuỗi ký tự bất kỳ và `?` thay cho một ký tự (`*shirt`). Không phân biệt chữ hoa/thường, và `_` được coi là dấu cách (`red_car` tìm `red car`). Nhãn có dấu cách không cần ngoặc kép ở đây: `wine glass, red car`. Các từ tiếng Việt thông dụng được dịch (`tóc` tìm `hair`); các từ công thức `nền` và `tất cả` không phải nhãn ở đây. Để trống thì giữ mọi nhãn. |
| `min_score` | FLOAT | `0.0` | Khoảng 0 đến 1. Loại các instance có điểm thấp hơn. |
| `min_area` | FLOAT | `0.0` | Khoảng 0 đến 1. Loại các instance nhỏ hơn tỉ lệ này của bức ảnh (`0.01` là 1%). |
| `max_area` | FLOAT | `1.0` | Khoảng 0 đến 1. Loại các instance lớn hơn tỉ lệ này của bức ảnh. |
| `sort_by` | COMBO | `score` | Thứ tự của các instance được giữ. Xem bảng bên dưới. |
| `start` | INT | `0` | Khoảng 0 đến 4096. Bỏ qua bấy nhiêu instance đầu của danh sách đã sắp xếp. `0` bắt đầu từ instance đầu tiên. |
| `count` | INT | `0` | Khoảng 0 đến 4096. Giữ bấy nhiêu instance kể từ `start`. `0` giữ tất cả. |
| Đầu ra | Kiểu | Mô tả |
|---|---|---|
| `segs` | NH_SEGS | Các instance được giữ, theo thứ tự đã sắp xếp. |
| `mask` | MASK | Hợp của các instance được giữ, một mask cho mỗi ảnh. |
| `count` | INT | Số instance được giữ, tính trên cả batch. |
| `sort_by` | Thứ tự |
|---|---|
| `score` | Điểm cao nhất trước. |
| `area` | Lớn nhất trước. |
| `left_to_right`, `right_to_left` | Theo tâm ngang của khung. |
| `top_to_bottom`, `bottom_to_top` | Theo tâm dọc của khung. |
| `center_first` | Instance có tâm khung gần tâm bức ảnh nhất đứng trước. |
| `as_found` | Theo thứ tự SAM3 Segment tìm ra: lần lượt từng concept của bạn, điểm cao nhất trước trong mỗi concept. |
**Cách hoạt động**
- **Ba bước, theo thứ tự này:** các bộ lọc (`labels`, `min_score`, `min_area`, `max_area`), rồi sắp xếp, rồi cắt lấy một đoạn (`start` và `count`).
- **Theo từng ảnh.** Mỗi ảnh trong batch được xử lý riêng, nên `start` `1` bỏ qua instance đầu tiên của từng ảnh.
- **Diện tích** là tỉ lệ số điểm ảnh của bức ảnh mà mask phủ, không phải kích thước khung của nó. Vị trí dùng tâm của khung. Khi bằng nhau, thứ tự lúc tìm ra được giữ nguyên.
- Các instance được giữ vẫn theo thứ tự đã sắp, nên [Segs To Masks](#segs-to-masks-nh) đặt sau node này trả về các mask theo đúng thứ tự đó.
| Muốn có | Đặt |
|---|---|
| Người thứ hai từ trái sang | `labels` `person`, `sort_by` `left_to_right`, `start` `1`, `count` `1` |
| Vật lớn nhất | `sort_by` `area`, `count` `1` |
| Ba instance chắc chắn nhất | `sort_by` `score`, `count` `3` |
| Không có gì nhỏ hơn 1% bức ảnh | `min_area` `0.01` |
| Instance gần giữa ảnh nhất | `sort_by` `center_first`, `count` `1` |
---
## Segs Compose (NH)
<p align="center"><img src="../assets/nodes/NH_SegsCompose.svg" alt="Segs Compose (NH) node" width="360"></p>
Tạo một mask từ nhãn của các instance, bằng chính công thức nhỏ của `concepts` trong SAM3 Segment: hợp, trừ, giao và "mọi thứ trừ...".
**Dùng khi** bạn đã có sẵn các instance và muốn có "phần nền", "người nhưng không có tóc", "bức tường nhưng không có cửa sổ", hoặc một tổ hợp khác của những gì SAM3 Segment tìm được: sau [Segs Select](#segs-select-nh), hoặc để tạo nhiều mask từ một lần chạy SAM 3. Với một mask duy nhất, bạn cũng có thể viết đúng công thức đó thẳng vào `concepts` của SAM3 Segment.
| Đầu vào | Kiểu | Mặc định | Mô tả |
|---|---|---|---|
| `segs` | NH_SEGS | — | Các instance, từ SAM3 Segment hoặc Segs Select. |
| `expression` | STRING | `all` | Công thức. Xem [Viết công thức](#viết-công-thức). Ô này có nhiều dòng: xuống dòng có tác dụng liệt kê, như dấu phẩy, và nối liền khi nằm cạnh một toán tử. |
| Đầu ra | Kiểu | Mô tả |
|---|---|---|
| `mask` | MASK | Kết quả của công thức, một mask cho mỗi ảnh, có kích thước bằng ảnh. |
**Ở đây có gì khác.** Ngôn ngữ chính là ngôn ngữ trong [Viết công thức](#viết-công-thức), với những khác biệt sau so với SAM3 Segment:
- `or`, `and` và `not` là toán tử: `a or b` là `a | b`, `a and b` là `a & b`, `not a` là `~a`.
- Một nhãn chọn các instance mang nhãn đó. Không phân biệt chữ hoa/thường, `_` được coi là dấu cách (`red_car` tìm `red car`), và `*`, `?` là ký tự đại diện (wildcard) (`*shirt`).
- Đặt nhãn có chứa ký tự toán tử trong ngoặc kép, `"` hoặc `'` (`"salt & pepper"`), và dùng ngoặc kép đôi khi nhãn còn có dấu nháy đơn. Nhiều từ liền nhau không cần ngoặc kép.
- Nhãn tiếng Việt được dịch (`tóc` tìm `hair`), cả khi nằm trong ngoặc kép. `nền` là từ khóa `background`.
| Biểu thức | Bạn nhận được |
|---|---|
| `person - hair` | Người nhưng bỏ phần tóc (hãy yêu cầu SAM3 Segment `person, hair`). |
| `~person` | Mọi thứ trừ người. |
| `background` | Mọi thứ nằm ngoài mọi instance trong segs. Khác với `~person`, nó cũng loại cả cửa sổ, ghế hay bất cứ thứ gì khác đã tìm được. |
| `(wall \| ceiling) - window` | Tường và trần, khoét bỏ các ô cửa sổ. |
| `wine glass, person` | Các ly rượu và người cùng lúc (một danh sách, giống `wine glass \| person`). |
| `all - person` | Mọi thứ đã tìm được trừ người. Khác với `~person`, nó không gồm phần nền mà không instance nào phủ tới. |
**Cách hoạt động**
- **Nhãn mà không instance nào có** cho một phần rỗng, và console báo điều đó (`no instance is labelled 'x'; those parts are empty`). Vì thế gõ sai `~persn` sẽ cho cả bức ảnh: hãy xem console khi mask trông lạ.
- **Các từ liền nhau là một nhãn.** Khi không instance nào có nhãn đó nhưng mỗi từ lại là nhãn của một instance (`person hair`), node dừng với một lỗi gợi ý `person + hair`. Với dấu gạch nối cũng vậy: khi cả `person` và `hair` đều có, `person-hair` làm node dừng với lỗi cho thấy `person - hair`.
- **Mỗi ảnh** trong batch có mask riêng, dựng từ chính các instance của nó.
- **Công thức không đọc được** làm node dừng lại với thông báo nêu vị trí của chỗ sai, ví dụ `unexpected '(' at position 8` với `person (adult)`.
<p align="center"><img src="../assets/examples/sam3-segment.jpg" alt="Các instance của SAM3 Segment do Segs Preview vẽ, và ba mask tạo từ chúng bằng Segs Compose và Segs Select trên một ảnh mẫu" width="720"><br><em>Các instance của "person, helmet, face" trên một ảnh mẫu, do Segs Preview vẽ, rồi ba mask tạo từ chúng bằng Segs Compose và Segs Select.</em></p>
---
## Segs To Masks (NH)
<p align="center"><img src="../assets/nodes/NH_SegsToMasks.svg" alt="Segs To Masks (NH) node" width="360"></p>
Đổi các instance thành mask thông thường: một mask cho mỗi instance, mask hợp của từng ảnh, và danh sách nhãn.
**Dùng khi** bạn cần mỗi instance là một mask riêng: để xử lý từng cái một, để lưu lại, hoặc để đọc nhãn của chúng.
| Đầu vào | Kiểu | Mặc định | Mô tả |
|---|---|---|---|
| `segs` | NH_SEGS | — | Các instance, từ SAM3 Segment hoặc Segs Select. |
| Đầu ra | Kiểu | Mô tả |
|---|---|---|
| `masks` | MASK | Một mask cho mỗi instance, ảnh này sang ảnh khác, theo thứ tự của `labels`. Khi hoàn toàn không có instance nào, đầu ra là một mask rỗng duy nhất, để các node phía sau không bị lỗi. |
| `union` | MASK | Một mask cho mỗi ảnh: hợp các instance của ảnh đó. |
| `labels` | STRING | Nhãn của từng instance, mỗi dòng một nhãn, theo thứ tự của `masks`. |
| `count` | INT | Số instance. Bằng 0 khi không có instance nào, dù lúc đó `masks` chứa một mask rỗng. |
> **Lưu ý:** Mỗi mask có kích thước bằng bức ảnh và 4 byte cho mỗi điểm ảnh: khoảng 145 MB trên ảnh 7360×4912. Khi các mask cần hơn khoảng một nửa dung lượng RAM còn trống của bạn (ví dụ hàng trăm instance trên ảnh 8K), node dừng lại với một thông báo thay vì làm đầy bộ nhớ. Hãy dùng [Segs Select](#segs-select-nh) trước để chỉ giữ những instance bạn cần, hoặc [Segs Compose](#segs-compose-nh) để lấy một mask duy nhất.
---
## Segs To Impact SEGS (NH)
<p align="center"><img src="../assets/nodes/NH_SegsToImpactSEGS.svg" alt="Segs To Impact SEGS (NH) node" width="360"></p>
Chuyển các instance của một ảnh sang `SEGS` của Impact Pack, để các detailer và công cụ SEGS của nó làm việc được với chúng.
**Dùng khi** bạn đã dùng Impact Pack và muốn các detailer của nó tinh chỉnh những gì SAM3 Segment tìm được. NH-Nodes không cần Impact Pack cho bất kỳ việc nào khác, nên hãy bỏ qua node này nếu bạn không dùng nó.
| Đầu vào | Kiểu | Mặc định | Mô tả |
|---|---|---|---|
| `segs` | NH_SEGS | — | Các instance, từ SAM3 Segment hoặc Segs Select. |
| `crop_factor` | FLOAT | `3.0` | Khoảng 1 đến 10. Kích thước vùng mà detailer xử lý quanh mỗi instance, tính bằng bội số của khung của nó (cũng là `crop_factor` của Impact Pack). Vùng được dịch để nằm trong ảnh và luôn chứa trọn khung. |
| `batch_index` | INT | `0` | Khoảng 0 đến 4095. SEGS của Impact Pack chỉ chứa một ảnh, nên đầu vào này chọn ảnh nào trong batch được chuyển. Số vượt quá ảnh cuối làm node dừng lại với một thông báo. |
| Đầu ra | Kiểu | Mô tả |
|---|---|---|
| `segs` | SEGS | `SEGS` của Impact Pack cho ảnh đó. Mỗi instance có khung, vùng cắt (crop region), mask, nhãn và điểm (là `confidence`). |
**Cách hoạt động**
- Đầu ra là kiểu riêng của Impact Pack, không phải `NH_SEGS`. NH-Nodes dựng nó với cùng các trường và không import Impact Pack, nên việc chuyển đổi không cần Impact Pack. Còn dùng kết quả thì cần.
- Muốn chuyển nhiều ảnh trong batch, hãy dùng mỗi `batch_index` một node.
---
## Segs Preview (NH)
<p align="center"><img src="../assets/nodes/NH_SegsPreview.svg" alt="Segs Preview (NH) node" width="360"></p>
Vẽ các instance lên ảnh của chúng: mỗi instance được tô màu riêng, có viền và gắn nhãn cùng điểm của nó.
**Dùng khi** bạn muốn xem các instance ở kích thước đầy đủ của bức ảnh, hoặc xem Segs Select đã chọn gì. SAM3 Segment đã hiển thị một bản xem trước nhỏ ngay trên node (tối đa 1536 px, kèm báo cáo). Hãy đặt node này sau SAM3 Segment để xem tất cả, hoặc sau Segs Select để chỉ xem phần đã chọn.
| Đầu vào | Kiểu | Mặc định | Mô tả |
|---|---|---|---|
| `image` | IMAGE | — | Bức ảnh mà segs được tìm ra từ đó: cùng số ảnh và cùng kích thước. Nếu không, node dừng lại và nói rõ. |
| `segs` | NH_SEGS | — | Các instance cần vẽ. |
| Đầu ra | Kiểu | Mô tả |
|---|---|---|
| `preview` | IMAGE | Bức ảnh với các instance được vẽ lên, cùng kích thước. |
Màu sắc theo thứ tự của các instance, mỗi nhãn nằm phía trên điểm cao nhất của instance của nó, và cỡ chữ của nhãn theo kích thước bức ảnh. Ảnh 8K cho ra bản xem trước 8K.
---
## Workflow với SAM 3
Trong mỗi workflow dưới đây, hãy kiểm tra những gì SAM3 Segment tìm được trước khi chỉnh sửa bất cứ thứ gì. Bản xem trước trên node cho thấy mọi instance với viền và nhãn, và báo cáo của nó cho biết cái gì không tìm thấy. Nếu thiếu một instance bạn muốn, hãy hạ `threshold` (ví dụ xuống 0.3) hoặc thử từ khác cho concept.
### Nền phía sau người
```text
Load Image → SAM3 Segment (concepts: background)
→ Mask Morphology (NH)
→ chỉnh sửa phần nền
```
`background` là mọi thứ nằm ngoài những người SAM 3 tìm thấy: node tự tìm `person`. Khi người đang cầm hoặc ngồi lên thứ gì, hãy nêu tên cả những thứ đó để chúng cũng nằm ngoài phần nền, ví dụ concepts `~(person + wine glass + chair)`. Dịch cạnh của mask vài pixel bằng [Mask Morphology (NH)](./mask.md#mask-morphology-nh): mở rộng để phần chỉnh sửa phủ lên đường viền cũ, hoặc thu hẹp nếu phần chỉnh sửa lấn vào người. Sau đó dùng mask này để làm mờ, đổi ánh sáng, đổi màu hoặc inpaint phần nền.
### Vật lớn nhất, rồi inpaint
```text
Load Image → SAM3 Segment (concepts: person, dog, car)
→ Segs Select (sort_by: area, count: 1)
→ Mask Morphology (NH, mở rộng)
→ inpaint
```
Khi `labels` để trống, Segs Select chọn cái lớn nhất trong mọi thứ bạn yêu cầu. Hãy đặt `labels` (ví dụ `car`) để chọn cái lớn nhất của một loại, `start` `1` để lấy cái lớn thứ hai, và tăng `min_score` để một kết quả đáng ngờ không thể bị chọn. Hãy mở rộng mask thêm vài pixel để vùng inpaint phủ lên đường viền cũ.
### Kiến trúc: mặt tiền không có cửa sổ
```text
Load Image → SAM3 Segment (concepts: building - (window + door))
→ đổi màu, đổi phong cách hoặc inpaint bức tường
```
`building` phủ cả mặt tiền cùng cửa sổ và cửa ra vào; công thức khoét chúng ra và chừa lại bức tường. Với một căn phòng, hãy dùng `(wall + ceiling) - window`. Nếu `building` không tìm thấy gì trên một tòa nhà có hình dạng lạ, hãy thử `structure` hoặc `monument` và sửa công thức cho khớp. Trên ảnh 4K đến 8K, hãy giữ `balanced`: mỗi cửa sổ được phân đoạn lại trên một ảnh cận cảnh của các điểm ảnh gốc. Đường viền của một mặt tiền chiếm trọn khung hình chỉ chính xác đến khoảng 1/300 bề ngang của nó (xem [Cách hai lượt phân tích hoạt động](#cách-hai-lượt-phân-tích-hoạt-động)), nên hãy kiểm tra đường mái trong bản xem trước.
---
## Khi có sự cố với SAM 3
| Bạn thấy | Cần làm |
|---|---|
| Lỗi báo SAM 3 cần gói sam3 của Meta | Cài nó như trong [Trước khi bắt đầu với SAM 3](#trước-khi-bắt-đầu-với-sam-3), rồi khởi động lại ComfyUI. |
| Lỗi báo gói sam3 đã cài nhưng không nạp được, hoặc thiếu một thành phần | Phiên bản đã cài không phải bản NH-Nodes hỗ trợ, hoặc thiếu một gói đi kèm. Hãy cài lại bằng hai dòng trong [Trước khi bắt đầu với SAM 3](#trước-khi-bắt-đầu-với-sam-3). Thông báo cho biết nguyên nhân. |
| Lỗi báo không tìm thấy checkpoint | Tải `sam3.pt` như trong [Trước khi bắt đầu với SAM 3](#trước-khi-bắt-đầu-với-sam-3), nhấn R trong ComfyUI và chọn tệp. Thông báo liệt kê các thư mục đã tìm. |
| Lỗi báo một tệp không phải checkpoint `.pt` của SAM 3 | Hãy dùng `sam3.pt`. Các tệp safetensors trên Hugging Face không dùng được với node này. |
| Lỗi báo không đọc được một tệp như checkpoint SAM 3, hoặc tệp đó không phải checkpoint SAM 3 | Tệp trong thư mục `sam3` là bản tải dở hoặc là `.pt` của model khác. Hãy tải lại `sam3.pt` như trong [Trước khi bắt đầu với SAM 3](#trước-khi-bắt-đầu-với-sam-3). |
| Lỗi báo SAM 3 cần GPU NVIDIA có CUDA | ComfyUI đang chạy trên CPU hoặc trên loại GPU khác. SAM3 Segment không có chế độ CPU. |
| Lỗi báo bạn phải viết ít nhất một concept | `concepts` đang trống, chỉ có dòng ghi chú, hoặc chỉ có các từ khóa `all`, `image` hay `none`. Hãy viết một từ như `person`. |
| Lỗi báo một concept có ký tự đại diện (wildcard) | Hãy bỏ `*` hoặc `?`: SAM 3 cần từ thông thường. Ký tự đại diện dùng được trong Segs Select và Segs Compose. |
| Lỗi từ SAM3 Segment có nêu một vị trí, như `unexpected '(' at position 8` | Kiểm tra công thức tại vị trí đó. Dấu ngoặc trong một nhãn được đọc là nhóm. Xem [Viết công thức](#viết-công-thức). |
| `count` bằng 0, hoặc thiếu một instance | Hãy đọc báo cáo: nó cho biết từ đó không tìm thấy gì, một ứng viên suýt đạt (hạ `threshold` xuống điểm của nó), hay từ đó không được hiểu (hãy viết bằng tiếng Anh). Bản xem trước trên node cho thấy những gì đã tìm được. |
| Báo cáo ghi một từ không được hiểu | Đó là một cụm tiếng Việt không có trong danh sách. Hãy viết bằng tiếng Anh. Sau một thứ, `xanh` không rõ nghĩa: hãy viết `xanh dương` (blue) hoặc `xanh lá` (green). |
| Mask là cả bức ảnh, hoặc không như bạn mong đợi | Danh sách là phép hợp: `person, background` là người cộng với mọi thứ nằm ngoài người. Hãy viết riêng `background`, hoặc một công thức như `person - bag`, và nhìn bản xem trước: phần sáng là mask. |
| SAM3 Segment vẫn chạy dù không có gì nối vào nó | Nó là node đầu ra (output node). Hãy vô hiệu hóa (Mute) hoặc xóa nó khi không cần; tắt `show_preview` không làm nó ngừng chạy. |
| Console báo đã giữ mask lượt đầu cho một số instance | Ảnh cận cảnh của chúng không khớp với lượt đầu, nên các instance đó giữ cạnh của lượt đầu. Không có gì sai. |
| Lỗi từ Segs Preview về kích thước hoặc số ảnh | Hãy nối đúng bức ảnh mà segs được tìm ra từ đó. |
| Lỗi từ Segs Compose có nêu một vị trí | Kiểm tra công thức tại vị trí đó. Xem [Viết công thức](#viết-công-thức). |
| Lỗi từ Segs Compose báo không instance nào có nhãn `person hair`, nhưng `person` và `hair` thì có | Các từ liền nhau là một nhãn. Hãy nối chúng lại: `person + hair`. |
| Mask của Segs Compose rỗng, và console báo không instance nào có một nhãn | Kiểm tra chính tả của nhãn. Đặt dấu cách quanh dấu trừ: `person - hair`, không phải `person-hair`. |
| Lỗi từ Segs To Masks báo các mask instance kích thước đầy đủ cần nhiều bộ nhớ hơn mức còn trống | Có quá nhiều instance so với kích thước bức ảnh. Dùng Segs Select để chỉ giữ những instance bạn cần, hoặc Segs Compose để lấy một mask duy nhất. |
| Console báo SAM 3 nhìn bức ảnh bị nén vào hình vuông và có thể bỏ sót vật nhỏ | Bức ảnh rộng hoặc cao hơn 4:1. Hãy chia nó thành các ô bằng Image Tile (NH) rồi chạy từng ô. |
| ComfyUI hết VRAM sau khi node chạy | Để `keep_model_loaded` tắt, để trọng số rời GPU sau mỗi lần chạy. |
| Nút Unload Models của ComfyUI không giải phóng VRAM của SAM 3 | `keep_model_loaded` đang bật. Hãy tắt nó và chạy node một lần, hoặc khởi động lại ComfyUI. |
---
[← Về trang hướng dẫn](./README.md)
+165
View File
@@ -0,0 +1,165 @@
"""SAM3 Segment (NH): find what you name with SAM 3 and get its mask, straight from a short formula.
The model work lives in core/sam3_backend.py and the formula (labels, + - & ~, background, Vietnamese words) in
core/segs.py; this module maps the node's inputs and outputs and draws its preview.
"""
from ...core import sam3_backend as sam3
from ...core.segs import (
SEGS_TYPE,
EmptyExpressionError,
ExpressionError,
count_segments,
describe_request,
expression_labels,
expression_masks,
normalize_label,
parse_concepts,
preview_picture,
uses_keyword,
)
from ...core.ui_preview import MAX_PREVIEWS, save_temp_previews
try:
from comfy.model_management import throw_exception_if_processing_interrupted
except Exception: # outside ComfyUI (tests)
def throw_exception_if_processing_interrupted():
return None
try:
from comfy.utils import ProgressBar
except Exception: # outside ComfyUI (tests)
ProgressBar = None
def concepts_to_find(tree):
"""(concepts SAM 3 must look for, the ones the node added) for a parsed request, without repeats.
``background`` needs to know where the subject is, so "person" is added when the request does not name it.
"""
names, seen = [], set()
for label in expression_labels(tree):
if any(ch in label for ch in "*?["):
raise ValueError(f"[NH-Nodes] SAM3 Segment: '{label}' has a wildcard; SAM 3 needs plain words (wildcards "
"work in Segs Select and Segs Compose).")
if normalize_label(label) not in seen:
seen.add(normalize_label(label))
names.append(label)
added = []
if uses_keyword(tree, "background") and "person" not in seen:
names.append("person")
added.append("person")
return names, added
class NH_SAM3Segment:
"""Find every instance of the concepts you name with SAM 3 (Meta), at the full resolution of the image."""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"image": ("IMAGE",),
"concepts": ("STRING", {
"default": "person", "multiline": True,
"tooltip": "What to find, in short English words (person, red car, window, sky); common "
"Vietnamese words work too (tóc, váy, túi, nền). Commas or new lines list several. "
"The mask follows the formula: person - bag (minus), hair + face (plus), ~person "
"(everything else), background (everything outside what was found; people are "
"searched for it), shirt & jacket (overlap), parentheses.",
}),
"threshold": ("FLOAT", {
"default": 0.5, "min": 0.05, "max": 0.95, "step": 0.01,
"tooltip": "Minimum confidence. Lower finds more (and more mistakes), higher keeps only sure ones. "
"The report says how close a concept that was not found came.",
}),
"detail_level": (list(sam3.DETAIL_LEVELS), {
"default": "balanced",
"tooltip": "balanced: each instance is segmented again on a close-up cut from the full-resolution "
"image, so small and medium objects get crisp edges even on 4K-8K pictures (about "
"0.1 s per instance on a fast GPU). A subject filling most of the frame keeps edges "
"at about 1/300 of its size. fast: one look at the whole image only.",
}),
"max_per_concept": ("INT", {
"default": 0, "min": 0, "max": 200, "step": 1,
"tooltip": "Keep at most this many instances of each concept, best first. 0 keeps all.",
}),
"checkpoint": (sam3.checkpoint_choices(), {
"tooltip": "SAM 3 checkpoint in models/sam3 (shared with other SAM 3 packs). Download sam3.pt from "
"huggingface.co/facebook/sam3 after accepting Meta's license.",
}),
"keep_model_loaded": ("BOOLEAN", {
"default": False,
"tooltip": "Keep SAM 3 on the GPU after the run (about 3.6 GB): faster when it runs again, but "
"ComfyUI cannot use that VRAM meanwhile, and its Unload Models button does not "
"release it (turn this off and run the node once, or restart ComfyUI).",
}),
},
# optional, so prompts saved before it existed still validate
"optional": {
"show_preview": ("BOOLEAN", {
"default": True,
"tooltip": "Show the mask (bright), the instances found (outlined and labelled) and the report on "
"the node itself. The node runs even with nothing connected.",
}),
},
}
RETURN_TYPES = (SEGS_TYPE, "MASK", "INT", "STRING")
RETURN_NAMES = ("segs", "mask", "count", "report")
OUTPUT_NODE = True # it shows its own preview, so it runs even with nothing connected
FUNCTION = "segment"
CATEGORY = "NH-Nodes/Vision"
DESCRIPTION = (
"Type what you want and get its mask: 'hair', 'person - bag', 'background', 'tóc, váy'. SAM 3 (Meta) finds "
"every instance at full resolution; the node shows them on itself with a report of what was and was not "
"found. segs feed Segs Select / Compose / To Masks for finer picking. Needs the sam3 package, "
"models/sam3/sam3.pt and a CUDA GPU."
)
SEARCH_ALIASES = ["sam3", "sam 3", "segment anything", "text segment", "concept segment", "object mask",
"detect objects", "instance segmentation", "background mask", "mask formula"]
def segment(self, image, concepts, threshold, detail_level, max_per_concept, checkpoint, keep_model_loaded,
show_preview=True):
try:
tree = parse_concepts(concepts)
except EmptyExpressionError:
raise ValueError("[NH-Nodes] SAM3 Segment: write at least one concept, e.g. 'person' or 'window'.") \
from None
except ExpressionError as exc:
raise ValueError(f"[NH-Nodes] SAM3 Segment: {exc}") from None
names, added = concepts_to_find(tree)
if not names:
raise ValueError("[NH-Nodes] SAM3 Segment: write at least one concept, e.g. 'person' or 'window'.")
bar = ProgressBar(image.shape[0] * len(names)) if ProgressBar is not None else None
misses = {}
segs = sam3.segment(
image, names, threshold, detail_level, max_per_concept, checkpoint, keep_model_loaded,
interrupt=throw_exception_if_processing_interrupted,
progress=(lambda done, total: bar.update_absolute(done, total)) if bar is not None else None,
misses=misses,
)
count = count_segments(segs)
lines = describe_request(tree, segs, threshold, misses, added)
report = "\n".join([f"SAM3 Segment: {count} instance(s) in {len(segs)} image(s)", *lines])
mask = expression_masks(tree, segs)
result = (segs, mask, count, report)
if not show_preview:
return result
try:
pictures = [preview_picture(picture, frame, lines, mask=held) for picture, frame, held in
zip(image[:MAX_PREVIEWS], segs[:MAX_PREVIEWS], mask[:MAX_PREVIEWS])]
images = save_temp_previews(pictures, "NH_sam3")
except Exception as exc: # noqa: BLE001 - a preview that cannot be drawn or saved must not lose the run
print(f"[NH-Nodes] SAM3 Segment: could not show the preview ({exc}); the outputs are fine")
return result
return {"ui": {"images": images}, "result": result}
NODE_CLASS_MAPPINGS = {
"NH_SAM3Segment": NH_SAM3Segment,
}
NODE_DISPLAY_NAME_MAPPINGS = {
"NH_SAM3Segment": "SAM3 Segment (NH)",
}
+227
View File
@@ -0,0 +1,227 @@
"""NH_SEGS nodes: pick instances, combine them into masks, preview them and hand them to Impact Pack.
All of them work on the instances SAM3 Segment (NH) finds; the logic lives in core/segs.py.
"""
from ...core.segs import (
SEGS_TYPE,
SORT_ORDERS,
count_segments,
expression_labels,
expression_masks,
instance_masks,
label_matches,
parse_expression,
parse_label_list,
preview_batch,
segs_union,
select_segments,
to_impact_segs,
)
from ...core.vi_words import translate
CATEGORY = "NH-Nodes/Vision"
class NH_SegsSelect:
"""Keep the instances that match labels, score and size, in a chosen order."""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"segs": (SEGS_TYPE,),
"labels": ("STRING", {
"default": "",
"tooltip": "Labels to keep, separated by commas; * and ? are wildcards (\"*shirt\"). Common "
"Vietnamese words work too (tóc, túi). Empty keeps every label.",
}),
"min_score": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.01,
"tooltip": "Drop instances with a lower confidence."}),
"min_area": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.0001,
"tooltip": "Drop instances smaller than this fraction of the image (0.01 = 1%)."}),
"max_area": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.0001,
"tooltip": "Drop instances larger than this fraction of the image."}),
"sort_by": (list(SORT_ORDERS), {
"default": "score",
"tooltip": "Order of the kept instances: highest score, largest area, by position (box centre), "
"nearest the image centre first, or as found.",
}),
"start": ("INT", {"default": 0, "min": 0, "max": 4096, "step": 1,
"tooltip": "Skip this many instances of the sorted list (0 starts at the first)."}),
"count": ("INT", {"default": 0, "min": 0, "max": 4096, "step": 1,
"tooltip": "Keep this many instances from start on; 0 keeps all of them."}),
}
}
RETURN_TYPES = (SEGS_TYPE, "MASK", "INT")
RETURN_NAMES = ("segs", "mask", "count")
FUNCTION = "select"
CATEGORY = CATEGORY
DESCRIPTION = (
"Filter and order instances, per image: by label (wildcards allowed), score and size, then sort and take a "
"slice. Example: the second person from the left = labels 'person', sort_by left_to_right, start 1, count 1."
)
SEARCH_ALIASES = ["segs select", "filter segs", "pick instance", "largest object", "leftmost person"]
def select(self, segs, labels, min_score, min_area, max_area, sort_by, start, count):
patterns = []
for pattern in parse_label_list(labels):
english, status = translate(pattern)
# "nền" / "tất cả" are formula words, not labels: Select matches labels only
patterns.append(english if status == "translated" and english not in ("background", "all", "image")
else pattern)
picked = tuple(
select_segments(frame, patterns, min_score, min_area, max_area, sort_by, start, count) for frame in segs
)
return picked, segs_union(picked), count_segments(picked)
class NH_SegsCompose:
"""Build a mask from instance labels with + - & ~ and parentheses."""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"segs": (SEGS_TYPE,),
"expression": ("STRING", {
"default": "all", "multiline": True,
"tooltip": "Labels combined with | or + (union), - (minus), & (overlap), ~ (everything except) "
"and parentheses; commas or new lines list several. all = every instance, background = "
"everything outside them, image = whole frame, none = nothing. Words in a row are one "
"label (red car); t-shirt needs no quotes; Vietnamese works (tóc, nền). Put spaces "
"around a minus to subtract. Examples: person - hair · background · "
"(wall | ceiling) - window.",
}),
}
}
RETURN_TYPES = ("MASK",)
RETURN_NAMES = ("mask",)
FUNCTION = "compose"
CATEGORY = CATEGORY
DESCRIPTION = (
"Turn instances into one mask with a small formula over their labels: | or + (or a comma) joins, - removes, "
"& keeps the overlap, ~ inverts, background is everything outside them. (building | roof) - window keeps the "
"facade. SAM3 Segment takes the same formula in its concepts, so this node is for mixing or re-using segs."
)
SEARCH_ALIASES = ["segs compose", "mask formula", "combine masks", "subtract mask", "background mask"]
def compose(self, segs, expression):
tree = parse_expression(expression)
labels = [seg.label for frame in segs for seg in frame.segments]
def found(pattern):
return any(label_matches(label, pattern) for label in labels)
unmatched = [pattern for pattern in expression_labels(tree) if not found(pattern)]
for pattern in unmatched:
words = pattern.split()
if len(words) > 1 and all(found(word) for word in words):
# words in a row make one label; with each of them found, an operator was left out
raise ValueError(f"[NH-Nodes] Segs Compose: no instance is labelled '{pattern}', but "
f"{' and '.join(repr(word) for word in words)} are. Join them with + or a comma: "
f"{' + '.join(words)}")
parts = pattern.split("-")
if len(parts) > 1 and all(part and found(part) for part in parts):
# "person-hair" is one label; with person and hair both found, a subtraction was meant
raise ValueError(f"[NH-Nodes] Segs Compose: no instance is labelled '{pattern}', but "
f"{' and '.join(repr(part) for part in parts)} are. To subtract, put spaces around "
f"the minus: {' - '.join(parts)}")
if unmatched:
print(f"[NH-Nodes] Segs Compose: no instance is labelled {', '.join(repr(p) for p in unmatched)}; "
"those parts are empty")
return (expression_masks(tree, segs),)
class NH_SegsToMasks:
"""One full-size mask per instance, plus the union and the labels."""
@classmethod
def INPUT_TYPES(cls):
return {"required": {"segs": (SEGS_TYPE,)}}
RETURN_TYPES = ("MASK", "MASK", "STRING", "INT")
RETURN_NAMES = ("masks", "union", "labels", "count")
FUNCTION = "convert"
CATEGORY = CATEGORY
DESCRIPTION = (
"masks: one mask per instance, image after image (a single empty mask when there is none). union: one mask "
"per image. labels: the instance labels, one per line, in the order of masks."
)
SEARCH_ALIASES = ["segs to masks", "instance masks", "segs to mask batch"]
def convert(self, segs):
masks, labels = instance_masks(segs)
return masks, segs_union(segs), "\n".join(labels), len(labels)
class NH_SegsToImpactSEGS:
"""Hand the instances of one image to Impact Pack detailers as its SEGS type."""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"segs": (SEGS_TYPE,),
"crop_factor": ("FLOAT", {
"default": 3.0, "min": 1.0, "max": 10.0, "step": 0.1,
"tooltip": "Size of the region a detailer crops around each instance, as a multiple of its box "
"(Impact Pack's crop_factor).",
}),
"batch_index": ("INT", {"default": 0, "min": 0, "max": 4095, "step": 1,
"tooltip": "Impact SEGS hold one image: which image of the batch to convert."}),
}
}
RETURN_TYPES = ("SEGS",)
RETURN_NAMES = ("segs",)
FUNCTION = "convert"
CATEGORY = CATEGORY
DESCRIPTION = (
"Convert the instances of one image to Impact Pack's SEGS, for its detailers and SEGS tools. Impact Pack is "
"not needed for anything else in NH-Nodes."
)
SEARCH_ALIASES = ["segs to impact", "impact segs", "detailer segs"]
def convert(self, segs, crop_factor, batch_index):
if batch_index >= len(segs):
raise ValueError(f"[NH-Nodes] Segs To Impact SEGS: batch_index {batch_index} but the segs hold "
f"{len(segs)} image(s)")
return (to_impact_segs(segs[batch_index], crop_factor),)
class NH_SegsPreview:
"""Draw the instances on their image."""
@classmethod
def INPUT_TYPES(cls):
return {"required": {"image": ("IMAGE",), "segs": (SEGS_TYPE,)}}
RETURN_TYPES = ("IMAGE",)
RETURN_NAMES = ("preview",)
FUNCTION = "preview"
CATEGORY = CATEGORY
DESCRIPTION = "Each instance tinted in its own colour, outlined and labelled with its label and score."
SEARCH_ALIASES = ["segs preview", "show segs", "draw instances"]
def preview(self, image, segs):
return (preview_batch(image, segs),)
NODE_CLASS_MAPPINGS = {
"NH_SegsSelect": NH_SegsSelect,
"NH_SegsCompose": NH_SegsCompose,
"NH_SegsToMasks": NH_SegsToMasks,
"NH_SegsToImpactSEGS": NH_SegsToImpactSEGS,
"NH_SegsPreview": NH_SegsPreview,
}
NODE_DISPLAY_NAME_MAPPINGS = {
"NH_SegsSelect": "Segs Select (NH)",
"NH_SegsCompose": "Segs Compose (NH)",
"NH_SegsToMasks": "Segs To Masks (NH)",
"NH_SegsToImpactSEGS": "Segs To Impact SEGS (NH)",
"NH_SegsPreview": "Segs Preview (NH)",
}
+186 -8
View File
@@ -1,14 +1,27 @@
"""Vision segmentation nodes: garment parts and portrait regions.
Both nodes run a SegFormer label map (see segformer_backend.py), union the
class ids for the requested region, refine the mask, and return
(mask, cutout_rgba, mask_image) at the input resolution.
Both nodes build a SegFormer label map of the whole picture (see segformer_backend.py), union the class ids for
the requested region, refine the mask, and return (mask, cutout_rgba, mask_image) at the input resolution. The
parsers see 512 x 512 pictures, so they do not get the whole frame squeezed: Portrait Segment parses a crop around
each face (found with YuNet, see face_detect.py), Garment Segment a crop of the model's shape around the clothes
(see parse_crops.py).
"""
import cv2
import numpy as np
import torch
from ...core.face_detect import detect_faces
from ...core.parse_crops import (
inner_mask,
nearest_owner,
parse_region,
people_labels,
person_box,
square_around,
square_frame,
visible,
)
from ...core.segformer_backend import load_segformer, offload, predict_labels
try:
@@ -126,15 +139,180 @@ def _image_batch_to_numpy(image):
return [np.clip(frame[:, :, :3], 0.0, 1.0) for frame in array]
def _run_batch(image, model_name, device, build_mask, expand, blur, holes):
"""Shared per-image loop: predict labels, build mask via callback, refine, pack, stack."""
# ---------------------------------------------------------------------------
# Label maps of the whole picture, parsed on crops
# ---------------------------------------------------------------------------
# Square crops around each face, in face-box sides: the wide one keeps long hair and the neck, the tight one (moved
# up a little, for the forehead and hair line) is framed like the faces the parser learnt on, for eyes, brows and lips.
FACE_WIDE = 3.6
FACE_TIGHT = 2.2
FACE_LIFT = 0.06
GARMENT_MARGIN = 0.08 # around the clothes found on the first look, as a fraction of their longer side
FACE_CONTEXT = 1 / 3 # grey added on each side for a second look when no face is found (a face cut by the frame)
BIG_FACE = 0.9 # a face whose tight crop would span this share of the picture's long side is parsed whole-frame
FACE_CORE = 1.3 # the tight crop decides only inside an ellipse of the face box grown this much, and only about
# the face itself (see FACE_DETAIL): hair, neck and background come from the wide crop, which sees them whole instead
# of cut by the tight crop's edges.
# CelebAMask-HQ classes of the face parser: what a face is made of, and the features that make it a face
FACE_PARTS = frozenset({1, 2, 3, 4, 5, 6, 7, 10, 11, 12}) # skin, nose, glasses, eyes, brows, mouth, lips
FACE_FEATURES = frozenset({2, 4, 5, 6, 7, 10, 11, 12}) # nose, eyes, brows, mouth, lips
FACE_DETAIL = FACE_PARTS | {8, 9} # with the ears: what the tight crop may decide
_detection_failed = False
def find_faces(image, second_look=True):
"""Face boxes of the uint8 RGB ``image``. A face cut by the frame is often found only once the picture gets a
grey margin around it, so that is tried when the first look finds none; boxes mostly outside the frame are left
out. If the detector cannot be loaded (offline first run, an old OpenCV, a damaged file) the node says so once
and parses whole frames, as before the detector existed."""
global _detection_failed
if _detection_failed:
return []
try:
faces = detect_faces(image)
if faces or not second_look:
return faces
height, width = image.shape[:2]
top, left = int(height * FACE_CONTEXT), int(width * FACE_CONTEXT)
padded = cv2.copyMakeBorder(np.ascontiguousarray(image), top, top, left, left, cv2.BORDER_CONSTANT,
value=(114, 114, 114))
found = []
for x1, y1, x2, y2, score in detect_faces(padded):
box = (max(0.0, x1 - left), max(0.0, y1 - top), min(float(width), x2 - left), min(float(height), y2 - top))
inside = max(0.0, box[2] - box[0]) * max(0.0, box[3] - box[1])
if inside >= 0.5 * (x2 - x1) * (y2 - y1):
found.append((*box, score))
return found
except Exception as exc: # noqa: BLE001 - any failure of the optional detector falls back to whole frames
_detection_failed = True
print(f"[NH-Nodes] Portrait Segment: the face detector is not available ({exc}); whole frames are parsed "
"instead. Restart ComfyUI to try again.")
return []
def is_face(labels, seen, face):
"""True when ``labels`` (the parse of the part ``seen`` of a tight crop) show a face inside the detected box:
mostly face parts, with some nose, eye, brow or lip. A hand or a pattern the detector took for a face is turned
down, so its crops change nothing."""
sx1, sy1 = seen[:2]
fx1, fy1, fx2, fy2 = (int(round(value)) for value in face[:4])
inside = labels[max(0, fy1 - sy1):max(0, fy2 - sy1), max(0, fx1 - sx1):max(0, fx2 - sx1)]
if inside.size == 0:
return False
return (np.isin(inside, list(FACE_PARTS)).mean() >= 0.3
and np.isin(inside, list(FACE_FEATURES)).mean() >= 0.005)
def portrait_labels(bundle, image):
"""Face-parsing label map of the uint8 RGB ``image``.
The whole frame is parsed first, as a padded square (the parser learnt on square face pictures), and that answer
is kept wherever the crops below do not reach. Each face found, unless it already fills the frame, is parsed again
on a tight square crop (eyes, brows, lips) and, once that parse confirms it is a face, on a wide one (hair, neck).
The wide crop is pasted inside its region away from its cut edges, the tight one only around the face itself, and
each only where its face is the nearest one, so neighbours in a group photo do not overwrite each other and no
straight crop edge shows in long hair.
"""
def predict(picture):
throw_exception_if_processing_interrupted()
return predict_labels(bundle, picture)
height, width = image.shape[:2]
canvas = parse_region(predict, image, square_frame((height, width)))[0]
long_side = max(height, width)
candidates = [face for face in find_faces(image)
if FACE_TIGHT * max(face[2] - face[0], face[3] - face[1]) < BIG_FACE * long_side]
tight = []
for face in candidates:
region = square_around(face, FACE_TIGHT, FACE_LIFT)
labels, seen = parse_region(predict, image, region)
if labels.size and is_face(labels, seen, face):
tight.append((face, region, labels, seen))
if not tight:
return canvas
faces = [face for face, _, _, _ in tight]
wide_regions = [square_around(face, FACE_WIDE) for face in faces]
# with the faces known, face parts the whole-frame parse saw far from all of them are mistakes (a reflection,
# a pattern): only hair, neck and clothes are kept there
near_faces = np.zeros((height, width), dtype=bool)
for x1, y1, x2, y2 in (visible(region, (height, width)) for region in wide_regions):
near_faces[y1:y2, x1:x2] = True
canvas[~near_faces & np.isin(canvas, list(FACE_DETAIL))] = 0
tight_regions = [region for _, region, _, _ in tight]
for index, region in enumerate(wide_regions):
labels, seen = parse_region(predict, image, region)
_paste(canvas, labels, seen,
nearest_owner(seen, faces, wide_regions, index) & inner_mask(region, (height, width)))
detail = list(FACE_DETAIL)
for index, (face, region, labels, seen) in enumerate(tight):
x1, y1, x2, y2 = seen
about_face = np.isin(labels, detail) | np.isin(canvas[y1:y2, x1:x2], detail)
_paste(canvas, labels, seen, nearest_owner(seen, faces, tight_regions, index) & _core(face, seen) & about_face)
return canvas
def _core(face, seen):
"""Bool mask over ``seen`` (a region inside the picture): the ellipse of the face box grown FACE_CORE times."""
x1, y1, x2, y2 = face[:4]
cx, cy = (x1 + x2) / 2.0, (y1 + y2) / 2.0
half_w, half_h = max(1.0, FACE_CORE * (x2 - x1) / 2.0), max(1.0, FACE_CORE * (y2 - y1) / 2.0)
sx1, sy1, sx2, sy2 = seen
ys = (np.arange(sy1, sy2, dtype=np.float32)[:, None] + 0.5 - cy) / half_h
xs = (np.arange(sx1, sx2, dtype=np.float32)[None, :] + 0.5 - cx) / half_w
return xs * xs + ys * ys <= 1.0
def _paste(canvas, labels, seen, keep):
if labels.size:
x1, y1, x2, y2 = seen
canvas[y1:y2, x1:x2][keep] = labels[keep]
def garment_labels(bundle, image):
"""Fashion-parsing label map of the uint8 RGB ``image``: a first look at the whole frame finds the clothes, then
crops of the model's shape around the people (found from the clothes and from their faces; a row of crops for a
group; not squeezed, padded where they leave the picture) are parsed again and merged by confidence (see
parse_crops.people_labels)."""
size = getattr(bundle.get("processor"), "size", None) or {"height": 1, "width": 1}
def predict(picture):
throw_exception_if_processing_interrupted()
return predict_labels(bundle, picture, confidence=True)
people = [person_box(face, image.shape[:2]) for face in find_faces(image, second_look=False)]
return people_labels(predict, image, size["width"] / size["height"], GARMENT_MARGIN, people)
WORK_SIDE = 4096 # label maps are worked out on a copy at most this long, then enlarged to the picture
def labels_at_work_size(labeler, bundle, frame):
"""Label map of the float HxWx3 ``frame`` at its size, worked out by ``labeler(bundle, uint8 RGB)`` on a copy at
most WORK_SIDE pixels long. The parsers look at 512 to 1024 px at a time, so their labels hold no finer detail
than that on an 8K photo, while merging crops at full size would take seconds."""
height, width = frame.shape[:2]
pixels = cv2.convertScaleAbs(frame, alpha=255.0) # rounds and saturates in one pass
scale = min(1.0, WORK_SIDE / max(height, width))
if scale >= 1.0:
return labeler(bundle, pixels)
small = cv2.resize(pixels, (max(1, round(width * scale)), max(1, round(height * scale))),
interpolation=cv2.INTER_AREA)
labels = labeler(bundle, small)
return cv2.resize(labels.astype(np.int32), (width, height), interpolation=cv2.INTER_NEAREST).astype(np.int64)
def _run_batch(image, model_name, device, build_mask, expand, blur, holes, labeler):
"""Shared per-image loop: label map via ``labeler(bundle, uint8 RGB)``, mask via ``build_mask``, refine, pack."""
bundle = load_segformer(model_name, device)
masks, cutouts, mask_images = [], [], []
frames = _image_batch_to_numpy(image)
try:
for frame in frames:
throw_exception_if_processing_interrupted()
labels = predict_labels(bundle, (frame * 255.0).round().astype(np.uint8))
labels = labels_at_work_size(labeler, bundle, frame)
mask = refine_mask(build_mask(labels), expand, blur, holes)
mask_t, rgba_t, mask_img_t = pack_outputs(frame, mask)
masks.append(mask_t)
@@ -195,7 +373,7 @@ class NH_GarmentSegment:
return _run_batch(
image, "segformer_fashion", device,
lambda labels: self.build_mask(labels, part, attach_parts),
mask_expand, mask_blur, fill_holes,
mask_expand, mask_blur, fill_holes, garment_labels,
)
@@ -228,7 +406,7 @@ class NH_PortraitSegment:
return _run_batch(
image, "face_parsing", device,
lambda labels: self.build_mask(labels, selected),
mask_expand, mask_blur, fill_holes,
mask_expand, mask_blur, fill_holes, portrait_labels,
)
+2 -2
View File
@@ -1,7 +1,7 @@
[project]
name = "nh-nodes"
description = "ComfyUI custom nodes for image workflows, tiling, masks, smart resize, logic routing, prompt and text tools, batch automation, garment/portrait segmentation, and VTON preprocessing."
version = "1.6.2"
description = "ComfyUI custom nodes for image workflows, tiling, masks, smart resize, logic routing, prompt and text tools, batch automation, SAM 3 object segmentation from mask formulas (Vietnamese words too), instance selection, close-up garment/portrait parsing, and VTON preprocessing."
version = "1.8.0"
license = { file = "LICENSE" }
dependencies = [
"numpy",
+224
View File
@@ -0,0 +1,224 @@
"""Crops for the parsers (faces, clothes) and YuNet face boxes, without models."""
import importlib
import numpy as np
import pytest
crops = importlib.import_module("NH-Nodes.core.parse_crops")
faces = importlib.import_module("NH-Nodes.core.face_detect")
def test_square_around_scales_the_longer_side_and_lifts():
assert crops.square_around((40, 50, 60, 80), 2.0) == (20, 35, 80, 95) # 30 px tall -> 60 px square
assert crops.square_around((40, 50, 60, 80), 2.0, lift=0.1) == (20, 29, 80, 89) # up by 0.1 * 60
assert crops.square_around((0, 0, 0, 0), 3.0) == (0, 0, 1, 1) # never empty
@pytest.mark.parametrize("box, aspect", [((100, 100, 140, 300), 1.0), ((10, 10, 50, 30), 2 / 3),
((500, 20, 590, 390), 0.75), ((0, 0, 600, 400), 1.0)])
def test_fit_box_has_the_model_shape_and_holds_the_box(box, aspect):
region = crops.fit_box(box, aspect, 0.08, (400, 600))
x1, y1, x2, y2 = region
assert x1 <= box[0] and y1 <= box[1] and x2 >= box[2] and y2 >= box[3]
assert abs((x2 - x1) / (y2 - y1) - aspect) < 0.02
if (x2 - x1) <= 600:
assert 0 <= x1 and x2 <= 600 # slid inside where it fits
if (y2 - y1) <= 400:
assert 0 <= y1 and y2 <= 400
def test_cut_padded_pads_the_outside_with_grey():
image = np.full((10, 20, 3), 200, np.uint8)
crop = crops.cut_padded(image, (-5, 5, 10, 15))
assert crop.shape == (10, 15, 3)
assert (crop[:5, 5:] == 200).all() and (crop[:, :5] == crops.PAD_VALUE).all() and (crop[5:] == crops.PAD_VALUE).all()
assert crops.visible((-5, 5, 10, 15), (10, 20)) == (0, 5, 10, 10)
def test_nearest_owner_splits_overlapping_crops_between_their_faces():
boxes = [(0, 0, 10, 10), (20, 0, 30, 10)] # face centres at x = 5 and x = 25
regions = [(0, 0, 30, 10), (0, 0, 30, 10)]
left = crops.nearest_owner((0, 0, 30, 10), boxes, regions, 0)
right = crops.nearest_owner((0, 0, 30, 10), boxes, regions, 1)
assert left[:, :15].all() and not left[:, 15:].any() and (left ^ right).all()
# a face whose crop does not reach a pixel cannot take it
alone = crops.nearest_owner((0, 0, 30, 10), boxes, [(0, 0, 30, 10), (20, 0, 30, 10)], 0)
assert alone[:, :20].all() and not alone[:, 20:].any()
def test_cover_box_uses_a_row_of_crops_for_a_wide_group():
group = (100, 300, 7300, 4800) # five people side by side on a 7360 x 4912 photo
regions = crops.cover_box(group, 2 / 3, 0.0, (4912, 7360))
assert len(regions) >= 3
for x1, y1, x2, y2 in regions:
assert abs((x2 - x1) / (y2 - y1) - 2 / 3) < 0.02 and y1 <= 300 and y2 >= 4800
assert regions[0][0] <= 100 and regions[-1][2] >= 7300 # together they cover the group
assert all(a[2] > b[0] for a, b in zip(regions, regions[1:])) # and overlap
assert crops.cover_box((100, 100, 300, 500), 2 / 3, 0.08, (4912, 7360)) == [
crops.fit_box((100, 100, 300, 500), 2 / 3, 0.08, (4912, 7360))] # one person: one crop
assert len(crops.cover_box((100, 0, 200, 4900), 1.0, 0.0, (4912, 7360))) == 1 # tall: never split
def test_square_frame_pads_the_short_side():
assert crops.square_frame((100, 200)) == (0, -50, 200, 150)
assert crops.square_frame((300, 100)) == (-100, 0, 200, 300)
def _people_picture():
"""300 x 900 picture: a wide group (red 40) in rows 100-280, columns 50-850; a plain backdrop (red 200) elsewhere."""
image = np.full((300, 900, 3), 200, np.uint8)
image[100:280, 50:850, 0] = 40
return image
def test_people_labels_refines_the_first_look_and_keeps_it_elsewhere():
image = _people_picture()
shown = []
def predict(picture):
shown.append(picture.shape[:2])
person = picture[..., 0] == 40
if picture.shape[:2] == (300, 900): # the first look: right kind of thing, wrong label, unsure
return np.where(person, 3, 0).astype(np.int64), np.full(picture.shape[:2], 0.6, np.float16)
return np.where(person, 5, 0).astype(np.int64), np.full(picture.shape[:2], 0.9, np.float16)
labels = crops.people_labels(predict, image, 2 / 3)
assert shown[0] == (300, 900) and len(shown) >= 4 # a first look, then a row of crops over the group
group = labels[100:280, 50:850]
assert (group == 5).mean() > 0.97 and set(np.unique(group)) <= {3, 5} # refined; a few cut edges keep the look
assert not labels[:100].any() and not labels[:, :40].any() # the backdrop stays empty
def test_people_labels_invents_nothing_far_from_people():
image = np.full((300, 900, 3), 200, np.uint8)
image[100:280, 50:150, 0] = 40 # one person at the left
def predict(picture):
person = picture[..., 0] == 40
labels = np.where(person, 5, 0).astype(np.int64)
if picture.shape[:2] != (300, 900):
labels[picture[..., 0] == 200] = 7 # a crop "sees" clothes on the plain backdrop
return labels, np.full(picture.shape[:2], 0.9, np.float16)
labels = crops.people_labels(predict, image, 2 / 3)
assert (labels[100:280, 50:150] == 5).all()
columns = np.argwhere(labels == 7)[:, 1]
reach = crops.NEAR_PEOPLE * 900 + 2
assert ((columns < 150 + reach) & (columns >= 50 - reach)).all() # only right next to the person, if at all
def test_people_labels_adds_people_found_by_their_faces():
image = np.full((300, 900, 3), 200, np.uint8)
image[100:280, 50:150, 0] = 40
image[100:280, 700:800, 0] = 40 # the first look misses this one
def predict(picture):
person = picture[..., 0] == 40
if picture.shape[:2] == (300, 900):
person[:, 400:] = False
return np.where(person, 5, 0).astype(np.int64), np.full(picture.shape[:2], 0.9, np.float16)
assert not crops.people_labels(predict, image, 2 / 3)[:, 700:800].any()
found = crops.people_labels(predict, image, 2 / 3, people=[(680, 90, 820, 300)])
assert (found[100:280, 700:800] == 5).mean() > 0.95
def test_a_crop_grows_when_its_people_run_past_a_cut_edge():
image = np.full((400, 400, 3), 200, np.uint8)
image[150:250, 100:300, 0] = 40
shown = []
def predict(picture):
shown.append(picture.shape[:2])
person = picture[..., 0] == 40
if len(shown) == 1:
person[:, 160:] = False # the first look sees only the left part
return np.where(person, 5, 0).astype(np.int64), np.full(picture.shape[:2], 0.9, np.float16)
labels = crops.people_labels(predict, image, 1.0)
assert len(shown) >= 3 and shown[2][1] > shown[1][1] # parsed again, wider
assert (labels[150:250, 100:300] == 5).mean() > 0.9
def test_foreground_clusters_groups_people_standing_together():
labels = np.zeros((200, 1000), np.int64)
labels[50:150, 100:200] = 1
labels[50:150, 210:300] = 1 # 10 px away: same group
labels[50:150, 800:900] = 1 # far away: its own group
labels[5, 5] = 1 # a speck
assert crops.foreground_clusters(labels) == [(100, 50, 300, 150), (800, 50, 900, 150)]
assert crops.foreground_clusters(labels, extra=[(500, 40, 560, 160)]) == [
(100, 50, 300, 150), (500, 40, 560, 160), (800, 50, 900, 150)]
assert crops.foreground_clusters(np.zeros((5, 5), np.int64)) == []
def test_inner_mask_leaves_out_cut_edges_but_not_picture_edges():
keep = crops.inner_mask((-50, 20, 150, 120), (200, 300), band=0.1)
assert keep.shape == (100, 150)
assert keep[10:90, 0].all() # the region reaches past the picture's left edge: nothing left out there
assert not keep[:10].any() and not keep[90:].any() # 10% of 100 px at the cut top and bottom
assert keep[10:90, :130].all() and not keep[:, 130:].any() # and 10% of 200 px at the cut right edge
def test_person_box_reaches_down_from_the_face():
assert crops.person_box((100, 50, 140, 100, 0.9), (1000, 1000)) == (20, 20, 220, 450)
assert crops.person_box((0, 0, 40, 50), (300, 300)) == (0, 0, 120, 300)
# --- YuNet ---------------------------------------------------------------------
class _FakeYuNet:
def __init__(self, rows):
self.rows, self.size = rows, None
def setInputSize(self, size): # noqa: N802 (OpenCV's name)
self.size = size
def detect(self, image):
assert image.shape[:2] == (self.size[1], self.size[0])
return 1, (np.array(self.rows, np.float32) if self.rows else None)
def test_merge_faces_keeps_one_box_per_face():
whole = (100, 100, 200, 220, 0.95)
half = (100, 100, 150, 220, 0.7) # half of the same face, found at another scale
other = (300, 100, 380, 200, 0.8)
assert faces.merge_faces([half, whole, other]) == [whole, other]
assert faces.merge_faces([(0, 0, 10, 10, 0.9), (2, 2, 12, 12, 0.8)]) == [(0, 0, 10, 10, 0.9)]
def test_detect_faces_maps_boxes_back_sorts_and_filters(monkeypatch):
# rows: x, y, w, h, 10 landmark values, score (on the copy shrunk to 2048 px)
rows = [[10, 10, 20, 20] + [0] * 10 + [0.9], [100, 50, 60, 80] + [0] * 10 + [0.95],
[0, 0, 5, 5] + [0] * 10 + [0.3], [2040, 0, 30, 30] + [0] * 10 + [0.8]]
fake = _FakeYuNet(rows)
monkeypatch.setattr(faces, "_get_detector", lambda size: (fake.setInputSize(size), fake)[1])
found = faces.detect_faces(np.zeros((2048, 4096, 3), np.uint8), sides=(2048,))
assert fake.size == (2048, 1024) # looked at a copy half the size
assert [tuple(round(v) for v in f[:4]) for f in found] == [(200, 100, 320, 260), (20, 20, 60, 60),
(4080, 0, 4096, 60)] # largest first, clipped
assert all(f[4] >= faces.MIN_SCORE for f in found)
fake.rows = []
assert faces.detect_faces(np.zeros((100, 100, 3), np.uint8)) == []
seen = []
monkeypatch.setattr(faces, "_detect_at", lambda img, side, score: seen.append(side) or [])
faces.detect_faces(np.zeros((100, 100, 3), np.uint8))
assert seen == list(faces.DETECT_SIDES) # a sharp copy for small faces, a small one for big or grainy faces
def test_model_path_finds_existing_weights_before_downloading(monkeypatch, tmp_path):
monkeypatch.setattr(faces, "register_model_folder", lambda *a: [str(tmp_path / "a"), str(tmp_path / "b")])
monkeypatch.setattr(faces, "default_folder", lambda: str(tmp_path / "a"))
downloads = []
def fake_download(repo, revision, target, patterns):
downloads.append((repo, revision, tuple(patterns)))
(tmp_path / "a" / "yunet").mkdir(parents=True)
(tmp_path / "a" / "yunet" / faces.YUNET["file"]).write_bytes(b"x")
monkeypatch.setattr(faces, "download_pinned", fake_download)
assert faces.model_path(download=False) is None
path = faces.model_path()
assert path.endswith(faces.YUNET["file"]) and downloads[0][1] == faces.YUNET["revision"]
assert "LICENSE" in downloads[0][2]
assert faces.model_path() == path and len(downloads) == 1 # found on disk the second time
+5 -4
View File
@@ -13,9 +13,10 @@ for path in (COMFY_ROOT, os.path.dirname(PACK_DIR)):
pack = importlib.import_module("NH-Nodes")
EXPECTED_NODE_COUNT = 69
# Native libraries that only VTON needs; importing the pack must not load them.
LAZY_ONLY = ("onnxruntime", "skimage")
EXPECTED_NODE_COUNT = 75
# Heavy libraries only some nodes need (VTON: onnxruntime, skimage; SAM 3: sam3, whose import also switches TF32
# on process-wide); importing the pack must not load them.
LAZY_ONLY = ("onnxruntime", "skimage", "sam3")
def test_registry_is_complete_and_consistent():
@@ -24,7 +25,7 @@ def test_registry_is_complete_and_consistent():
assert pack.WEB_DIRECTORY == "./web"
def test_pack_import_does_not_load_vton_native_libraries():
def test_pack_import_does_not_load_heavy_optional_libraries():
# Fresh interpreter: other tests in this session may import these libraries legitimately.
code = (
"import importlib, sys\n"
+647
View File
@@ -0,0 +1,647 @@
"""SAM 3 backend: geometry helpers, the two-pass pipeline with a fake sam3, package/checkpoint checks.
Everything but the last test runs without sam3, a checkpoint or a GPU. The last one runs the real model and is
skipped unless all three are present.
"""
import contextlib
import importlib
import os
import re
import sys
import types
import numpy as np
import pytest
import torch
backend = importlib.import_module("NH-Nodes.core.sam3_backend")
sam3_nodes = importlib.import_module("NH-Nodes.nodes.vision.sam3_nodes")
segs = importlib.import_module("NH-Nodes.core.segs")
# --- geometry ----------------------------------------------------------------
def test_sample_region_maps_pixel_centres_onto_the_model_view():
size, height, width = 16, 30, 60
u = torch.arange(size, dtype=torch.float32).repeat(size, 1) # value = model column index
out = backend.sample_region(u, (height, width), (0, 0, width, height), (height, width))
expected = ((torch.arange(width) + 0.5) * size / width - 0.5).clamp(0, size - 1)
assert torch.allclose(out[height // 2], expected, atol=1e-4)
# a sub-region samples the same values as the full grid
sub = backend.sample_region(u, (height, width), (10, 5, 40, 20), (15, 30))
assert torch.allclose(sub, out[5:20, 10:40], atol=1e-4)
def test_sample_region_chunks_give_the_same_result(monkeypatch):
logits = torch.randn(32, 32)
whole = backend.sample_region(logits, (90, 70), (3, 4, 66, 88), (84, 63))
monkeypatch.setattr(backend, "_SAMPLE_CHUNK", 100)
assert torch.equal(backend.sample_region(logits, (90, 70), (3, 4, 66, 88), (84, 63)), whole)
def test_model_box_to_image_adds_a_margin_inside_the_image():
assert backend.model_box_to_image((16, 16, 48, 32), 64, (300, 600)) == (140, 70, 460, 155)
assert backend.model_box_to_image((0, 0, 64, 64), 64, (300, 600)) == (0, 0, 600, 300)
@pytest.mark.parametrize("box", [(100, 100, 120, 400), (0, 0, 30, 30), (1900, 950, 2000, 1000), (10, 500, 1990, 520),
(0, 0, 2000, 1000)])
def test_crop_region_contains_the_box_with_margin_and_bounded_aspect(box):
x1, y1, x2, y2 = box
cx1, cy1, cx2, cy2 = backend.crop_region(box, (1000, 2000))
assert 0 <= cx1 <= x1 and 0 <= cy1 <= y1 and x2 <= cx2 <= 2000 and y2 <= cy2 <= 1000
width, height = cx2 - cx1, cy2 - cy1
if width < 2000 and height < 1000: # not clamped by the image
assert max(width, height) / min(width, height) <= 2.0 + 1e-6
assert cx1 < x1 or cx1 == 0
def test_to_logits_inverts_probabilities_and_passes_logits_through():
logits = torch.tensor([-5.0, -0.5, 0.0, 2.0, 6.0])
assert torch.allclose(backend.to_logits(torch.sigmoid(logits)), logits, atol=1e-3)
assert torch.equal(backend.to_logits(logits), logits)
def test_drop_stray_parts():
coarse = np.zeros((20, 20), bool)
coarse[5:15, 5:15] = True
refined = np.zeros((20, 20), bool)
refined[4:14, 4:14] = True
refined[18:20, 0:2] = True # speck far from the object
cleaned = backend.drop_stray_parts(refined, coarse)
assert cleaned[4:14, 4:14].all() and not cleaned[18:20, 0:2].any()
def _blob(shape, box):
mask = np.zeros(shape, bool)
x1, y1, x2, y2 = box
mask[y1:y2, x1:x2] = True
return mask
def test_close_up_agrees_keeps_sharp_thin_objects_and_rejects_other_objects():
coarse_pole = _blob((400, 300), (130, 20, 170, 380)) # pass 1: a 40 px wide blur around a pole
sharp_pole = _blob((400, 300), (145, 25, 155, 375)) # close-up: the 10 px pole, IoU only 0.24
assert backend.close_up_agrees(sharp_pole, coarse_pole, tolerance=16)
coarse_person = _blob((400, 300), (60, 40, 240, 390))
assert backend.close_up_agrees(_blob((400, 300), (55, 35, 245, 395)), coarse_person, tolerance=16)
assert not backend.close_up_agrees(_blob((400, 300), (0, 0, 40, 40)), coarse_person, 16) # something else
assert not backend.close_up_agrees(_blob((400, 300), (100, 40, 140, 80)), coarse_person, 16) # only a part
assert not backend.close_up_agrees(_blob((400, 300), (0, 0, 300, 400)), coarse_person, 16) # far too much
assert not backend.close_up_agrees(np.zeros((400, 300), bool), coarse_person, 16)
def test_close_up_agrees_on_large_masks_at_a_reduced_scale(monkeypatch):
monkeypatch.setattr(backend, "_AGREE_PIXELS", 1000)
coarse = _blob((400, 300), (130, 20, 170, 380))
assert backend.close_up_agrees(_blob((400, 300), (148, 25, 151, 375)), coarse, tolerance=16) # 3 px pole
assert backend.close_up_agrees(_blob((400, 300), (149, 25, 150, 375)), coarse, tolerance=16) # thinner than a step
# --- fake sam3 ---------------------------------------------------------------
SQUARE = (16, 16, 48, 32) # model-space box (x1, y1, x2, y2) of the fake "square" instance
def _prob_map(box, size=64, inside=0.99):
prob = torch.full((size, size), 0.01)
x1, y1, x2, y2 = box
prob[y1:y2, x1:x2] = inside
return prob
class FakeProcessor:
resolution = 64
def __init__(self, threshold=0.5):
self.threshold = threshold
self.calls = []
def set_image(self, image, state=None):
self.calls.append(("image", tuple(image.shape)))
return {"hw": tuple(image.shape[-2:])}
def reset_all_prompts(self, state):
self.calls.append(("reset",))
state.pop("prompt", None)
def set_text_prompt(self, prompt, state):
self.calls.append(("text", prompt))
state["prompt"] = prompt
if prompt == "square":
# best instance, an exact duplicate with a lower score, and a second distinct one
maps = [_prob_map(SQUARE), _prob_map(SQUARE), _prob_map((50, 40, 60, 60))]
scores = [0.9, 0.6, 0.7]
elif prompt == "floor":
# runs off the bottom of the picture; like SAM 3, the outermost row fades to "not sure"
floor = _prob_map((0, 48, 64, 64))
floor[63, :] = 0.45
maps, scores = [floor], [0.8]
elif prompt == "faint":
maps, scores = [_prob_map((4, 4, 12, 12))], [0.3]
else:
maps, scores = [], []
masks = torch.stack(maps)[:, None] if maps else torch.zeros((0, 1, 64, 64))
return {"masks_logits": masks, "scores": torch.tensor(scores), "boxes": torch.zeros((len(scores), 4))}
class FakeModel(torch.nn.Module):
"""predict_inst answers with a mask filling the box prompt (or the opposite corner when ``stray``); with
``fade`` the outer 2 rows and columns of the close-up are "not sure", as SAM 3's are."""
def __init__(self, stray=False, fade=False):
super().__init__()
self.weight = torch.nn.Parameter(torch.zeros(1))
self.stray = stray
self.fade = fade
self.prompts = []
def predict_inst(self, state, box=None, mask_input=None, multimask_output=True, return_logits=False):
height, width = state["hw"]
self.prompts.append((tuple(np.round(box).astype(int)), tuple(mask_input.shape)))
logits = np.full((1, height, width), -10.0, np.float32)
if self.stray:
logits[0, :3, :3] = 10.0
else:
x1, y1, x2, y2 = np.round(box).astype(int)
logits[0, y1:y2, x1:x2] = 10.0
if self.fade:
logits[0, :2, :] = logits[0, -2:, :] = logits[0, :, :2] = logits[0, :, -2:] = -0.5
return logits, np.array([0.9]), np.zeros((1, 288, 288), np.float32)
def _run(detail, model=None, concepts=("square", "nothing"), **kwargs):
model = model or FakeModel()
processors = []
def make_processor(threshold):
processors.append(FakeProcessor(threshold))
return processors[-1]
images = torch.rand(2, 300, 600, 3)
frames = backend.segment_images(model, make_processor, images, list(concepts), 0.4, detail, device=torch.device("cpu"),
**kwargs)
return frames, processors, model
def test_extend_to_border_lifts_the_faded_band_and_never_lowers_it():
logits = torch.full((7, 7), -0.3) # the fade: unsure everywhere near the border
logits[:5, 2:5] = 4.0 # an object reaching down to row 4 (inside the band of 2)
logits[5, 0] = -6.0 # a sure "no" inside the band
logits[6, 5] = 2.0 # higher than the row inside: kept
out = backend.extend_to_border(logits, 2, 1, edges=("bottom",))
assert (out[5:, 2:5] == 4.0).all() # the object reaches the edge
assert float(out[5, 0]) == -6.0 and float(out[6, 5]) == 2.0 # sure values and higher ones stay
assert float(out[6, 1]) == pytest.approx(-0.3) # nothing inside to lift it to
assert torch.equal(logits[:5], out[:5]) # outside the band nothing changes
both = backend.extend_to_border(logits.numpy(), 2, 2)
assert isinstance(both, np.ndarray) and (both[5:, 2:5] == 4.0).all()
assert backend.extend_to_border(torch.zeros(2, 2), 9, 9).shape == (2, 2) # a band wider than the map is cut
assert backend.border_band(1008) == 3 and backend.border_band(64) == 1 and backend.border_band(7360) == 22
assert backend.border_band(2400, cells=2) == 5 # a close-up fades over about 2 rows of its view
def test_find_instances_drops_duplicates_and_honours_the_cap():
processor = FakeProcessor()
state = processor.set_image(torch.zeros(3, 64, 64))
found = backend.find_instances(processor, state, "square")
assert [round(score, 2) for score, _, _ in found] == [0.9, 0.7]
assert found[0][2] == SQUARE
assert len(backend.find_instances(processor, state, "square", max_instances=1)) == 1
assert backend.find_instances(processor, state, "nothing") == []
assert processor.calls.count(("reset",)) == 3 # every prompt starts from a clean state
def test_find_instances_keeps_the_best_score_under_the_threshold_as_a_near_miss():
processor = FakeProcessor()
state = processor.set_image(torch.zeros(3, 64, 64))
misses = {}
found = backend.find_instances(processor, state, "square", threshold=0.65, misses=misses)
assert [round(score, 2) for score, _, _ in found] == [0.9, 0.7] and misses == {"square": pytest.approx(0.6)}
# SAM 3 keeps scores above the threshold: one exactly on it is a miss, as before
on_it = backend.find_instances(processor, state, "square", threshold=0.7, misses=misses)
assert [round(score, 2) for score, _, _ in on_it] == [0.9]
assert backend.find_instances(processor, state, "faint", threshold=0.5, misses=misses) == []
assert misses["faint"] == pytest.approx(0.3)
def test_instances_running_off_the_picture_reach_its_edge():
for detail in ("fast", "balanced"):
frames, _, _ = _run(detail, model=FakeModel(fade=True), concepts=("floor",))
floor = frames[0].segments[0]
x1, y1, x2, y2 = floor.box
assert (x1, x2, y2) == (0, 600, 300), detail # the full width, down to the last row
assert bool(floor.mask[-1].all()) and bool(floor.mask[:, 0].all()), detail
def test_fast_pass_maps_instances_to_full_resolution():
frames, processors, model = _run("fast")
assert len(frames) == 2 and (frames[0].height, frames[0].width) == (300, 600)
first = frames[0].segments[0]
assert first.label == "square" and first.score == pytest.approx(0.9)
# model box (16..48, 16..32) of a 64 px square view -> (150..450, 75..150) in the 600 x 300 image
assert all(abs(a - b) <= 2 for a, b in zip(first.box, (150, 75, 450, 150)))
assert len(frames[0].segments) == 2 and model.prompts == []
# candidates are asked for down to NEAR_MISS so that the node can report how close the misses came
assert processors[0].threshold == backend.NEAR_MISS and processors[0].calls[0] == ("image", (3, 64, 64))
def test_balanced_pass_replaces_masks_with_the_close_up_result():
frames, processors, model = _run("balanced")
target = backend.model_box_to_image(SQUARE, 64, (300, 600))
first = frames[0].segments[0]
assert first.box == target # the fake close-up fills the box prompt exactly
assert len(model.prompts) == 4 # 2 instances x 2 images
assert all(shape == (1, 288, 288) for _, shape in model.prompts)
crops = [call[1] for call in processors[0].calls if call[0] == "image"][1:]
assert all(shape[0] == 3 and shape[1] < 300 for shape in crops) # close-ups are crops of the original
def test_balanced_pass_keeps_the_first_pass_mask_when_the_close_up_strays(capsys):
fast, _, _ = _run("fast")
kept, _, _ = _run("balanced", model=FakeModel(stray=True))
assert kept[0].segments[0].box == fast[0].segments[0].box
assert "kept the first-pass mask for 4 instance(s)" in capsys.readouterr().out
def test_interrupt_and_progress_hooks():
seen, steps = [], []
_run("balanced", interrupt=lambda: seen.append(1), progress=lambda done, total: steps.append((done, total)))
assert steps == [(1, 4), (2, 4), (3, 4), (4, 4)]
assert len(seen) == 4 + 4 # each concept of each image + each close-up
def stop():
raise RuntimeError("interrupted")
with pytest.raises(RuntimeError, match="interrupted"):
_run("fast", interrupt=stop)
def test_segment_offloads_unless_asked_to_keep(monkeypatch):
model = FakeModel()
moves = []
monkeypatch.setattr(backend, "load_model", lambda checkpoint: model)
monkeypatch.setattr(backend, "ensure_sam3", lambda: (None, lambda m, device, confidence_threshold: FakeProcessor()))
monkeypatch.setattr(backend, "sam3_device", lambda: torch.device("cpu"))
monkeypatch.setattr(backend, "move_model", lambda m, device: moves.append("to"))
monkeypatch.setattr(backend, "offload", lambda m: moves.append("off"))
frames = backend.segment(torch.rand(1, 64, 64, 3), ["square"], keep_model_loaded=False)
assert len(frames) == 1 and moves == ["to", "off"]
moves.clear()
backend.segment(torch.rand(1, 64, 64, 3), ["square"], keep_model_loaded=True)
assert moves == ["to"]
# --- package, device and checkpoint checks -----------------------------------
def _install_fake_sam3(monkeypatch, build=None, processor=None, fail=None, flip_tf32=False):
monkeypatch.setattr(backend, "_API", None)
monkeypatch.setattr(backend.importlib.util, "find_spec", lambda name, *a: object() if name == "sam3" else None)
def import_module(name):
if fail:
raise ImportError(fail)
if name == "sam3.model_builder":
if flip_tf32: # like sam3's _setup_tf32() at import
torch.backends.cuda.matmul.allow_tf32 = not torch.backends.cuda.matmul.allow_tf32
torch.backends.cudnn.allow_tf32 = not torch.backends.cudnn.allow_tf32
return types.SimpleNamespace(build_sam3_image_model=build)
return types.SimpleNamespace(Sam3Processor=processor)
monkeypatch.setattr(backend.importlib, "import_module", import_module)
def _good_build(bpe_path=None, checkpoint_path=None, load_from_HF=True, enable_inst_interactivity=False,
device="cuda", eval_mode=True):
return None
class _GoodProcessor:
def __init__(self, model, resolution=1008, device="cuda", confidence_threshold=0.5):
pass
def set_image(self):
pass
def set_text_prompt(self):
pass
def reset_all_prompts(self):
pass
def test_missing_sam3_explains_how_to_install(monkeypatch):
monkeypatch.setattr(backend, "_API", None)
monkeypatch.setattr(backend.importlib.util, "find_spec", lambda name, *a: None)
with pytest.raises(RuntimeError, match="pip install --no-deps git\\+https://github.com/facebookresearch/sam3"):
backend.ensure_sam3()
def test_broken_sam3_import_is_reported(monkeypatch):
_install_fake_sam3(monkeypatch, fail="No module named 'pkg_resources'")
with pytest.raises(RuntimeError, match="failed to load: No module named 'pkg_resources'"):
backend.ensure_sam3()
def test_sam3_without_the_needed_api_is_reported(monkeypatch):
_install_fake_sam3(monkeypatch, build=lambda checkpoint_path=None: None, processor=types.SimpleNamespace())
with pytest.raises(RuntimeError, match="load_from_HF.*eval_mode.*Sam3Processor.set_image"):
backend.ensure_sam3()
class OldProcessor(_GoodProcessor):
def __init__(self, model):
pass
_install_fake_sam3(monkeypatch, build=_good_build, processor=OldProcessor)
with pytest.raises(RuntimeError, match=r"Sam3Processor\(device=\), Sam3Processor\(confidence_threshold=\)"):
backend.ensure_sam3()
def test_importing_sam3_does_not_change_tf32(monkeypatch):
before = (torch.backends.cuda.matmul.allow_tf32, torch.backends.cudnn.allow_tf32)
_install_fake_sam3(monkeypatch, build=_good_build, processor=_GoodProcessor, flip_tf32=True)
assert backend.ensure_sam3() == (_good_build, _GoodProcessor)
assert (torch.backends.cuda.matmul.allow_tf32, torch.backends.cudnn.allow_tf32) == before
def test_cpu_only_comfyui_gets_a_clear_error(monkeypatch):
monkeypatch.setattr(backend, "_mm", types.SimpleNamespace(get_torch_device=lambda: torch.device("cpu")))
with pytest.raises(RuntimeError, match="needs an NVIDIA GPU"):
backend.sam3_device()
def test_missing_checkpoint_says_where_to_get_it(monkeypatch, tmp_path):
monkeypatch.setattr(backend, "folder_paths", None)
monkeypatch.setattr(backend, "checkpoint_folders", lambda: [str(tmp_path)])
with pytest.raises(RuntimeError, match="huggingface.co/facebook/sam3"):
backend.resolve_checkpoint("sam3.pt")
(tmp_path / "sam3.pt").write_bytes(b"x")
assert backend.resolve_checkpoint("sam3.pt") == os.path.join(str(tmp_path), "sam3.pt")
with pytest.raises(RuntimeError, match="not a SAM 3 .pt checkpoint"):
backend.resolve_checkpoint("model.safetensors")
with pytest.raises(RuntimeError, match="must be a file"):
backend.resolve_checkpoint("../secret.pt")
def test_checkpoint_list_always_has_an_entry(monkeypatch, tmp_path):
monkeypatch.setattr(backend, "folder_paths", None)
monkeypatch.setattr(backend, "checkpoint_folders", lambda: [str(tmp_path)])
assert backend.checkpoint_choices() == ["sam3.pt"]
(tmp_path / "sam3.1.pt").write_bytes(b"x")
(tmp_path / "config.json").write_text("{}")
assert backend.checkpoint_choices() == ["sam3.1.pt"]
# --- node ----------------------------------------------------------------------
def _node_frame():
"""10 x 8 picture: a person filling columns 0-4, a bag inside it (columns 3-4, rows 2-3), a cat at columns 7-8."""
def seg(label, score, x1, y1, x2, y2):
return segs.make_segment(label, score, torch.ones((y2 - y1, x2 - x1), dtype=torch.bool), (x1, y1))
return segs.SegsFrame(8, 10, (seg("person", 0.9, 0, 0, 5, 8), seg("bag", 0.8, 3, 2, 5, 4),
seg("cat", 0.7, 7, 1, 9, 3)))
def _run_node(monkeypatch, concepts, misses=None, show_preview=False, frame=None):
calls, saved = [], []
def fake_segment(images, names, *args, **kwargs):
calls.append(names)
kwargs["misses"].update(misses or {})
frames = frame or _node_frame()
return tuple(frames.with_segments([s for s in frames.segments if s.label in names]) for _ in images)
monkeypatch.setattr(sam3_nodes.sam3, "segment", fake_segment)
monkeypatch.setattr(sam3_nodes, "save_temp_previews",
lambda pictures, prefix: saved.extend(pictures) or [{"filename": "x.jpg", "type": "temp"}])
out = sam3_nodes.NH_SAM3Segment().segment(torch.full((1, 8, 10, 3), 0.8), concepts, 0.5, "fast", 0, "sam3.pt",
False, show_preview)
return out, calls, saved
def test_segment_node_lists_formulas_and_background(monkeypatch):
(result, mask, count, report), calls, _ = _run_node(monkeypatch, "cat, dog\ncat")
assert calls == [["cat", "dog"]] and count == 1 and int(mask.sum()) == 4 # a plain list: the union, as before
assert sam3_nodes.NH_SAM3Segment.RETURN_NAMES == ("segs", "mask", "count", "report")
(_, mask, _, _), calls, _ = _run_node(monkeypatch, "person - bag")
assert calls == [["person", "bag"]] and int(mask.sum()) == 40 - 4
(_, mask, _, report), calls, _ = _run_node(monkeypatch, "background")
# background needs the subject: person is searched for on its own, and the report says so
assert calls == [["person"]] and int(mask.sum()) == 80 - 40
assert "background = everything outside person" in report and "person (searched for background)" in report
(_, mask, _, report), calls, _ = _run_node(monkeypatch, "nền - mèo, túi")
assert calls == [["cat", "bag", "person"]] and int(mask.sum()) == 80 - 40 - 4 + 4
assert "mèo -> cat: 1 found (best 0.70)" in report and "túi -> bag: 1 found (best 0.80)" in report
def test_segment_node_reports_what_it_could_not_find(monkeypatch):
(_, mask, count, report), _, _ = _run_node(monkeypatch, "dog, giraffe, áo xanh", misses={"dog": 0.31})
assert count == 0 and int(mask.sum()) == 0
assert "dog: none at threshold 0.50; the best candidate scored 0.31, lower the threshold" in report
assert "giraffe: none found; try another English word" in report
assert "áo xanh: not understood; write it in English" in report
@pytest.mark.parametrize("concepts, message", [
(" ,\n# none", "at least one concept"), ("all", "at least one concept"), ("*shirt", "wildcard"),
("person - (bag", "not closed"),
])
def test_segment_node_errors(monkeypatch, concepts, message):
with pytest.raises(ValueError, match=re.escape(message)):
_run_node(monkeypatch, concepts)
def test_segment_node_shows_a_small_preview_with_the_report(monkeypatch):
out, _, saved = _run_node(monkeypatch, "cat, dog", show_preview=True)
assert set(out) == {"ui", "result"} and out["ui"]["images"] == [{"filename": "x.jpg", "type": "temp"}]
assert len(out["result"]) == 4 and len(saved) == 1
picture = saved[0]
assert picture.dtype == np.uint8 and picture.shape[1] == 10 and picture.shape[0] > 8 # a caption strip below
assert sam3_nodes.NH_SAM3Segment.OUTPUT_NODE is True
spec = sam3_nodes.NH_SAM3Segment.INPUT_TYPES()
# optional, so an API prompt saved before 1.8 (without it) still validates; the editor still shows it
assert "show_preview" not in spec["required"] and spec["optional"]["show_preview"][1]["default"] is True
assert spec["required"]["checkpoint"][0]
def test_segment_node_runs_without_show_preview_and_survives_a_failed_preview(monkeypatch, capsys):
frame = _node_frame()
monkeypatch.setattr(sam3_nodes.sam3, "segment", lambda images, names, *a, **k: (frame,))
node = sam3_nodes.NH_SAM3Segment()
out = node.segment(torch.full((1, 8, 10, 3), 0.8), "cat", 0.5, "fast", 0, "sam3.pt", False) # as a 1.7 prompt
assert set(out) == {"ui", "result"} or len(out) == 4
def broken(pictures, prefix):
raise OSError("disk full")
monkeypatch.setattr(sam3_nodes, "save_temp_previews", broken)
result = node.segment(torch.full((1, 8, 10, 3), 0.8), "cat", 0.5, "fast", 0, "sam3.pt", False, True)
assert isinstance(result, tuple) and len(result) == 4 and int(result[1].sum()) == 4
assert "could not show the preview" in capsys.readouterr().out
# --- real model ----------------------------------------------------------------
def _real_sam3_ready():
if not torch.cuda.is_available() or importlib.util.find_spec("sam3") is None:
return False
try:
backend.resolve_checkpoint(backend.DEFAULT_CHECKPOINT)
except RuntimeError:
return False
return True
@pytest.mark.skipif(not _real_sam3_ready(), reason="needs CUDA, the sam3 package and models/sam3/sam3.pt")
def test_real_sam3_finds_the_astronaut_and_releases_vram():
from skimage import data
image = torch.from_numpy(data.astronaut().astype(np.float32) / 255.0)[None]
frames = backend.segment(image, ["person"], 0.5, "balanced", 0, backend.DEFAULT_CHECKPOINT, False)
people = frames[0].segments
assert len(people) >= 1 and people[0].label == "person" and people[0].score > 0.5
x1, y1, x2, y2 = people[0].box
assert 0 <= x1 < x2 <= 512 and 0 <= y1 < y2 <= 512 and people[0].area > 0.1 * 512 * 512
assert next(backend.load_model(backend.DEFAULT_CHECKPOINT).parameters()).device.type == "cpu"
assert "sam3" in sys.modules
def test_unreadable_tf32_switches_are_left_alone(monkeypatch):
class Refusing:
@property
def allow_tf32(self):
raise RuntimeError("mixing legacy and new precision APIs")
monkeypatch.setattr(torch.backends.cuda, "matmul", Refusing())
assert backend._tf32_flags() is None
backend._restore_tf32(None) # nothing to put back, no error
def test_checkpoint_check_rejects_foreign_and_broken_files(tmp_path):
good = tmp_path / "sam3.pt"
torch.save({"model": {"detector.backbone.w": torch.zeros(1), "tracker.decoder.w": torch.zeros(1)}}, good)
backend.check_checkpoint(str(good))
other = tmp_path / "other.pt"
torch.save({"encoder.w": torch.zeros(1)}, other)
with pytest.raises(RuntimeError, match="not a SAM 3 checkpoint"):
backend.check_checkpoint(str(other))
broken = tmp_path / "broken.pt"
broken.write_bytes(good.read_bytes()[:100])
with pytest.raises(RuntimeError, match="(?s)incomplete download.*huggingface.co/facebook/sam3"):
backend.check_checkpoint(str(broken))
def test_load_model_builds_once_and_keeps_real_holes(monkeypatch, tmp_path):
path = tmp_path / "sam3.pt"
path.write_bytes(b"x")
builds = []
def build(**kwargs):
builds.append(kwargs)
model = FakeModel()
model.inst_interactive_predictor = types.SimpleNamespace(_transforms=types.SimpleNamespace(max_hole_area=256.0))
return model
monkeypatch.setattr(backend, "_CACHE", {})
monkeypatch.setattr(backend, "resolve_checkpoint", lambda name: str(path))
monkeypatch.setattr(backend, "ensure_sam3", lambda: (build, None))
monkeypatch.setattr(backend, "sam3_device", lambda: torch.device("cpu"))
monkeypatch.setattr(backend, "check_checkpoint", lambda p: None)
monkeypatch.setattr(backend, "bpe_path", lambda: "vocab.txt.gz")
monkeypatch.setattr(backend.torch.cuda, "device", lambda device: contextlib.nullcontext())
model = backend.load_model("sam3.pt")
assert backend.load_model("sam3.pt") is model and len(builds) == 1
assert builds[0]["enable_inst_interactivity"] is True and builds[0]["load_from_HF"] is False
assert builds[0]["device"] == "cpu" and builds[0]["checkpoint_path"] == str(path)
assert builds[0]["bpe_path"] == "vocab.txt.gz" # always passed: bundled copies of sam3 look in the wrong place
assert model.inst_interactive_predictor._transforms.max_hole_area == 0.0
assert not any(p.requires_grad for p in model.parameters())
def test_failed_gpu_move_puts_the_model_back(monkeypatch):
class Unmovable(FakeModel):
def to(self, device, *args, **kwargs):
if torch.device(device).type != "cpu":
raise RuntimeError("CUDA out of memory")
return super().to(device, *args, **kwargs)
offloaded = []
monkeypatch.setattr(backend, "make_room_on_gpu", lambda nbytes: None)
monkeypatch.setattr(backend, "offload", lambda model: offloaded.append(model))
model = Unmovable()
with pytest.raises(RuntimeError, match="out of memory"):
backend.move_model(model, torch.device("cuda", 0))
assert offloaded == [model]
def test_segment_offloads_when_the_run_fails(monkeypatch):
model = FakeModel()
offloaded = []
monkeypatch.setattr(backend, "load_model", lambda checkpoint: model)
monkeypatch.setattr(backend, "ensure_sam3", lambda: (None, FakeProcessor))
monkeypatch.setattr(backend, "sam3_device", lambda: torch.device("cpu"))
monkeypatch.setattr(backend, "move_model", lambda m, device: None)
monkeypatch.setattr(backend, "offload", lambda m: offloaded.append(m))
def broken(*args, **kwargs):
raise RuntimeError("boom")
monkeypatch.setattr(backend, "segment_images", broken)
with pytest.raises(RuntimeError, match="boom"):
backend.segment(torch.rand(1, 8, 8, 3), ["x"])
assert offloaded == [model]
with pytest.raises(ValueError, match="empty image batch"):
backend.segment(torch.rand(0, 8, 8, 3), ["x"])
def test_offload_clears_position_caches_and_moves_rotary_tables():
PositionEmbeddingSine = type("PositionEmbeddingSine", (torch.nn.Module,), {})
model = FakeModel()
model.pos = PositionEmbeddingSine()
model.pos.cache = {(72, 72): torch.zeros(4)}
model.rope = torch.nn.Module()
model.rope.freqs_cis = torch.zeros(3)
backend._move_tables(model, torch.device("meta"))
assert model.rope.freqs_cis.device.type == "meta"
model.rope.freqs_cis = torch.zeros(3) # meta tensors hold no data to bring back
backend.offload(model)
assert model.pos.cache == {} and model.rope.freqs_cis.device.type == "cpu"
def test_wide_pictures_get_a_note(capsys):
model = FakeModel()
backend.segment_images(model, lambda threshold: FakeProcessor(threshold), torch.rand(1, 100, 600, 3), ["nothing"],
detail_level="fast", device=torch.device("cpu"))
assert "wider than 4:1" in capsys.readouterr().out
def test_one_and_four_channel_images_become_rgb():
gray = torch.rand(6, 8, 1)
assert torch.equal(backend._image_chw(gray), gray.permute(2, 0, 1).expand(3, -1, -1))
rgba = torch.rand(6, 8, 4)
assert torch.equal(backend._image_chw(rgba), rgba[..., :3].permute(2, 0, 1))
with pytest.raises(ValueError, match="1, 3 or 4 channels"):
backend._image_chw(torch.rand(6, 8, 2))
def _fake_sam3_package(monkeypatch, folder):
folder.mkdir(parents=True, exist_ok=True)
(folder / "__init__.py").write_text("")
monkeypatch.setitem(sys.modules, "sam3", types.SimpleNamespace(__file__=str(folder / "__init__.py"),
__path__=[str(folder)]))
def test_bpe_path_is_found_in_the_imported_sam3_copy(monkeypatch, tmp_path):
# ComfyUI-RMBG's layout: <pack>/models/sam3/assets/<vocab>, and its builder wrongly looks in <pack>/models/assets
package = tmp_path / "ComfyUI-RMBG" / "models" / "sam3"
_fake_sam3_package(monkeypatch, package)
with pytest.raises(RuntimeError, match="has no assets/bpe_simple_vocab_16e6.txt.gz"):
backend.bpe_path()
(package / "assets").mkdir()
(package / "assets" / backend.BPE_FILE).write_bytes(b"x")
assert backend.bpe_path() == os.path.normpath(str(package / "assets" / backend.BPE_FILE))
def test_bpe_path_also_accepts_assets_next_to_the_package(monkeypatch, tmp_path):
_fake_sam3_package(monkeypatch, tmp_path / "lib" / "sam3")
(tmp_path / "lib" / "assets").mkdir()
(tmp_path / "lib" / "assets" / backend.BPE_FILE).write_bytes(b"x")
assert backend.bpe_path() == os.path.normpath(str(tmp_path / "lib" / "assets" / backend.BPE_FILE))
+373
View File
@@ -0,0 +1,373 @@
"""NH_SEGS data model and the select / compose / convert / preview helpers (no model needed)."""
import importlib
import re
import numpy as np
import pytest
import torch
segs = importlib.import_module("NH-Nodes.core.segs")
segs_nodes = importlib.import_module("NH-Nodes.nodes.vision.segs_nodes")
def _seg(label, score, box, height=40, width=60):
"""Segment filling ``box`` (x1, y1, x2, y2) of a height x width image."""
mask = torch.zeros((height, width), dtype=torch.bool)
x1, y1, x2, y2 = box
mask[y1:y2, x1:x2] = True
return segs.make_segment(label, score, mask)
def _frame(*segments, height=40, width=60):
return segs.SegsFrame(height, width, tuple(segments))
# --- data model --------------------------------------------------------------
def test_make_segment_keeps_a_compact_tight_crop_with_origin_offset():
mask = torch.zeros((10, 12), dtype=torch.bool)
mask[2:5, 3:7] = True
seg = segs.make_segment("car", 0.9, mask, origin=(100, 50))
assert seg.box == (103, 52, 107, 55)
assert tuple(seg.mask.shape) == (3, 4) and seg.mask.all()
assert seg.area == 12 and seg.center == (105.0, 53.5)
# a compact copy, not a view keeping the big mask alive
assert seg.mask.untyped_storage().nbytes() == seg.mask.numel()
def test_make_segment_thresholds_float_masks_and_returns_none_when_empty():
assert segs.make_segment("x", 1.0, torch.zeros((4, 4))) is None
soft = torch.tensor([[0.2, 0.6], [0.5, 0.9]])
seg = segs.make_segment("x", 1.0, soft)
assert seg.box == (1, 0, 2, 2) and seg.mask.tolist() == [[True], [True]] # values above 0.5, tight box
assert segs.make_segment("x", 1.0, np.array([[0, 1]], dtype=bool)).box == (1, 0, 2, 1)
def test_union_and_instance_masks():
frame = _frame(_seg("a", 0.9, (0, 0, 10, 10)), _seg("b", 0.8, (5, 5, 20, 15)))
union = segs.segs_union((frame, frame))
assert union.shape == (2, 40, 60) and union.dtype == torch.float32
assert int(union[0].sum()) == 100 + 150 - 25
masks, labels = segs.instance_masks((frame,))
assert masks.shape == (2, 40, 60) and labels == ["a", "b"]
empty, none = segs.instance_masks((_frame(),))
assert empty.shape == (1, 40, 60) and float(empty.sum()) == 0 and none == []
def test_parse_label_list_splits_trims_dedupes_and_skips_comments():
text = "person, Red car\n# a comment\n\nwindow,person ,red_car"
assert segs.parse_label_list(text) == ["person", "Red car", "window"]
assert segs.parse_label_list("") == [] and segs.parse_label_list(None) == []
def test_label_matching_ignores_case_spacing_and_supports_wildcards():
assert segs.label_matches("Red Car", "red_car")
assert segs.label_matches("t-shirt", "*shirt")
assert segs.label_matches("person", "pers?n")
assert not segs.label_matches("person", "persons")
# --- select ------------------------------------------------------------------
def test_select_filters_by_label_score_and_area():
frame = _frame(_seg("person", 0.9, (0, 0, 10, 10)), _seg("person", 0.4, (20, 0, 30, 10)),
_seg("car", 0.95, (0, 20, 60, 40)))
picked = segs.select_segments(frame, ["person"], min_score=0.5)
assert [s.score for s in picked.segments] == [0.9]
big = segs.select_segments(frame, min_area=0.2) # car = 1200 / 2400 px
assert [s.label for s in big.segments] == ["car"]
small = segs.select_segments(frame, max_area=0.1)
assert sorted(s.score for s in small.segments) == [0.4, 0.9]
@pytest.mark.parametrize("order, expected", [
("score", ["c", "a", "b"]), ("area", ["b", "a", "c"]), ("left_to_right", ["a", "c", "b"]),
("right_to_left", ["b", "c", "a"]), ("top_to_bottom", ["c", "a", "b"]), ("bottom_to_top", ["b", "a", "c"]),
("center_first", ["c", "b", "a"]), ("as_found", ["a", "b", "c"]),
])
def test_sort_orders(order, expected):
frame = _frame(_seg("a", 0.5, (0, 10, 6, 16)), _seg("b", 0.2, (40, 20, 60, 40)), _seg("c", 0.9, (28, 0, 32, 4)))
assert [s.label for s in segs.select_segments(frame, order=order).segments] == expected
def test_select_start_and_count_slice_after_sorting():
frame = _frame(*[_seg(f"p{i}", 0.5, (i * 10, 0, i * 10 + 5, 5)) for i in range(5)])
second = segs.select_segments(frame, order="left_to_right", start=1, count=1)
assert [s.label for s in second.segments] == ["p1"]
rest = segs.select_segments(frame, order="right_to_left", start=3, count=0)
assert [s.label for s in rest.segments] == ["p1", "p0"]
assert segs.select_segments(frame, start=9).segments == ()
# --- compose -----------------------------------------------------------------
def _compose_frame():
return _frame(_seg("person", 0.9, (0, 0, 30, 40)), _seg("hair", 0.8, (0, 0, 30, 10)),
_seg("red car", 0.7, (40, 30, 60, 40)), _seg("t-shirt", 0.6, (0, 10, 30, 25)))
def _px(expression, frame=None):
return int(segs.evaluate(segs.parse_expression(expression), frame or _compose_frame()).sum())
def test_compose_operators_and_keywords():
assert _px("person") == 1200
assert _px("person - hair") == 900
assert _px("person | 'red car'") == _px("person + \"red car\"") == _px("person or 'red car'") == 1400
assert _px("person & hair") == _px("person and hair") == 300
assert _px("~person") == _px("!person") == _px("not person") == 2400 - 1200
assert _px("all") == 1400 and _px("image") == 2400 and _px("none") == 0
assert _px("'t-shirt'") == 450 and _px("*shirt") == 450 and _px("red_car") == 200
assert _px("unknown") == 0
def test_compose_hyphens_and_apostrophes_inside_words_belong_to_the_label():
assert _px("t-shirt") == 450 # no quotes needed
assert _px("person - hair") == _px("person -hair") == _px("person- hair") == 900
assert _px("person-hair") == 0 # one (unknown) label: subtracting needs spaces
frame = _frame(_seg("women's dress", 0.9, (0, 0, 10, 10)))
assert _px("\"women's dress\"", frame) == 100 and _px("women's_dress", frame) == 100
def test_compose_precedence_and_associativity():
# & binds tighter than | and -, which read left to right
assert _px("hair | person & 'red car'") == _px("hair | (person & 'red car')") == 300
assert _px("person - hair - 't-shirt'") == 1200 - 300 - 450
assert _px("person - (hair - 't-shirt')") == 900
assert _px("~(person | 'red car')") == 1000
@pytest.mark.parametrize("expression, message", [
("", "empty"), (" ", "empty"), (" , \n # only a comment", "empty"), ("'red car", "never closed"),
("(person | hair", "not closed"), ("person |", "ends where"), (")", "unexpected ')'"),
("person !", "unexpected '!' at position 8"), ("(person or)", "unexpected ')' at position 11"),
("person | , hair", "unexpected ','"), ("()", "holds nothing"), ("hair (person)", "unexpected '(' at position 6"),
])
def test_compose_errors_explain_where(expression, message):
with pytest.raises(segs.ExpressionError, match=re.escape(message)):
segs.parse_expression(expression)
def test_lists_comments_and_words_in_a_row():
# commas and new lines join like |, after every other operator; words in a row are one label
assert _px("person, red car") == _px("person\nred car") == _px("person | 'red car'") == 1400
assert _px("red car") == _px("Red Car") == 200 and _px("person hair") == 0 # one (unknown) label
assert _px("person - hair, red car") == 900 + 200
assert _px("(hair, red car) - person") == 200 and _px("person,,\n\nhair,") == 1200
assert _px("hair # the hair only\n# person\n") == 300
def test_background_is_everything_outside_the_instances():
assert _px("background") == 2400 - 1400 == _px("~all") and _px("~background") == _px("all")
assert _px("background - red car") == 1000 and _px("background | hair") == 1300
def test_vietnamese_labels_and_keywords():
frame = _frame(_seg("hair", 0.8, (0, 0, 30, 10)), _seg("bag", 0.7, (40, 30, 60, 40)))
assert _px("tóc", frame) == 300 and _px("tóc, túi", frame) == 500 and _px("nền", frame) == 2400 - 500
assert _px("tất cả - túi", frame) == 300 and _px("'nền'", frame) == 0 # quoted: a label, never a keyword
tree = segs.parse_expression("tóc - túi, nền, áo xanh, sky")
assert segs.expression_labels(tree) == ["hair", "bag", "áo xanh", "sky"]
assert segs.expression_terms(tree) == [("tóc", "hair", "translated"), ("túi", "bag", "translated"),
("nền", "background", "keyword"), ("áo xanh", "áo xanh", "unknown"),
("sky", "sky", "english")]
assert segs.uses_keyword(tree, "background") and not segs.uses_keyword(tree, "all")
def test_sam3_concepts_read_and_or_not_as_words():
tree = segs.parse_concepts("black and white dog, person - bag")
assert segs.expression_labels(tree) == ["black and white dog", "person", "bag"]
assert segs.expression_labels(segs.parse_expression("person and hair")) == ["person", "hair"]
def test_compose_node_reports_unmatched_labels(capsys):
(mask,) = segs_nodes.NH_SegsCompose().compose((_compose_frame(),), "person - ghost")
assert mask.shape == (1, 40, 60) and int(mask.sum()) == 1200
assert "'ghost'" in capsys.readouterr().out
def test_compose_node_catches_a_minus_without_spaces():
with pytest.raises(ValueError, match="person - hair"):
segs_nodes.NH_SegsCompose().compose((_compose_frame(),), "person-hair")
# a real hyphenated label, or parts that are not both labels, stay plain labels
assert int(segs_nodes.NH_SegsCompose().compose((_compose_frame(),), "t-shirt")[0].sum()) == 450
assert int(segs_nodes.NH_SegsCompose().compose((_compose_frame(),), "person-ghost")[0].sum()) == 0
def test_preview_picture_shrinks_darkens_outside_the_mask_and_adds_notes():
frame = segs.SegsFrame(400, 600, (_seg("car", 0.9, (300, 200, 500, 380), 400, 600),))
image = torch.full((400, 600, 3), 0.8)
mask = torch.zeros((400, 600))
mask[200:380, 300:500] = 1.0
picture = segs.preview_picture(image, frame, ["car: 1 found"], max_side=300, mask=mask)
assert picture.dtype == np.uint8 and picture.shape[1] == 300 and picture.shape[0] > 200 # caption strip below
assert abs(int(picture[20, 20, 0]) - round(204 * 0.35)) <= 1 # outside the mask: darkened
assert int(picture[150, 200].min()) > 140 # inside: bright, lightly tinted
plain = segs.preview_picture(image, frame)
assert plain.shape == (400, 600, 3) and int(plain[20, 20, 0]) == 204 # no mask, no notes: as Segs Preview
def test_scale_frame_keeps_tiny_instances_visible():
frame = segs.SegsFrame(400, 600, (_seg("dot", 0.9, (10, 10, 12, 12), 400, 600),
_seg("car", 0.8, (300, 200, 500, 380), 400, 600)))
small = segs.scale_frame(frame, 0.1)
assert (small.height, small.width) == (40, 60) and [s.label for s in small.segments] == ["dot", "car"]
assert small.segments[0].area == 1 and small.segments[1].box == (30, 20, 50, 38)
# --- Impact Pack SEGS --------------------------------------------------------
@pytest.mark.parametrize("box, factor, expected", [
((20, 10, 30, 20), 3.0, [10, 0, 40, 30]), # 30 x 30 centred on the box
((0, 0, 10, 10), 3.0, [0, 0, 30, 30]), # slid right/down to stay inside
((50, 30, 60, 40), 2.0, [40, 20, 60, 40]), # slid left/up
((5, 5, 55, 35), 3.0, [0, 0, 60, 40]), # larger than the image: the whole image
((11, 13, 12, 14), 1.0, [11, 13, 12, 14]), # factor 1: the box itself
])
def test_detail_crop_region(box, factor, expected):
assert segs.detail_crop_region(60, 40, box, factor) == expected
def test_detail_crop_region_always_contains_the_box_and_stays_inside():
rng = np.random.default_rng(0)
for _ in range(2000):
width, height = int(rng.integers(1, 300)), int(rng.integers(1, 300))
x1, y1 = int(rng.integers(0, width)), int(rng.integers(0, height))
x2, y2 = int(rng.integers(x1 + 1, width + 1)), int(rng.integers(y1 + 1, height + 1))
factor = float(rng.uniform(1.0, 6.0))
rx1, ry1, rx2, ry2 = segs.detail_crop_region(width, height, (x1, y1, x2, y2), factor)
assert 0 <= rx1 <= x1 and 0 <= ry1 <= y1 and x2 <= rx2 <= width and y2 <= ry2 <= height
def test_to_impact_segs_places_each_mask_in_its_crop_region():
frame = _frame(_seg("person", 0.9, (20, 10, 30, 20)))
(height, width), items = segs.to_impact_segs(frame, 3.0)
assert (height, width) == (40, 60) and len(items) == 1
item = items[0]
assert item._fields == ("cropped_image", "cropped_mask", "confidence", "crop_region", "bbox", "label",
"control_net_wrapper")
rx1, ry1, rx2, ry2 = item.crop_region
assert item.cropped_mask.dtype == np.float32 and item.cropped_mask.shape == (ry2 - ry1, rx2 - rx1)
full = np.zeros((40, 60), np.float32)
full[ry1:ry2, rx1:rx2] = item.cropped_mask
assert full[10:20, 20:30].all() and full.sum() == 100
assert item.bbox == [20, 10, 30, 20] and item.label == "person" and item.confidence == pytest.approx(0.9)
assert item.cropped_image is None
def test_preview_labels_stack_instead_of_hiding_each_other():
placed = [(30, 10, 100, 30)]
assert segs._free_spot(40, 10, 70, 20, placed, 200) == 31 # moved below the label already there
assert segs._free_spot(40, 185, 70, 20, [(30, 180, 100, 200)], 200) == 159 # near the bottom: moved above
assert segs._free_spot(200, 10, 70, 20, placed, 200) == 10 # no overlap: stays
frame = _frame(_seg("person", 0.9, (10, 10, 50, 40)), _seg("hair", 0.8, (10, 10, 50, 20)), height=200, width=200)
assert segs.render_preview(np.full((200, 200, 3), 128, np.uint8), frame).shape == (200, 200, 3)
def test_impact_node_checks_batch_index():
node = segs_nodes.NH_SegsToImpactSEGS()
frame = _frame(_seg("a", 0.9, (0, 0, 5, 5)))
assert node.convert((frame,), 3.0, 0)[0][0] == (40, 60)
with pytest.raises(ValueError, match="batch_index 1"):
node.convert((frame,), 3.0, 1)
# --- preview and nodes -------------------------------------------------------
def test_render_preview_tints_only_instances_and_handles_any_label_text():
image = np.full((40, 60, 3), 128, np.uint8)
frame = _frame(_seg("áo dài", 0.91, (10, 10, 30, 30)))
out = segs.render_preview(image, frame)
assert out.shape == image.shape and out.dtype == np.uint8
assert (out[20, 20] != 128).any() # inside: tinted
assert (out[38, 58] == 128).all() # far outside: untouched
assert (image == 128).all() # the input is not modified
assert (segs.render_preview(image, _frame()) == image).all()
def test_preview_batch_rejects_segs_from_another_image():
frame = _frame(_seg("a", 0.9, (0, 0, 5, 5)))
assert segs.preview_batch(torch.rand(1, 40, 60, 3), (frame,)).shape == (1, 40, 60, 3)
assert segs.preview_batch(torch.rand(1, 40, 60, 1), (frame,)).shape == (1, 40, 60, 3)
with pytest.raises(ValueError, match="batch"):
segs.preview_batch(torch.rand(2, 40, 60, 3), (frame,))
with pytest.raises(ValueError, match="60x40"):
segs.preview_batch(torch.rand(1, 20, 30, 3), (frame,))
def test_select_and_to_masks_nodes():
frame = _frame(_seg("person", 0.9, (0, 0, 10, 10)), _seg("person", 0.8, (40, 0, 50, 10)),
_seg("dog", 0.7, (20, 20, 30, 30)))
picked, mask, count = segs_nodes.NH_SegsSelect().select((frame,), "person", 0.0, 0.0, 1.0, "right_to_left", 0, 1)
assert count == 1 and picked[0].segments[0].box == (40, 0, 50, 10)
assert mask.shape == (1, 40, 60) and int(mask.sum()) == 100
masks, union, labels, n = segs_nodes.NH_SegsToMasks().convert((frame,))
assert masks.shape == (3, 40, 60) and union.shape == (1, 40, 60)
assert labels == "person\nperson\ndog" and n == 3
def test_to_masks_refuses_when_ram_is_short(monkeypatch):
import psutil
frame = _frame(*[_seg("window", 0.9, (i, 0, i + 2, 2)) for i in range(0, 20, 4)])
monkeypatch.setattr(psutil, "virtual_memory", lambda: type("Memory", (), {"available": 4 * 40 * 60 * 4})())
with pytest.raises(MemoryError, match="Segs Select"):
segs_nodes.NH_SegsToMasks().convert((frame,))
monkeypatch.setattr(psutil, "virtual_memory", lambda: type("Memory", (), {"available": 1 << 30})())
assert segs_nodes.NH_SegsToMasks().convert((frame,))[0].shape == (5, 40, 60)
def test_frames_tell_comfyui_cache_what_they_hold():
frame = _frame(_seg("a", 0.9, (0, 0, 10, 10)), _seg("b", 0.8, (5, 5, 20, 15)))
tensors = frame._comfy_cache_tensors()
assert len(tensors) == 2 and all(t is s.mask for t, s in zip(tensors, frame.segments))
def test_segs_preview_node_and_every_node_declares_matching_outputs():
frame = _frame(_seg("a", 0.9, (0, 0, 5, 5)))
(preview,) = segs_nodes.NH_SegsPreview().preview(torch.rand(1, 40, 60, 3), (frame,))
assert preview.shape == (1, 40, 60, 3) and preview.dtype == torch.float32
sam3_nodes = importlib.import_module("NH-Nodes.nodes.vision.sam3_nodes")
classes = [*segs_nodes.NODE_CLASS_MAPPINGS.values(), *sam3_nodes.NODE_CLASS_MAPPINGS.values()]
for cls in classes:
required = cls.INPUT_TYPES()["required"]
assert len(cls.RETURN_TYPES) == len(cls.RETURN_NAMES) and cls.CATEGORY == "NH-Nodes/Vision"
assert callable(getattr(cls(), cls.FUNCTION)) and required
assert all(isinstance(spec, tuple) and spec for spec in required.values())
def test_formulas_laid_out_over_several_lines_still_read():
# a line break next to an operator joins (as on 1.7), a line break between two labels lists them
assert _px("person\n- hair") == _px("person -\nhair") == _px("person\r\n- hair") == 900
assert _px("(person | 'red car')\n- hair") == 1100 and _px("~\nperson") == 1200
assert _px("hair\n| 'red car'") == 500 and _px("person &\nhair") == 300
assert _px("(\nperson\n)") == 1200 and _px("hair\n\n- hair") == 0
assert _px("person\nhair") == 1200 and _px("hair\n~person") == 300 + 1200 # a list; ~person is an item
def test_comments_start_only_where_a_word_could():
frame = _frame(_seg("car#2", 0.9, (0, 0, 10, 10)), _seg("car", 0.8, (20, 20, 30, 30)))
assert _px("car#2", frame) == 100 and _px("car # the second\n", frame) == 100
with pytest.raises(segs.EmptyExpressionError):
segs.parse_expression("# nothing but a comment")
with pytest.raises(segs.ExpressionError) as caught:
segs.parse_expression("(person) empty")
assert not isinstance(caught.value, segs.EmptyExpressionError)
def test_report_trusts_what_was_found_over_the_dictionary():
frame = _frame(_seg("café", 0.9, (0, 0, 10, 10)))
lines = segs.describe_request(segs.parse_concepts("café, bánh xèo"), (frame,), 0.5)
assert lines[0] == "café: 1 found (best 0.90)" # not Vietnamese after all: SAM 3 found it
assert lines[1].startswith("bánh xèo: not understood")
def test_compose_node_catches_words_left_without_an_operator():
with pytest.raises(ValueError, match=re.escape("person + hair")):
segs_nodes.NH_SegsCompose().compose((_compose_frame(),), "person hair")
assert int(segs_nodes.NH_SegsCompose().compose((_compose_frame(),), "red car")[0].sum()) == 200
assert int(segs_nodes.NH_SegsCompose().compose((_compose_frame(),), "person ghost")[0].sum()) == 0
def test_segs_select_reads_vietnamese_labels_but_not_formula_words():
frame = _frame(_seg("hair", 0.9, (0, 0, 10, 10)), _seg("all", 0.8, (20, 20, 30, 30)))
picked, _, count = segs_nodes.NH_SegsSelect().select((frame,), "tóc", 0.0, 0.0, 1.0, "score", 0, 0)
assert count == 1 and picked[0].segments[0].label == "hair"
picked, _, count = segs_nodes.NH_SegsSelect().select((frame,), "tất cả", 0.0, 0.0, 1.0, "score", 0, 0)
assert count == 0 # "tất cả" is the formula word all, not a label named "all"
+47
View File
@@ -0,0 +1,47 @@
"""Vietnamese words -> the English SAM 3 and the parsers understand."""
import importlib
import unicodedata
import pytest
vi = importlib.import_module("NH-Nodes.core.vi_words")
@pytest.mark.parametrize("written, english", [
("tóc", "hair"), ("Tóc", "hair"), (" tóc ", "hair"), ("váy", "dress"), ("túi", "bag"), ("người", "person"),
("cửa sổ", "window"), ("cửa", "door"), # the longer phrase wins
("da mặt", "face skin"), ("tóc vàng", "blonde hair"), # a phrase of its own beats noun + colour
("áo đỏ", "red top"), ("áo sơ mi trắng", "white shirt"), ("tóc dài", "long hair"),
("áo xanh lá", "green top"), ("cây to", "large tree"), ("quần jean xanh dương", "blue jeans"),
("nền", "background"), ("phông nền", "background"), ("tất cả", "all"),
("balo", "backpack"), # Vietnamese written without accents, listed on purpose
("người đàn ông", "man"), ("người phụ nữ", "woman"), ("các áo", "top"), ("những chiếc xe", "vehicle"),
("áo màu đỏ", "red top"), ("nền trắng", "background"), ("phông nền xám", "background"),
])
def test_translate_known_vietnamese(written, english):
assert vi.translate(written) == (english, "translated")
def test_decomposed_accents_from_some_keyboards_are_read_the_same():
decomposed = unicodedata.normalize("NFD", "tóc")
assert decomposed != "tóc" and vi.translate(decomposed) == ("hair", "translated")
@pytest.mark.parametrize("written", ["person", "red car", "t-shirt", "*shirt", "Window", "to", "be"])
def test_english_passes_through_unchanged(written):
assert vi.translate(written) == (written, "english")
@pytest.mark.parametrize("written", ["áo xanh", "con rồng", "tóc rất đẹp"])
def test_unknown_vietnamese_is_reported_not_guessed(written):
# "xanh" alone is blue or green: the node asks instead of guessing
assert vi.translate(written) == (written, "unknown")
assert vi.is_vietnamese(written)
def test_every_listed_word_is_lower_case_composed_and_translates_to_ascii():
for table in (vi.NOUNS, vi.ADJECTIVES):
for word, english in table.items():
assert word == vi.normalize(word), word
assert english.isascii() and english == english.strip().lower(), english
+129 -1
View File
@@ -10,7 +10,9 @@ Run from the ComfyUI root:
import importlib
import os
import sys
import types
import cv2
import numpy as np
import pytest
import torch
@@ -176,8 +178,16 @@ def test_nodes_end_to_end_with_stub_backend(monkeypatch):
labels[0:4, :] = 2 # top
labels[4:8, :] = 7 # pants
monkeypatch.setattr(vs, "load_segformer", lambda name, device: {"name": name})
monkeypatch.setattr(vs, "predict_labels", lambda bundle, img: labels)
# the label map scaled to whatever picture (whole frame or crop) the parser is shown
def parser(bundle, img, confidence=False):
scaled = cv2.resize(labels.astype(np.int32), (img.shape[1], img.shape[0]),
interpolation=cv2.INTER_NEAREST).astype(np.int64)
return (scaled, np.full(img.shape[:2], 0.9, np.float16)) if confidence else scaled
monkeypatch.setattr(vs, "predict_labels", parser)
monkeypatch.setattr(vs, "offload", lambda bundle: None)
monkeypatch.setattr(vs, "detect_faces", lambda img: []) # no face: Portrait Segment parses the whole frame
image = torch.rand(2, 8, 8, 3)
mask, rgba, mask_img = vs.NH_GarmentSegment().segment(image, "lower_garment", True, 0, 0, True, "cpu")
@@ -192,6 +202,124 @@ def test_nodes_end_to_end_with_stub_backend(monkeypatch):
assert mask[0, :4, :].all() and mask[0, 4:, :].sum() == 0 # label 2 is nose in the face model
def _centre_marker(bundle, img):
"""Stub parser: skin (1) everywhere and a 4 x 4 nose (2) at the centre of the picture it is shown."""
labels = np.ones(img.shape[:2], np.int64)
h, w = labels.shape
labels[h // 2 - 2:h // 2 + 2, w // 2 - 2:w // 2 + 2] = 2
return labels
def _face_parser(shown):
"""Stub face parser: hair (13) everywhere on the whole frame (the first call); on a crop, a face in its middle
(skin 1 with a nose 2 and lips 11 at the centre) and hair around it."""
def parser(bundle, img):
shown.append(img.shape[:2])
labels = np.full(img.shape[:2], 13, np.int64)
if len(shown) > 1:
h, w = labels.shape
labels[h // 4:3 * h // 4, w // 4:3 * w // 4] = 1
labels[h // 2 - 2:h // 2 + 2, w // 2 - 2:w // 2 + 2] = 2
labels[h // 2 + 3:h // 2 + 5, w // 2 - 2:w // 2 + 2] = 11
return labels
return parser
def test_portrait_labels_refine_the_whole_frame_around_each_face(monkeypatch):
shown = []
monkeypatch.setattr(vs, "predict_labels", _face_parser(shown))
monkeypatch.setattr(vs, "detect_faces", lambda img: [(40, 40, 60, 60, 0.9), (140, 40, 160, 60, 0.9)])
canvas = vs.portrait_labels({}, np.zeros((100, 200, 3), np.uint8))
# the whole frame first (a padded square), then a tight crop per face to confirm it, then a wide one per face
assert shown == [(200, 200), (44, 44), (44, 44), (72, 72), (72, 72)]
assert canvas[47, 50] == 2 and canvas[47, 150] == 2 # each face has its nose (tight crops, moved up a little)
assert canvas[95, 100] == 13 and canvas[50, 100] == 13 # outside the crops the whole-frame parse stays
def test_portrait_labels_ignore_boxes_that_are_not_faces(monkeypatch):
shown = []
monkeypatch.setattr(vs, "predict_labels", lambda bundle, img: shown.append(img.shape[:2]) or np.full(
img.shape[:2], 13, np.int64)) # the parser sees only hair in the "face" box (a hand, a pattern)
monkeypatch.setattr(vs, "detect_faces", lambda img: [(40, 40, 60, 60, 0.61)])
canvas = vs.portrait_labels({}, np.zeros((100, 200, 3), np.uint8))
assert shown == [(200, 200), (44, 44)] and (canvas == 13).all() # turned down: no wide crop, nothing pasted
def test_portrait_labels_parse_a_face_filling_the_frame_whole(monkeypatch):
shown = []
monkeypatch.setattr(vs, "predict_labels", _face_parser(shown))
monkeypatch.setattr(vs, "detect_faces", lambda img: [(20, 10, 180, 190, 0.95)]) # a close-up
vs.portrait_labels({}, np.zeros((200, 200, 3), np.uint8))
assert shown == [(200, 200)] # the same as the whole-frame parse
def test_face_detector_failure_falls_back_to_whole_frames(monkeypatch, capsys):
monkeypatch.setattr(vs, "_detection_failed", False)
def broken(img):
raise RuntimeError("Downloading opencv/face_detection_yunet failed")
monkeypatch.setattr(vs, "detect_faces", broken)
assert vs.find_faces(np.zeros((50, 50, 3), np.uint8)) == []
assert "face detector is not available" in capsys.readouterr().out
assert vs.find_faces(np.zeros((50, 50, 3), np.uint8)) == [] # not tried again, said once
assert capsys.readouterr().out == ""
def test_portrait_labels_without_a_face_parse_the_whole_frame(monkeypatch):
shown = []
monkeypatch.setattr(vs, "predict_labels", lambda bundle, img: shown.append(img.shape[:2]) or _centre_marker(
bundle, img))
looked = []
monkeypatch.setattr(vs, "detect_faces", lambda img: looked.append(img.shape[:2]) or [])
canvas = vs.portrait_labels({}, np.zeros((100, 200, 3), np.uint8))
assert looked == [(100, 200), (166, 332)] # a second look with a grey margin, for a face cut by the frame
# the whole frame, as a padded square (the parser learnt on square faces), nose at the centre of the frame
assert shown == [(200, 200)] and canvas.shape == (100, 200) and canvas[50, 100] == 2
def test_find_faces_takes_a_second_look_with_a_margin_for_faces_cut_by_the_frame(monkeypatch):
calls = []
def detector(img):
calls.append(img.shape[:2])
return [] if len(calls) == 1 else [(30.0, 40.0, 120.0, 150.0, 0.7)] # found on the padded copy
monkeypatch.setattr(vs, "detect_faces", detector)
faces = vs.find_faces(np.zeros((90, 120, 3), np.uint8))
assert calls == [(90, 120), (150, 200)] and faces == [(0.0, 10.0, 80.0, 90.0, 0.7)] # mapped back, clipped
monkeypatch.setattr(vs, "detect_faces", lambda img: [(1.0, 2.0, 3.0, 4.0, 0.9)])
assert vs.find_faces(np.zeros((90, 120, 3), np.uint8)) == [(1.0, 2.0, 3.0, 4.0, 0.9)]
def test_garment_labels_refine_a_crop_of_the_model_shape_around_the_clothes(monkeypatch):
shown = []
def parser(bundle, img, confidence=False):
shown.append(img.shape[:2])
labels = np.zeros(img.shape[:2], np.int64)
if len(shown) == 1: # the first look at the whole picture: the clothes, under another label, unsure
labels[50:150, 120:180] = 3
return labels, np.full(img.shape[:2], 0.5, np.float16)
labels[np.asarray(img)[..., 0] == 9] = 5 # the close look
return labels, np.full(img.shape[:2], 0.9, np.float16)
monkeypatch.setattr(vs, "predict_labels", parser)
monkeypatch.setattr(vs, "detect_faces", lambda img: [])
bundle = {"processor": types.SimpleNamespace(size={"height": 576, "width": 384})}
image = np.zeros((200, 300, 3), np.uint8)
image[50:150, 120:180, 0] = 9
canvas = vs.garment_labels(bundle, image)
crop_h, crop_w = shown[1]
assert shown[0] == (200, 300) and abs(crop_w / crop_h - 384 / 576) < 0.02 # the model's 2:3, not squeezed
assert (canvas[50:150, 120:180] == 5).all() and set(np.unique(canvas)) == {0, 5}
shown.clear()
monkeypatch.setattr(vs, "predict_labels", lambda bundle, img, confidence=False: shown.append(1) or (
np.zeros(img.shape[:2], np.int64), np.ones(img.shape[:2], np.float16)))
assert not vs.garment_labels(bundle, np.zeros((200, 300, 3), np.uint8)).any() and shown == [1] # no clothes
def test_input_types_expose_expected_widgets():
garment = vs.NH_GarmentSegment.INPUT_TYPES()["required"]
assert garment["part"][0] == ["upper_garment", "lower_garment", "footwear", "headwear", "all_garments"]