Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Vito Sansevero
|
cfc963bb55 |
@@ -1,9 +1,9 @@
|
||||
name: Windows
|
||||
name: Windows and macOS
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# Non-blocking Windows runs (#420; macOS dropped from CI in #571): never on pull requests or in the merge queue, and no context
|
||||
# Non-blocking portability runs (#420): never on pull requests or in the merge queue, and no context
|
||||
# here is a required check. A red run is fixed at its cause before the workspace cutover (#334).
|
||||
on:
|
||||
push:
|
||||
@@ -25,7 +25,10 @@ env:
|
||||
# run counts (tests/vendor/ci-workflow.test.mjs recomputes it, #493).
|
||||
# Measured 2026-10-29 on main runs 40935337099 (dd247e3), 40938407252 (dd247e3), 40941240624 (dd247e3):
|
||||
# slowest leg: 7m24s (py3.12 windows) / 7m31s (py3.13 windows) / 7m32s (py3.14 windows) -> 15 min
|
||||
# (windows 4m56s-7m32s)
|
||||
# (windows 4m56s-7m32s, macos 4m39s-6m14s)
|
||||
# native-trash (#499) has no green main runs yet: its probe, run 40949370943 (2026-10-29), took
|
||||
# about 1 minute per macOS leg, so ceil(1.5 x 2) + 3 gives it 6 minutes (step 4). Recalibrate
|
||||
# after three green runs.
|
||||
# native-trash-linux (#543) has no green main runs yet: it installs the dev requirements and runs
|
||||
# one test module, about 1 minute by the same estimate, so ceil(1.5 x 2) + 3 gives it 6 minutes.
|
||||
# Recalibrate after three green runs.
|
||||
@@ -37,7 +40,7 @@ jobs:
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
os: [windows-latest]
|
||||
os: [windows-latest, macos-latest]
|
||||
python-version: ["3.12", "3.13", "3.14"]
|
||||
|
||||
steps:
|
||||
@@ -57,7 +60,7 @@ jobs:
|
||||
|
||||
# The runner's TEMP is on C:, the OS disk, where a flush took about 6 ms
|
||||
# and a test database's create-to-delete about 111 ms; on the D: work
|
||||
# disk the same took 0.5 ms and 23 ms. Tests write their
|
||||
# disk the same took 0.5 ms and 23 ms (macOS: 19 ms). Tests write their
|
||||
# databases under TEMP, so it moves to a folder on D: (measured in #487,
|
||||
# run 40915866444). The runner's TEMP was an 8.3 short name
|
||||
# (C:\Users\RUNNER~1\...), so the new one is spelled by its short name too
|
||||
@@ -80,6 +83,31 @@ jobs:
|
||||
PM_TEST_SLOWEST: "40"
|
||||
run: python -m tests.run_suite -v
|
||||
|
||||
# The real Trash on both macOS CPUs (#499 design §10.2). It runs apart from the suite, so a red
|
||||
# suite never skips the native proof.
|
||||
native-trash:
|
||||
name: native trash on ${{ matrix.os }}
|
||||
runs-on: ${{ matrix.os }}
|
||||
timeout-minutes: 6
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
os: [macos-latest, macos-15-intel]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v7
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Native trash proof
|
||||
timeout-minutes: 4
|
||||
env:
|
||||
PM_NATIVE_TRASH: macos
|
||||
run: python -m unittest tests.test_trash_macos_native tests.test_trash_macos -v
|
||||
|
||||
# The same tests as the fake mount table, on a real ext4 mount used as the output root (#543). A
|
||||
# runner can mount with sudo; the tests run as the normal user, so owner and mode checks are real.
|
||||
native-trash-linux:
|
||||
|
||||
@@ -28,7 +28,7 @@ env:
|
||||
# py matrix, slowest py3.10 on windows-latest: 12m43s / 15m54s / 14m53s -> 27 min
|
||||
# (ubuntu 5m27s-6m27s, macos 6m14s-8m30s, windows 11m42s-15m54s)
|
||||
# 3.10 and 3.11 left the matrix and 3.14 joined it in #356; re-measure after three green main runs.
|
||||
# Windows moved to portability.yml (#420; macOS left CI in #571), which has its own measured limit (#493);
|
||||
# Windows and macOS moved to portability.yml (#420), which has its own measured limit (#493);
|
||||
# this Linux-only matrix keeps 27 minutes until it is re-measured.
|
||||
# real ComfyUI compatibility: 13m32s / 8m55s / 11m46s -> 24 min
|
||||
# coverage gate (85%): 7m26s / 7m22s / 7m54s -> 15 min
|
||||
@@ -39,7 +39,7 @@ env:
|
||||
# Pull requests run a fast subset (#420): Python 3.12 on ubuntu-latest, coverage, lint and frontend
|
||||
# modules. The merge queue (merge_group), pushes to main and manual runs add Python 3.13 and 3.14 on
|
||||
# Linux and the real ComfyUI job. "all tests" is the required gate that holds the queue to that set.
|
||||
# Windows runs in portability.yml (push to main, nightly, on demand) and never gate a merge.
|
||||
# Windows and macOS run in portability.yml (push to main, nightly, on demand) and never gate a merge.
|
||||
jobs:
|
||||
test:
|
||||
name: py${{ matrix.python-version }} on ${{ matrix.os }}
|
||||
|
||||
@@ -236,12 +236,12 @@ The documentation for those pages is in [README-legacy.md](README-legacy.md).
|
||||
|
||||
| Layer | What runs |
|
||||
| --- | --- |
|
||||
| Python | Over 200 test files on Linux and Windows with Python 3.12 to 3.14, with an 85% coverage gate |
|
||||
| Python | Over 200 test files on Linux, Windows and macOS with Python 3.12 to 3.14, with an 85% coverage gate |
|
||||
| Frontend | About 100 unit test files, with coverage floors per file |
|
||||
| Browser | Over 60 Playwright spec files in Chromium, Firefox, WebKit and touch emulation |
|
||||
| Real ComfyUI | A pinned CPU ComfyUI, started in Classic and Nodes 2.0, running real workflows |
|
||||
|
||||
Linux runs on every pull request and in the merge queue. Windows runs on every push to main and nightly. macOS is not tested; it should work, but support is best-effort. To run the tests yourself, see [TESTING.md](TESTING.md).
|
||||
Linux runs on every pull request and in the merge queue. Windows and macOS run on every push to main and nightly. To run the tests yourself, see [TESTING.md](TESTING.md).
|
||||
|
||||
## License
|
||||
|
||||
|
||||
+11
-25
@@ -261,7 +261,7 @@ the process exits non-zero.
|
||||
job fails anyway and the dump is what matters.
|
||||
- `PM_TEST_SLOWEST=N` (off by default) lists the N slowest tests after the
|
||||
summary, each timed from its `start:` line to the next one like the hang
|
||||
limit. The Windows jobs (`portability.yml`) set it to 40, so every
|
||||
limit. The Windows and macOS jobs (`portability.yml`) set it to 40, so every
|
||||
run shows where its time went (#487). The list prints only when the run
|
||||
ends: a job GitHub cancels or a watchdog exit prints none, so read the
|
||||
timestamps between `start:` lines instead.
|
||||
@@ -327,7 +327,7 @@ Python 3.13/3.14, real ComfyUI and Firefox/WebKit failures first show up in the
|
||||
merge queue, on the merge-group commit rather than the
|
||||
pushed one. Find that run with `gh run list --event merge_group --workflow test.yml`
|
||||
(or `viewerjs-check.yml`) and match its head branch
|
||||
(`gh-readonly-queue/main/pr-<N>-...`). Windows failures show up only
|
||||
(`gh-readonly-queue/main/pr-<N>-...`). Windows and macOS failures show up only
|
||||
after the merge, in `gh run list --workflow portability.yml`.
|
||||
|
||||
Start fixing as soon as a failed log shows the cause, but push the fix only after
|
||||
@@ -445,25 +445,12 @@ A hot SQLite read that returns many rows aggregates them into one step
|
||||
(`json_group_array`) instead of stepping once per row, so it does not wait for
|
||||
the GIL once per row while ComfyUI generates (#354, `get_image_link_matches`).
|
||||
|
||||
A metadata read never decodes pixels (#523). Pillow's `PngImageFile.text` loads
|
||||
every pixel to reach text chunks after IDAT; the scan (`_extract_comfyui_metadata`)
|
||||
and the image monitor (`ComfyUIMetadataExtractor.extract_metadata`) read PNG text
|
||||
through `utils/png_text.py`, which runs Pillow's own chunk handlers after the
|
||||
first IDAT and seeks over pixel data. The legacy `.text` read is the floor:
|
||||
`tests/test_png_text.py` compares both readers on every image under
|
||||
`tests/fixtures`, a curated corpus (text after IDAT, zTXt, iTXt, split IDAT, bad
|
||||
checksums, truncations, oversized text, APNG) and a seeded fuzz, and asserts that
|
||||
neither reader decodes pixels by patching `ImageFile.load`. The only reads that
|
||||
differ are files whose pixel data is undecodable or cut short: legacy found
|
||||
nothing, the walk keeps the text before the pixels. Animated PNGs still use
|
||||
`.text`.
|
||||
|
||||
The `perf budget` job in `.github/workflows/test.yml` is the reference
|
||||
measurement: `ubuntu-latest`, one Python, no coverage, only the wall-clock tests.
|
||||
It runs on every push to `main` and on `workflow_dispatch`, never on pull
|
||||
requests, because shared-runner timing must not block a merge. Dispatch it on a
|
||||
branch to measure before merging.
|
||||
Shared Windows runners do not run them; the structural tests cover
|
||||
Shared Windows and macOS runners do not run them; the structural tests cover
|
||||
portability. Run it locally with
|
||||
`PROMPTMANAGER_PERF_BUDGET=1 python -m unittest tests.test_api_workspace_images.TestWorkspaceImages.test_filter_budget`
|
||||
and quote the printed p95 in the PR body.
|
||||
@@ -659,10 +646,10 @@ Before changing paths, SQLite, streams, restore or cleanup:
|
||||
ordinary busy timeout; cross-thread handoffs wait on events with generous
|
||||
ceilings, never a short wall clock (#242).
|
||||
- Run native Windows CI early for these changes. Pull requests and the merge
|
||||
queue run only Linux, so start Windows on the branch with
|
||||
queue run only Linux, so start Windows and macOS on the branch with
|
||||
`gh workflow run portability.yml --ref <branch>`.
|
||||
Include a Windows impact and evidence note in the PR. All three Windows
|
||||
legs in that run, every Linux leg and every applicable
|
||||
Include a Windows impact and evidence note in the PR. All six Windows/macOS
|
||||
combinations in that run, every Linux leg and every applicable
|
||||
exact-head check must pass before merge. Portable `ntpath`/`PureWindowsPath`
|
||||
controls do not establish native filesystem, locking or cleanup success.
|
||||
|
||||
@@ -681,7 +668,7 @@ what runs:
|
||||
| `page workflows (firefox\|webkit, shard N/2)` | skipped | run |
|
||||
| `all tests`, `viewerjs` (gates) | run | run |
|
||||
| `perf budget` | skipped | push and dispatch only (not in the queue) |
|
||||
| `py3.12-3.14 on windows-latest` (`portability.yml`) | not run | push to `main`, nightly, `workflow_dispatch` only; never required |
|
||||
| `py3.12-3.14 on windows-latest/macos-latest` (`portability.yml`) | not run | push to `main`, nightly, `workflow_dispatch` only; never required |
|
||||
|
||||
GitHub has one required set for pull requests and the queue. A job skipped by
|
||||
its `if:` reports Success. A dropped matrix leg never reports, so it would stay
|
||||
@@ -694,9 +681,8 @@ guarantee:
|
||||
So a merge group cannot turn green unless all three Linux Python legs, real
|
||||
ComfyUI and the four browser shards passed.
|
||||
|
||||
Windows (Python 3.12-3.14) is out of the blocking path for now
|
||||
(maintainer, 2026-10-28). macOS is not tested in CI (maintainer, 2026-10-31, #571);
|
||||
its code stays and support is best-effort. They run in `portability.yml` on every push to `main`,
|
||||
Windows and macOS (Python 3.12-3.14) are out of the blocking path for now
|
||||
(maintainer, 2026-10-28). They run in `portability.yml` on every push to `main`,
|
||||
nightly and on demand, and no context there is required. A red portability run
|
||||
is still a failure: fix it at its cause, not by rerunning, and have every one
|
||||
green before the workspace cutover (#334). `tests/vendor/ci-workflow.test.mjs` pins the
|
||||
@@ -2565,8 +2551,8 @@ files to the system trash. **No code path unlinks or overwrites a media file.**
|
||||
("Moving files to the Recycle Bin isn't available on Windows: files larger than the bin's
|
||||
capacity would be deleted permanently."). Library-only removal works everywhere. Tests use `tests/fake_trash.py`; the
|
||||
browser test server installs it, so no test touches a real trash. The native proof,
|
||||
`tests/test_trash_macos_native.py`, runs only with `PM_NATIVE_TRASH=macos` on a Mac, by hand
|
||||
(CI dropped its `native-trash` macOS job in #571; it last passed on `macos-latest` and `macos-15-intel`): the startup
|
||||
`tests/test_trash_macos_native.py`, runs only with `PM_NATIVE_TRASH=macos` in
|
||||
portability.yml's `native-trash` job (`macos-latest` and `macos-15-intel`): the startup
|
||||
volume, a name clash, fresh APFS and FAT32 disk images, an image used as an output root,
|
||||
a read-only image and an unwritable folder (design and probe:
|
||||
`docs/egm/499-macos-trash/`, local).
|
||||
|
||||
+9
-6
@@ -435,9 +435,12 @@ except Exception as e:
|
||||
|
||||
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS", "WEB_DIRECTORY"]
|
||||
|
||||
# One startup line. ASCII only: a Windows console using a legacy code page
|
||||
# cannot print emoji, and a UnicodeEncodeError here would abort loading the node.
|
||||
print(
|
||||
f"\033[94m[ComfyUI-PromptManager] Version:\033[0m {get_version()}"
|
||||
f" - {len(NODE_CLASS_MAPPINGS)} tools loaded"
|
||||
)
|
||||
# Print startup message with loaded tools
|
||||
print()
|
||||
print(f"\033[94m[ComfyUI-PromptManager] Version:\033[0m {get_version()}")
|
||||
for node_key, display_name in NODE_DISPLAY_NAME_MAPPINGS.items():
|
||||
# ASCII only: a Windows console using a legacy code page cannot print emoji,
|
||||
# and a UnicodeEncodeError here would abort loading the whole node.
|
||||
print(f" \033[94mLoaded:\033[0m {display_name}")
|
||||
print(f"\033[94mTotal: {len(NODE_CLASS_MAPPINGS)} tools loaded\033[0m")
|
||||
print()
|
||||
|
||||
+1
-5
@@ -85,7 +85,6 @@ try:
|
||||
from ...database.operations import PromptDatabase
|
||||
from ...utils.file_prompt import parse_comfyui_prompt
|
||||
from ...utils.logging_config import get_logger
|
||||
from ...utils.png_text import png_text
|
||||
from ...utils.video_metadata import is_video_path, read_video_metadata
|
||||
except ImportError:
|
||||
import sys
|
||||
@@ -96,7 +95,6 @@ except ImportError:
|
||||
from database.operations import PromptDatabase
|
||||
from utils.file_prompt import parse_comfyui_prompt
|
||||
from utils.logging_config import get_logger
|
||||
from utils.png_text import png_text
|
||||
from utils.video_metadata import is_video_path, read_video_metadata
|
||||
|
||||
|
||||
@@ -1050,8 +1048,6 @@ class PromptManagerAPI(
|
||||
def _extract_comfyui_metadata(self, image_path):
|
||||
"""Embedded ComfyUI metadata of an image (PNG text chunks) or a video.
|
||||
|
||||
PNG text is read without decoding the pixels (``utils.png_text``, #523).
|
||||
|
||||
Videos go through ffprobe (see utils/video_metadata). Unreadable files
|
||||
are a warning, not an error: a scan over a large output tree meets
|
||||
truncated and foreign files, and it carries on past them.
|
||||
@@ -1060,7 +1056,7 @@ class PromptManagerAPI(
|
||||
return read_video_metadata(image_path)
|
||||
try:
|
||||
with Image.open(image_path) as img:
|
||||
return png_text(img)
|
||||
return dict(getattr(img, "text", None) or {})
|
||||
except Exception as e:
|
||||
self.logger.warning(
|
||||
f"Unreadable image {os.path.basename(str(image_path))}: {e}"
|
||||
|
||||
+3
-3
@@ -1668,9 +1668,9 @@ class AdminRoutesMixin:
|
||||
def _extract_batch_metadata_sync(self, file_batch):
|
||||
"""Extract metadata from a batch of files in one executor call.
|
||||
|
||||
Files are read on ``PromptManagerConfig.WORKER_THREADS`` threads.
|
||||
PNG text is read chunk by chunk without decoding pixels (#523); file
|
||||
reads and ffprobe release the GIL.
|
||||
Files are read on ``PromptManagerConfig.WORKER_THREADS`` threads:
|
||||
Pillow decodes the whole image to reach trailing text chunks and
|
||||
releases the GIL while doing so.
|
||||
"""
|
||||
from ..config import PromptManagerConfig
|
||||
|
||||
|
||||
@@ -145,10 +145,9 @@ test('long vocabularies append and Load more is keyboard reachable',async({page,
|
||||
await expect(loadMore(page)).toHaveCount(0);
|
||||
await expect(rows(page).nth(200).getByRole('button',{name:'Browse',exact:true})).toBeFocused();
|
||||
await expect(rows(page).locator('td:nth-child(1)')).toHaveText(manifest.tags);
|
||||
// Open the screen as a new history entry, not a reload: Firefox restores a reloaded page's scrolled tag window, and
|
||||
// its end then appends a page before the check (#567). Scrolling the window to its top first does not always stick.
|
||||
await page.goto('about:blank');
|
||||
await page.goto(route);
|
||||
// Reload with the window at its top: Firefox restores a scrolled window, whose end then appends a page before the check.
|
||||
await main(page).locator('.two-col .table-wrap').evaluate(node=>new Promise(resolve=>{node.scrollTop=0;requestAnimationFrame(()=>requestAnimationFrame(resolve));}));
|
||||
await page.reload();
|
||||
await expect(rows(page)).toHaveCount(100);
|
||||
// D183: the end of the list arrives inside the tag window.
|
||||
const scroll=top=>main(page).locator('.two-col .table-wrap').evaluate((node,y)=>new Promise(resolve=>{node.scrollTop=y ?? node.scrollHeight;requestAnimationFrame(()=>requestAnimationFrame(resolve));}),top);
|
||||
|
||||
@@ -390,16 +390,7 @@ class TestConsoleBanner(unittest.TestCase):
|
||||
|
||||
stream.flush()
|
||||
banner = stream.buffer.getvalue().decode("cp1252")
|
||||
lines = [
|
||||
line
|
||||
for line in banner.splitlines()
|
||||
if "PromptManager" in line or "loaded" in line
|
||||
]
|
||||
self.assertEqual(len(lines), 1, banner)
|
||||
self.assertRegex(
|
||||
lines[0], r"\[ComfyUI-PromptManager\] Version:.* \S+ - \d+ tools loaded$"
|
||||
)
|
||||
self.assertNotIn("Loaded:", banner)
|
||||
self.assertIn("Loaded:", banner)
|
||||
|
||||
|
||||
class PriorFileHandler(logging.FileHandler):
|
||||
@@ -562,7 +553,7 @@ class TestPackageLoggerIsolation(unittest.TestCase):
|
||||
original_print = print
|
||||
|
||||
def fail_banner(*args, **kwargs):
|
||||
if any("tools loaded" in str(arg) for arg in args):
|
||||
if any("Total:" in str(arg) for arg in args):
|
||||
captured["package"] = sys.modules[name]
|
||||
captured["handlers"] = [
|
||||
h
|
||||
|
||||
@@ -1,455 +0,0 @@
|
||||
"""PNG text without a pixel decode (#523), against the legacy reader floor.
|
||||
|
||||
Pillow's ``PngImageFile.text`` loads (decodes) every pixel to reach text
|
||||
chunks written after IDAT. ``utils.png_text.png_text`` walks the remaining
|
||||
chunks instead. The legacy reader (``Image.open(path).text``, every failure
|
||||
read as ``{}``) is the floor: a file it read must read the same, and the
|
||||
curated corpus below must read identically. The documented difference is
|
||||
pixel data, which the walk never decodes: undecodable or cut-short pixel data
|
||||
made the legacy reader fail, the walk still reads the text before it.
|
||||
"""
|
||||
|
||||
import io
|
||||
import random
|
||||
import struct
|
||||
import tempfile
|
||||
import unittest
|
||||
import zlib
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from PIL import Image, ImageFile, PngImagePlugin
|
||||
|
||||
from utils.metadata_extractor import ComfyUIMetadataExtractor
|
||||
from utils.png_text import png_text
|
||||
|
||||
FIXTURES = Path(__file__).resolve().parent / "fixtures"
|
||||
IMAGE_SUFFIXES = {".png", ".jpg", ".jpeg", ".webp", ".gif", ".bmp"}
|
||||
SIGNATURE = b"\x89PNG\r\n\x1a\n"
|
||||
FUZZ_SEED = 523
|
||||
FUZZ_CASES = 400
|
||||
|
||||
|
||||
def legacy_text(path):
|
||||
"""The legacy scan reader: Pillow's ``.text`` (a full decode)."""
|
||||
try:
|
||||
with Image.open(path) as image:
|
||||
return dict(getattr(image, "text", None) or {})
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def walked_text(path):
|
||||
"""The new reader under the same failure rule."""
|
||||
try:
|
||||
with Image.open(path) as image:
|
||||
return png_text(image)
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def chunk(kind, data):
|
||||
return (
|
||||
struct.pack(">I", len(data))
|
||||
+ kind
|
||||
+ data
|
||||
+ struct.pack(">I", zlib.crc32(kind + data) & 0xFFFFFFFF)
|
||||
)
|
||||
|
||||
|
||||
def chunks_of(raw):
|
||||
"""``[(kind, data)]`` of a well-formed PNG."""
|
||||
found, offset = [], 8
|
||||
while offset < len(raw):
|
||||
length, kind = struct.unpack(">I4s", raw[offset : offset + 8])
|
||||
found.append((kind, raw[offset + 8 : offset + 8 + length]))
|
||||
offset += 12 + length
|
||||
return found
|
||||
|
||||
|
||||
def assemble(parts):
|
||||
return SIGNATURE + b"".join(
|
||||
part if isinstance(part, bytes) else chunk(*part) for part in parts
|
||||
)
|
||||
|
||||
|
||||
def base_png(mode="RGB", size=(64, 64), text=None, noise=False):
|
||||
"""A PNG from Pillow (text before IDAT), as ``[(kind, data)]``."""
|
||||
image = Image.new(mode, size)
|
||||
if noise:
|
||||
rng = random.Random(7)
|
||||
image = Image.frombytes(
|
||||
"RGB", size, bytes(rng.randrange(256) for _ in range(size[0] * size[1] * 3))
|
||||
)
|
||||
info = PngImagePlugin.PngInfo()
|
||||
for key, value in (text or {}).items():
|
||||
info.add_text(key, value)
|
||||
buffer = io.BytesIO()
|
||||
image.save(buffer, format="PNG", pnginfo=info)
|
||||
return chunks_of(buffer.getvalue())
|
||||
|
||||
|
||||
def split_idat(parts, pieces=3):
|
||||
"""Spread the IDAT payload over ``pieces`` consecutive IDAT chunks."""
|
||||
data = b"".join(d for k, d in parts if k == b"IDAT")
|
||||
step = max(1, len(data) // pieces)
|
||||
idats = [(b"IDAT", data[i : i + step]) for i in range(0, len(data), step)]
|
||||
first = next(i for i, (k, _) in enumerate(parts) if k == b"IDAT")
|
||||
rest = [p for p in parts if p[0] != b"IDAT"]
|
||||
return rest[:first] + idats + rest[first:]
|
||||
|
||||
|
||||
def before_iend(parts, *extra):
|
||||
return parts[:-1] + list(extra) + parts[-1:]
|
||||
|
||||
|
||||
def text_chunk(key, value):
|
||||
return (b"tEXt", key.encode("latin-1") + b"\0" + value.encode("latin-1"))
|
||||
|
||||
|
||||
def ztxt_chunk(key, value):
|
||||
return (b"zTXt", key.encode() + b"\0\0" + zlib.compress(value.encode("latin-1")))
|
||||
|
||||
|
||||
def itxt_chunk(key, value, compressed=False):
|
||||
body = value.encode("utf-8")
|
||||
if compressed:
|
||||
body = zlib.compress(body)
|
||||
flag = b"\1\0" if compressed else b"\0\0"
|
||||
return (b"iTXt", key.encode() + b"\0" + flag + b"en\0" + b"Title\0" + body)
|
||||
|
||||
|
||||
def bad_crc(kind, data):
|
||||
good = chunk(kind, data)
|
||||
return good[:-4] + bytes(b ^ 0xFF for b in good[-4:])
|
||||
|
||||
|
||||
PROMPT = '{"3": {"class_type": "KSampler", "inputs": {"seed": 1}}}'
|
||||
WORKFLOW = '{"nodes": [], "links": []}'
|
||||
|
||||
|
||||
def curated():
|
||||
"""Name to PNG bytes; each must read the same with both readers."""
|
||||
plain = base_png(text={"prompt": PROMPT})
|
||||
trailing = before_iend(
|
||||
base_png(), text_chunk("prompt", PROMPT), text_chunk("workflow", WORKFLOW)
|
||||
)
|
||||
cases = {
|
||||
"no-text": assemble(base_png()),
|
||||
"text-before-pixels": assemble(plain),
|
||||
"text-after-pixels": assemble(trailing),
|
||||
"ztxt-after-pixels": assemble(
|
||||
before_iend(base_png(), ztxt_chunk("prompt", PROMPT))
|
||||
),
|
||||
"itxt-plain-and-compressed": assemble(
|
||||
before_iend(
|
||||
base_png(),
|
||||
itxt_chunk("prompt", "café " + PROMPT),
|
||||
itxt_chunk("workflow", WORKFLOW, compressed=True),
|
||||
)
|
||||
),
|
||||
"xmp-after-pixels": assemble(
|
||||
before_iend(
|
||||
base_png(),
|
||||
itxt_chunk("XML:com.adobe.xmp", "<x:xmpmeta/>"),
|
||||
text_chunk("prompt", PROMPT),
|
||||
)
|
||||
),
|
||||
"itxt-bad-utf8": assemble(
|
||||
before_iend(
|
||||
base_png(),
|
||||
(b"iTXt", b"prompt\0\0\0\0\0\xff\xfe"),
|
||||
text_chunk("workflow", WORKFLOW),
|
||||
)
|
||||
),
|
||||
"later-copy-wins": assemble(
|
||||
before_iend(plain, text_chunk("prompt", '{"later": 1}'))
|
||||
),
|
||||
"empty-key-and-no-separator": assemble(
|
||||
before_iend(base_png(), (b"tEXt", b"\0value"), (b"tEXt", b"lonely"))
|
||||
),
|
||||
"split-pixels-text-between": assemble(
|
||||
split_idat(before_iend(base_png(noise=True), text_chunk("prompt", PROMPT)))
|
||||
),
|
||||
"palette": assemble(
|
||||
before_iend(base_png(mode="P"), text_chunk("prompt", PROMPT))
|
||||
),
|
||||
"sixteen-bit": assemble(
|
||||
before_iend(base_png(mode="I;16"), text_chunk("prompt", PROMPT))
|
||||
),
|
||||
"bad-crc-before-pixels": assemble(
|
||||
[base_png()[0], bad_crc(*text_chunk("prompt", PROMPT))] + base_png()[1:]
|
||||
),
|
||||
"bad-crc-after-pixels": assemble(
|
||||
base_png()[:-1] + [bad_crc(*text_chunk("prompt", PROMPT))] + base_png()[-1:]
|
||||
),
|
||||
"unknown-chunk-after-pixels": assemble(
|
||||
before_iend(base_png(), (b"prVt", b"x" * 40), text_chunk("prompt", PROMPT))
|
||||
),
|
||||
"garbage-chunk-id-after-pixels": assemble(
|
||||
base_png()[:-1]
|
||||
+ [b"\0\0\0\4\x00\x01\x02\x03abcd\0\0\0\0"]
|
||||
+ [chunk(*text_chunk("prompt", PROMPT))]
|
||||
+ base_png()[-1:]
|
||||
),
|
||||
"oversized-ztxt-after-pixels": assemble(
|
||||
before_iend(
|
||||
base_png(),
|
||||
(
|
||||
b"zTXt",
|
||||
b"prompt\0\0"
|
||||
+ zlib.compress(b"a" * (PngImagePlugin.MAX_TEXT_CHUNK + 10)),
|
||||
),
|
||||
)
|
||||
),
|
||||
"pixels-length-past-end": assemble(
|
||||
[p for p in base_png() if p[0] != b"IDAT" and p[0] != b"IEND"]
|
||||
)
|
||||
+ b"\xff\xff\x00\x00IDAT"
|
||||
+ b"\0" * 64,
|
||||
}
|
||||
full = assemble(trailing)
|
||||
idat_at = full.index(b"IDAT")
|
||||
cases["truncated-in-pixels"] = full[: idat_at + 20]
|
||||
cases["no-end-after-pixels"] = full[: full.rindex(b"IEND") - 4]
|
||||
cases["truncated-in-trailing-text"] = full[: full.rindex(b"workflow") + 4]
|
||||
cases["unknown-chunk-past-end"] = (
|
||||
assemble(before_iend(base_png(), ztxt_chunk("prompt", PROMPT))[:-1])
|
||||
+ chunk(b"prVt", b"x" * 40)[:-20]
|
||||
)
|
||||
cases["not-a-png"] = b"nope, not an image at all"
|
||||
cases["signature-only"] = SIGNATURE
|
||||
return cases
|
||||
|
||||
|
||||
def corrupt_pixels():
|
||||
"""Complete IDAT chunks with valid CRCs over data zlib cannot inflate."""
|
||||
parts = before_iend(base_png(), text_chunk("prompt", PROMPT))
|
||||
return assemble(
|
||||
[(k, b"\x78\x9c" + b"\xff" * 30) if k == b"IDAT" else (k, d) for k, d in parts]
|
||||
)
|
||||
|
||||
|
||||
def apng():
|
||||
frames = [Image.new("RGB", (16, 16), colour) for colour in ("red", "blue")]
|
||||
info = PngImagePlugin.PngInfo()
|
||||
info.add_text("prompt", PROMPT)
|
||||
buffer = io.BytesIO()
|
||||
frames[0].save(
|
||||
buffer, format="PNG", save_all=True, append_images=frames[1:], pnginfo=info
|
||||
)
|
||||
return assemble(
|
||||
before_iend(chunks_of(buffer.getvalue()), text_chunk("workflow", WORKFLOW))
|
||||
)
|
||||
|
||||
|
||||
def fixture_images():
|
||||
return sorted(p for p in FIXTURES.rglob("*") if p.suffix.lower() in IMAGE_SUFFIXES)
|
||||
|
||||
|
||||
class _TempFiles(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self._tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self._tmp.cleanup)
|
||||
|
||||
def write(self, name, raw):
|
||||
path = Path(self._tmp.name) / name
|
||||
path.write_bytes(raw)
|
||||
return path
|
||||
|
||||
|
||||
class PngTextParityTests(_TempFiles):
|
||||
def test_every_fixture_image_reads_like_the_legacy_reader(self):
|
||||
images = fixture_images()
|
||||
self.assertGreaterEqual(len(images), 25)
|
||||
for path in images:
|
||||
with self.subTest(path=path.relative_to(FIXTURES)):
|
||||
self.assertEqual(walked_text(path), legacy_text(path))
|
||||
|
||||
def test_curated_corpus_reads_like_the_legacy_reader(self):
|
||||
for name, raw in curated().items():
|
||||
with self.subTest(name):
|
||||
path = self.write(name + ".png", raw)
|
||||
self.assertEqual(walked_text(path), legacy_text(path))
|
||||
|
||||
def test_curated_corpus_exercises_trailing_text(self):
|
||||
"""The corpus is not vacuous: trailing text exists and is read."""
|
||||
cases = curated()
|
||||
path = self.write("t.png", cases["text-after-pixels"])
|
||||
self.assertEqual(walked_text(path), {"prompt": PROMPT, "workflow": WORKFLOW})
|
||||
with Image.open(path) as image:
|
||||
self.assertNotIn("prompt", image.info)
|
||||
self.assertEqual(
|
||||
walked_text(self.write("x.png", cases["truncated-in-pixels"])), {}
|
||||
)
|
||||
|
||||
def test_itxt_values_keep_their_language_and_type(self):
|
||||
path = self.write("i.png", curated()["itxt-plain-and-compressed"])
|
||||
new, old = walked_text(path), legacy_text(path)
|
||||
self.assertIsInstance(new["prompt"], PngImagePlugin.iTXt)
|
||||
self.assertEqual(new["prompt"].lang, old["prompt"].lang)
|
||||
self.assertEqual(new["prompt"].tkey, old["prompt"].tkey)
|
||||
|
||||
def test_animated_png_falls_back_to_the_legacy_read(self):
|
||||
path = self.write("a.png", apng())
|
||||
self.assertEqual(walked_text(path), legacy_text(path))
|
||||
self.assertEqual(walked_text(path)["workflow"], WORKFLOW)
|
||||
|
||||
def test_truncated_pixels_read_alike_when_truncated_images_load(self):
|
||||
"""With LOAD_TRUNCATED_IMAGES set (a node pack may set it) both readers
|
||||
accept a short file alike."""
|
||||
path = self.write("t.png", curated()["truncated-in-pixels"])
|
||||
with patch.object(ImageFile, "LOAD_TRUNCATED_IMAGES", True):
|
||||
self.assertEqual(walked_text(path), legacy_text(path))
|
||||
|
||||
def test_a_text_decode_error_ends_the_walk_like_legacy(self):
|
||||
"""Pillow's ``load_end`` stops at a chunk whose handler raises
|
||||
``UnicodeDecodeError`` and keeps the text read so far."""
|
||||
path = self.write(
|
||||
"u.png",
|
||||
assemble(
|
||||
before_iend(
|
||||
base_png(),
|
||||
ztxt_chunk("prompt", PROMPT),
|
||||
text_chunk("workflow", WORKFLOW),
|
||||
ztxt_chunk("parameters", "steps 20"),
|
||||
)
|
||||
),
|
||||
)
|
||||
|
||||
def undecodable(stream, position, length):
|
||||
raise UnicodeDecodeError("latin-1", b"", 0, 1, "test")
|
||||
|
||||
with patch.object(PngImagePlugin.PngStream, "chunk_tEXt", undecodable):
|
||||
old = legacy_text(path)
|
||||
self.assertEqual(walked_text(path), old)
|
||||
self.assertEqual(old, {"prompt": PROMPT})
|
||||
|
||||
def test_the_text_memory_budget_counts_text_before_the_pixels(self):
|
||||
"""Pillow's MAX_TEXT_MEMORY spans the whole file: text before and
|
||||
after the pixels, each under it, fail together in both readers."""
|
||||
path = self.write(
|
||||
"m.png",
|
||||
assemble(
|
||||
before_iend(
|
||||
base_png(text={"prompt": PROMPT}), text_chunk("workflow", WORKFLOW)
|
||||
)
|
||||
),
|
||||
)
|
||||
budget = len(PROMPT) + len(WORKFLOW) - 1
|
||||
with patch.object(PngImagePlugin, "MAX_TEXT_MEMORY", budget):
|
||||
self.assertEqual(legacy_text(path), {})
|
||||
self.assertEqual(walked_text(path), {})
|
||||
with patch.object(PngImagePlugin, "MAX_TEXT_MEMORY", budget + 1):
|
||||
self.assertEqual(walked_text(path), legacy_text(path))
|
||||
self.assertEqual(len(walked_text(path)), 2)
|
||||
|
||||
def test_complete_but_undecodable_pixels_read_more_than_legacy(self):
|
||||
path = self.write("c.png", corrupt_pixels())
|
||||
self.assertEqual(legacy_text(path), {})
|
||||
self.assertEqual(walked_text(path), {"prompt": PROMPT})
|
||||
|
||||
def test_cut_short_pixels_read_the_text_before_them(self):
|
||||
full = assemble(base_png(text={"prompt": PROMPT}))
|
||||
path = self.write("s.png", full[: full.index(b"IDAT") + 12])
|
||||
self.assertEqual(legacy_text(path), {})
|
||||
self.assertEqual(walked_text(path), {"prompt": PROMPT})
|
||||
|
||||
def test_mutated_files_never_read_less_than_legacy(self):
|
||||
"""Seeded fuzz: truncations and byte flips of a file with text before,
|
||||
between and after its pixel chunks. Wherever the legacy reader found
|
||||
text, the walk finds exactly the same."""
|
||||
base = assemble(
|
||||
split_idat(
|
||||
before_iend(
|
||||
base_png(text={"workflow": WORKFLOW}),
|
||||
ztxt_chunk("prompt", PROMPT),
|
||||
itxt_chunk("parameters", "steps 20"),
|
||||
)
|
||||
)
|
||||
)
|
||||
rng = random.Random(FUZZ_SEED)
|
||||
compared = 0
|
||||
for case in range(FUZZ_CASES):
|
||||
raw = bytearray(base)
|
||||
if case % 2:
|
||||
del raw[rng.randrange(8, len(raw)) :]
|
||||
else:
|
||||
for _ in range(rng.randrange(1, 4)):
|
||||
raw[rng.randrange(8, len(raw))] = rng.randrange(256)
|
||||
path = self.write(f"fuzz{case}.png", bytes(raw))
|
||||
old = legacy_text(path)
|
||||
if old:
|
||||
compared += 1
|
||||
with self.subTest(case=case):
|
||||
self.assertEqual(walked_text(path), old)
|
||||
self.assertGreater(compared, FUZZ_CASES // 5)
|
||||
|
||||
def test_non_png_images_have_no_text(self):
|
||||
for name in ("tiny.jpg", "tiny.webp"):
|
||||
with Image.open(FIXTURES / "local-inspect" / name) as image:
|
||||
self.assertEqual(png_text(image), {})
|
||||
|
||||
|
||||
class NoPixelDecodeTests(_TempFiles):
|
||||
"""Metadata reads never decode pixels (the #523 memory finding)."""
|
||||
|
||||
def png_fixtures(self):
|
||||
return sorted((FIXTURES / "inspection_parity").glob("*.png"))
|
||||
|
||||
def test_png_text_does_not_load_pixels_and_leaves_the_image_usable(self):
|
||||
path = self.write("t.png", curated()["text-after-pixels"])
|
||||
with Image.open(path) as image:
|
||||
with patch.object(ImageFile.ImageFile, "load", autospec=True) as load:
|
||||
self.assertEqual(png_text(image)["workflow"], WORKFLOW)
|
||||
load.assert_not_called()
|
||||
image.load()
|
||||
self.assertEqual(image.size, (64, 64))
|
||||
self.assertEqual(image.text["workflow"], WORKFLOW)
|
||||
|
||||
def test_scan_reader_does_not_decode_pixels(self):
|
||||
from py.api import PromptManagerAPI
|
||||
|
||||
api = PromptManagerAPI.__new__(PromptManagerAPI)
|
||||
fixtures = self.png_fixtures() + [
|
||||
self.write("t.png", curated()["text-after-pixels"])
|
||||
]
|
||||
with patch.object(ImageFile.ImageFile, "load", autospec=True) as load:
|
||||
found = [api._extract_comfyui_metadata(str(p)) for p in fixtures]
|
||||
load.assert_not_called()
|
||||
self.assertEqual(found, [legacy_text(p) for p in fixtures])
|
||||
self.assertTrue(all(found))
|
||||
|
||||
def test_monitor_extractor_does_not_decode_pixels(self):
|
||||
extractor = ComfyUIMetadataExtractor()
|
||||
fixtures = self.png_fixtures()
|
||||
with patch.object(ImageFile.ImageFile, "load", autospec=True) as load:
|
||||
found = [extractor.extract_metadata(str(p)) for p in fixtures]
|
||||
load.assert_not_called()
|
||||
self.assertTrue(all(metadata and len(metadata) > 1 for metadata in found))
|
||||
|
||||
|
||||
class MonitorExtractorParityTests(_TempFiles):
|
||||
"""``extract_metadata`` answers exactly as with the legacy ``.text``."""
|
||||
|
||||
def legacy_extract(self, path):
|
||||
def old_text(image):
|
||||
return dict(getattr(image, "text", None) or {})
|
||||
|
||||
with patch("utils.metadata_extractor.png_text", old_text):
|
||||
return ComfyUIMetadataExtractor().extract_metadata(str(path))
|
||||
|
||||
def test_every_fixture_and_curated_file_extracts_like_legacy(self):
|
||||
paths = fixture_images() + [
|
||||
self.write(name + ".png", raw) for name, raw in curated().items()
|
||||
]
|
||||
extractor = ComfyUIMetadataExtractor()
|
||||
for path in paths:
|
||||
with self.subTest(path=path.name):
|
||||
self.assertEqual(
|
||||
extractor.extract_metadata(str(path)), self.legacy_extract(path)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Vendored
+18
-5
@@ -331,7 +331,7 @@ const testJobs = () => jobs(readWorkflow('test.yml'));
|
||||
|
||||
test('the Python matrix is Linux 3.12 on a pull request and Linux 3.12-3.14 on every other event', () => {
|
||||
const body = testJobs().get('test');
|
||||
assert.deepEqual(matrixList(body, 'os'), ['ubuntu-latest'], 'Windows never gates a merge');
|
||||
assert.deepEqual(matrixList(body, 'os'), ['ubuntu-latest'], 'Windows and macOS never gate a merge');
|
||||
const match = /^ {8}python-version: \$\{\{ github\.event_name == 'pull_request' && fromJSON\('(\[[^\]]*\])'\) \|\| fromJSON\('(\[[^\]]*\])'\) \}\}$/m.exec(body);
|
||||
assert.ok(match, 'python-version is chosen by event');
|
||||
assert.deepEqual({ pr: JSON.parse(match[1]), other: JSON.parse(match[2]) }, { pr: ['3.12'], other: ['3.12', '3.13', '3.14'] });
|
||||
@@ -339,21 +339,34 @@ test('the Python matrix is Linux 3.12 on a pull request and Linux 3.12-3.14 on e
|
||||
assert.match(body, /^ {4}name: py\$\{\{ matrix\.python-version \}\} on \$\{\{ matrix\.os \}\}$/m);
|
||||
});
|
||||
|
||||
test('Windows runs 3.12-3.14 on main, nightly and on demand, never on a PR or in the queue', () => {
|
||||
test('Windows and macOS run 3.12-3.14 on main, nightly and on demand, never on a PR or in the queue', () => {
|
||||
const text = readWorkflow('portability.yml');
|
||||
const triggers = text.slice(text.indexOf('\non:'), text.indexOf('\nconcurrency:'));
|
||||
for (const event of ['push', 'schedule', 'workflow_dispatch']) assert.match(triggers, new RegExp(`^ {2}${event}:`, 'm'));
|
||||
for (const event of ['pull_request', 'merge_group']) assert.doesNotMatch(triggers, new RegExp(`^ {2}${event}`, 'm'));
|
||||
assert.match(triggers, /^ {4}branches: \[main\]$/m, 'push runs only for main');
|
||||
const body = jobs(text).get('test');
|
||||
assert.deepEqual(matrixList(body, 'os'), ['windows-latest'], 'macOS is not tested in CI (#571)');
|
||||
assert.doesNotMatch(text, /macos-/, 'no macOS runner is used');
|
||||
assert.deepEqual(matrixList(body, 'os'), ['windows-latest', 'macos-latest']);
|
||||
assert.deepEqual(matrixList(body, 'python-version'), ['"3.12"', '"3.13"', '"3.14"']);
|
||||
assert.match(body, /^ {6}fail-fast: false$/m);
|
||||
assert.match(body, /^ {8}run: python -m tests\.run_suite -v$/m);
|
||||
assert.doesNotMatch(text, /cancel-in-progress: true|cancel-in-progress: \$/, 'no portability run is cancelled');
|
||||
});
|
||||
|
||||
// #499: the native trash proof runs on both macOS CPUs, apart from the suite so a red suite never skips it.
|
||||
test('portability runs the native trash proof on arm64 and Intel macOS', () => {
|
||||
const body = jobs(readWorkflow('portability.yml')).get('native-trash');
|
||||
assert.ok(body, 'portability.yml has a native-trash job');
|
||||
assert.deepEqual(matrixList(body, 'os'), ['macos-latest', 'macos-15-intel']);
|
||||
assert.match(body, /^ {6}fail-fast: false$/m);
|
||||
assert.doesNotMatch(body, /^ {4}needs:/m, 'it does not wait for the suite job');
|
||||
assert.match(body, /^ {10}PM_NATIVE_TRASH: macos$/m);
|
||||
assert.match(body, /^ {8}run: python -m unittest tests\.test_trash_macos_native tests\.test_trash_macos -v$/m);
|
||||
const jobMinutes = Number(/^ {4}timeout-minutes: (\d+)$/m.exec(body)[1]);
|
||||
const stepMinutes = Number(/^ {8}timeout-minutes: (\d+)$/m.exec(body)?.[1]);
|
||||
assert.ok(stepMinutes > 0 && jobMinutes - stepMinutes >= 2, 'the proof step stops before the job limit');
|
||||
});
|
||||
|
||||
// #493: the portability limit follows the rule in its comment: ceil(1.5 x the slowest of three
|
||||
// green main runs) + 3 minutes, never under 5, with those runs and times written above jobs:.
|
||||
// The times are whole-job wall times. That the runs were green and on main is the author's word.
|
||||
@@ -414,5 +427,5 @@ test('every required check reports on a pull request and the gates carry the ful
|
||||
].sort());
|
||||
assert.equal(contexts.filter(context => /^py/.test(context)).length, 1,
|
||||
'a dropped matrix leg never reports on a pull request, so it cannot be required');
|
||||
assert.ok(!contexts.some(context => /windows|macos/.test(context)), 'Windows never gates a merge');
|
||||
assert.ok(!contexts.some(context => /windows|macos/.test(context)), 'Windows and macOS never gate a merge');
|
||||
});
|
||||
|
||||
@@ -37,7 +37,6 @@ from typing import Optional, Dict, Any
|
||||
from PIL import Image
|
||||
|
||||
from .logging_config import get_logger
|
||||
from .png_text import png_text
|
||||
|
||||
|
||||
class ComfyUIMetadataExtractor:
|
||||
@@ -89,14 +88,12 @@ class ComfyUIMetadataExtractor:
|
||||
# Add basic file information
|
||||
metadata["file_info"] = self.get_file_info(image_path, image)
|
||||
|
||||
# Extract ComfyUI-specific metadata from PNG text chunks,
|
||||
# read without decoding the pixels (#523)
|
||||
text = png_text(image)
|
||||
if text:
|
||||
# Extract ComfyUI-specific metadata from PNG text chunks
|
||||
if hasattr(image, "text") and image.text:
|
||||
# Look for ComfyUI workflow data
|
||||
if "workflow" in text:
|
||||
if "workflow" in image.text:
|
||||
try:
|
||||
workflow_data = json.loads(text["workflow"])
|
||||
workflow_data = json.loads(image.text["workflow"])
|
||||
metadata["workflow"] = workflow_data
|
||||
|
||||
# Extract text encoder nodes from workflow
|
||||
@@ -110,9 +107,9 @@ class ComfyUIMetadataExtractor:
|
||||
self.logger.warning(f"Failed to parse workflow JSON: {e}")
|
||||
|
||||
# Look for prompt data
|
||||
if "prompt" in text:
|
||||
if "prompt" in image.text:
|
||||
try:
|
||||
prompt_data = json.loads(text["prompt"])
|
||||
prompt_data = json.loads(image.text["prompt"])
|
||||
metadata["prompt"] = prompt_data
|
||||
except json.JSONDecodeError as e:
|
||||
self.logger.warning(f"Failed to parse prompt JSON: {e}")
|
||||
@@ -132,13 +129,13 @@ class ComfyUIMetadataExtractor:
|
||||
]
|
||||
|
||||
for field in metadata_fields:
|
||||
if field in text:
|
||||
if field in image.text:
|
||||
try:
|
||||
# Try to parse as JSON first
|
||||
metadata[field] = json.loads(text[field])
|
||||
metadata[field] = json.loads(image.text[field])
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
# Store as string if not valid JSON
|
||||
metadata[field] = text[field]
|
||||
metadata[field] = image.text[field]
|
||||
|
||||
return (
|
||||
metadata
|
||||
|
||||
@@ -1,116 +0,0 @@
|
||||
"""PNG text chunks without decoding pixels (#523).
|
||||
|
||||
Pillow's ``PngImageFile.text`` calls ``load()``, which decodes every pixel, to
|
||||
reach text chunks written after the image data. On a scan or a busy image
|
||||
monitor that is a full-size buffer per file for a few kilobytes of text.
|
||||
|
||||
``png_text`` returns the same dictionary as ``image.text`` for an opened,
|
||||
single-frame PNG, by walking the chunks after the first IDAT with Pillow's own
|
||||
chunk handlers (same decoding, limits and duplicate rule) and seeking over the
|
||||
pixel data. It follows Pillow's ``load_end`` loop: chunk checksums after the
|
||||
pixels are not checked, an unreadable chunk header ends the walk, and another
|
||||
chunk that runs past the end of the file fails like ``ImageFile._safe_read``
|
||||
(unless ``ImageFile.LOAD_TRUNCATED_IMAGES``).
|
||||
|
||||
The one difference is pixel data, which is never decoded: pixel data that is
|
||||
undecodable, or cut short, no longer hides the text read before it. Pillow
|
||||
itself accepts a file cut inside its last pixel chunk once the decoder has
|
||||
all rows, and only a decode can tell those apart, so the walk never reads
|
||||
less than ``.text`` did (the PR #170 "legacy parity floor").
|
||||
|
||||
Anything else (other formats, animated PNGs, an image already loaded) is
|
||||
answered by Pillow itself.
|
||||
"""
|
||||
|
||||
import os
|
||||
import struct
|
||||
|
||||
from PIL import ImageFile
|
||||
|
||||
_PIXEL_CHUNKS = (b"IDAT", b"fdAT")
|
||||
# Signature, IHDR chunk and IDAT chunk header: the earliest pixel data offset.
|
||||
_FIRST_DATA_OFFSET = 8 + 25 + 8
|
||||
|
||||
|
||||
def png_text(image) -> dict:
|
||||
"""``dict(image.text)`` for an opened image, without a pixel decode.
|
||||
|
||||
The image stays usable: its file position and Pillow state are restored.
|
||||
Non-PNG images answer ``{}`` (only PNG has text chunks in Pillow).
|
||||
"""
|
||||
if not _walkable(image):
|
||||
return dict(getattr(image, "text", None) or {})
|
||||
fp = image.fp
|
||||
resume = fp.tell()
|
||||
try:
|
||||
return _walk_after_pixels(image.png, fp, image.tile[0][2] - 8)
|
||||
finally:
|
||||
fp.seek(resume)
|
||||
|
||||
|
||||
def _walkable(image) -> bool:
|
||||
"""A plain single-frame PNG whose header pass stopped at its first IDAT."""
|
||||
tile = getattr(image, "tile", None)
|
||||
return (
|
||||
image.format == "PNG"
|
||||
and getattr(image, "png", None) is not None
|
||||
and getattr(image, "fp", None) is not None
|
||||
and not getattr(image, "is_animated", False)
|
||||
and isinstance(tile, list)
|
||||
and len(tile) == 1
|
||||
and tile[0][0] == "zip"
|
||||
# The first IDAT's data offset; anything else is not a layout we know.
|
||||
and isinstance(tile[0][2], int)
|
||||
and tile[0][2] >= _FIRST_DATA_OFFSET
|
||||
)
|
||||
|
||||
|
||||
def _walk_after_pixels(stream, fp, offset: int) -> dict:
|
||||
"""Pillow's ``load_end`` chunk loop from the first IDAT header, skipping pixels.
|
||||
|
||||
``stream`` is the image's own ``PngStream``: its text so far and its text
|
||||
memory budget carry on exactly as a ``load()`` would carry them. The text
|
||||
and info dictionaries are copied first, so ``image.text`` and
|
||||
``image.info`` are unchanged. Like ``load_end``, a handler for a chunk
|
||||
misplaced after the pixels (a second IHDR or PLTE) may still update the
|
||||
stream's other header fields; Pillow ignores them once the image is open.
|
||||
"""
|
||||
end = fp.seek(0, os.SEEK_END)
|
||||
saved = stream.im_text, stream.im_info, stream.text_memory
|
||||
stream.im_text, stream.im_info = dict(saved[0]), dict(saved[1])
|
||||
try:
|
||||
fp.seek(offset)
|
||||
while True:
|
||||
try:
|
||||
kind, position, length = stream.read()
|
||||
except (struct.error, SyntaxError):
|
||||
break
|
||||
if kind == b"IEND":
|
||||
break
|
||||
try:
|
||||
handled = kind not in _PIXEL_CHUNKS and _handle(
|
||||
stream, kind, position, length
|
||||
)
|
||||
except UnicodeDecodeError:
|
||||
break
|
||||
if not handled:
|
||||
if position + length > end:
|
||||
if kind in _PIXEL_CHUNKS or ImageFile.LOAD_TRUNCATED_IMAGES:
|
||||
break
|
||||
raise OSError("Truncated File Read")
|
||||
fp.seek(position + length + 4)
|
||||
continue
|
||||
fp.read(4) # checksum: not verified after the pixels, as in Pillow
|
||||
return stream.im_text
|
||||
finally:
|
||||
stream.im_text, stream.im_info, stream.text_memory = saved
|
||||
|
||||
|
||||
def _handle(stream, kind: bytes, position: int, length: int) -> bool:
|
||||
"""Run Pillow's handler; ``False`` when the chunk's bytes must be skipped."""
|
||||
try:
|
||||
stream.call(kind, position, length)
|
||||
except (EOFError, AttributeError):
|
||||
# A chunk Pillow reads past (EOFError) or has no handler for.
|
||||
return False
|
||||
return True
|
||||
Reference in New Issue
Block a user