Update documentation and version for 1.3.5

.comfyignore

@@ -0,0 +1,3 @@
+# Kept in git, left out of the Registry archive
+docs/
+images/

CHANGELOG.md

@@ -3,7 +3,36 @@
 All notable changes to ComfyUI-ScriptFlow will be documented in this file.
 
 
-## [Unreleased]
+## [1.3.5] - 2026-10-12
+
+- The ComfyUI Registry archive no longer includes `docs/` and `images/` (added `.comfyignore`), which reduces it from about 3 MB to about 50 KB
+  - No change to the nodes; 1.3.4 was uploaded to the Registry but stayed pending review, and this release republishes it
+  - Both folders stay in the GitHub repository; in a Registry install, the README's demo image and the link to the technical report point to files that are not installed
+
+
+## [1.3.4] - 2026-10-09
+
+- `MultiOutputScript (Jev)` runs local d1 GGUF models, Liquid AI's open-weight decision model
+  - A GGUF whose `lfm2.decision.type` metadata is `lfm2-d1` (such as `LiquidAI/d1-3B-GGUF`) is asked in the prompt format d1 was tuned on, and the answer is read from the next-token probabilities of its yes/no, option code, or digit reply, instead of the lettered-option approximation used for general local models
+  - One forward pass per question; `jev.choice` and `jev.probabilities` take 2 to 26 options and `jev.score` takes 2 to 10 levels
+  - Text only, with the `state` and one question limited to 8,192 tokens
+  - Tested with d1-3B Q8_0 and the JamePeng fork of llama-cpp-python 0.4.2 on Windows (Intel Arc, SYCL build); Linux and macOS are untested
+- Added a technical report, `docs/local-model-uncertainty/` (English and Japanese), on how Jev, Clef-Flash, and a general local model (Qwen3.5-9B) answer questions whose answer cannot be known
+  - With a general local model, a low `jev.noul` value can mean "cannot say it is true" rather than "unlikely"
+  - Asking about both a statement and its negation, or picking a percentage level with `jev.score`, separates the two; the report has the measurements, example scripts, and the statements and results as JSON
+  - `README.md` and `README.ja.md` link to the report from the notes of `MultiOutputScript (Jev)`
+
+## [1.3.3] - 2026-10-05
+
+- `MultiOutputScript (Jev)` runs local Clef GGUF models, Cloudflare's open-weight decision model
+  - A GGUF whose architecture is `clef` (llama.cpp native format, such as `ggml-org/Clef-Flash-GGUF`) is answered by the model's decision head instead of the lettered-option approximation used for general local models
+  - Every question about one `state` is answered in one forward pass, so batched questions cost one pass per request
+  - Needs the JamePeng fork of llama-cpp-python 0.4.2 or later; other local models keep working with older builds
+  - Text only, with the `state` and all questions of a request limited to 8,192 tokens
+  - Tested with Clef-Flash Q8_0 on Windows (Intel Arc, SYCL build); Linux and macOS are untested
+- Added an application recipe to `README.md` and `README.ja.md` that checks a prompt enhancer's output against its system prompt with `MultiOutputScript (Jev)`
+  - The script splits the system prompt into rules, asks Jev whether the output breaks each one in a single request, and reports a score and the rules that are likely broken
+  - A second script for long system prompts checks the system prompt in parts of about 1,000 characters, staying under the script step limit
 
 
 ## [1.3.2] - 2026-10-03

README.ja.md

@@ -3,7 +3,7 @@
 
 ---
 
-**Version:** 1.3.2
+**Version:** 1.3.5
 **License:** GPL-3.0
 
 System One の判断モデルを使える、安全な Python 風スクリプトノードです。
@@ -55,6 +55,8 @@ System One モデルはテキストを生成せず、1 回の高速な処理で
 
 - ComfyUI を実行している Python 環境に [llama-cpp-python](https://github.com/JamePeng/llama-cpp-python) をインストールします。JamePeng 版と、本家の [abetlen/llama-cpp-python](https://github.com/abetlen/llama-cpp-python) 0.3.35 で動作を確認しています。
 - 指示追従型（instruction-tuned）の GGUF モデルを `ComfyUI/models/LLM`（または `models/text_encoders`）に置きます。GGUF にはチャットテンプレート（`tokenizer.chat_template` メタデータ）が含まれている必要があります。ほとんどの指示追従型 GGUF には含まれています。Qwen3.5-9B Q8_0 と Gemma-4-E4B Q8 で動作を確認しています。
+- Cloudflare のオープンウェイトの判断モデル [Clef](https://huggingface.co/Cloudflare/clef-flash) を使う場合は、llama.cpp ネイティブ形式の Clef の GGUF（[ggml-org/Clef-Flash-GGUF](https://huggingface.co/ggml-org/Clef-Flash-GGUF) など）を同じフォルダに置きます。JamePeng 版の llama-cpp-python 0.4.2 以降が必要です。通常のチャットモデルとして変換された Clef の GGUF は汎用モデルとして扱われ、Clef の判断ヘッドは使われません。Windows（Intel Arc、SYCL ビルド）の Clef-Flash Q8_0 で動作を確認しています。Linux と macOS では未確認です。
+- Liquid AI のオープンウェイトの判断モデル [d1](https://huggingface.co/LiquidAI/d1-3B) を使う場合は、d1 の GGUF（[LiquidAI/d1-3B-GGUF](https://huggingface.co/LiquidAI/d1-3B-GGUF) など）を同じフォルダに置きます。LFM2 モデルを読み込める llama-cpp-python が必要です。`mmproj` ファイルは不要です。Windows（Intel Arc、SYCL ビルド）の d1-3B Q8_0 と JamePeng 版の llama-cpp-python 0.4.2 で動作を確認しています。Linux と macOS では未確認です。
 
 ## 主な機能
 - 複数の入力：数値とテキストの入力を自由に組み合わせられます。
@@ -124,8 +126,10 @@ ov1, ov2 = (512, 384) if w >= h else (384, 512)
 
 - **TypeSafe API**（基本）：質問をインターネット経由で Jev に送ります。
 - **ローカル GGUF モデル**（フォールバック）：Jev API を使えない場合や、テキストを手元のマシンから出したくない場合に使います。汎用の指示追従型モデルで Jev 風の判断を近似するもので、[SemIf (OpenJev)](https://github.com/TheoLeeCJ/SemIf-OpenJev) の方式に従い、文字を割り当てた選択肢の次トークン確率を 1 回の順伝播で読み取ります。テキストは生成せず、ネットワークにも何も送りません。答えや確率は Jev とは異なります。
+- **ローカルの Clef モデル**：[Clef](https://huggingface.co/Cloudflare/clef-flash) は、Jev と同じ種類の質問に答えるオープンウェイトの判断モデルです。GGUF のメタデータ `general.architecture` が `clef` のモデルを、Clef として扱います。Clef の GGUF を選ぶと、上の近似の代わりにモデルの判断ヘッドを使い、同じ `state` についての質問すべてに 1 回の順伝播で答えます。`state` と質問を Clef の入力書式に並べ、判断ヘッドが出す選択肢ごとのスコアを、質問ごとに softmax で確率にします。文章は生成せず、ネットワークにも何も送信しません。答えと確率は Jev とは異なります。
+- **ローカルの d1 モデル**：[d1](https://huggingface.co/LiquidAI/d1-3B) は、Jev と同じ種類の質問に答えるオープンウェイトの判断モデルです。GGUF のメタデータ `lfm2.decision.type` が `lfm2-d1` のモデルを、d1 として扱います。d1 の GGUF を選ぶと、上の文字を割り当てた選択肢の代わりに、d1 が学習したプロンプト形式で質問します。答えは、d1 が学習した返答の次トークン確率から読み取ります。`jev.noul` では yes か no、`jev.choice` では選択肢のコード、`jev.score` では数字です。文章は生成せず、ネットワークにも何も送信しません。答えと確率は Jev とは異なります。
 
-スクリプトはどちらでも同じです。各バックエンドのセットアップは[インストール](#インストール)を参照してください。
+スクリプトはどれでも同じです。各バックエンドのセットアップは[インストール](#インストール)を参照してください。
 
 追加の入力：
 - `model`：`TypeSafe API`、またはローカルの GGUF モデル（`mmproj-*` ファイルは表示されません）。
@@ -134,9 +138,9 @@ ov1, ov2 = (512, 384) if w >= h else (384, 512)
 関数（`state` は文字列・辞書・リスト。テキストのみ）：
 - `jev.yes(state, question[, threshold])` → `bool`（Yes の確率が `threshold` 以上か。初期値 `0.5`）
 - `jev.noul(state, question)` → Yes の確率（`0.0`〜`1.0`）。しきい値と比較して使ってください。そのまま条件に使うとエラーになります。
-- `jev.choice(state, question, options)` → 最も確からしい選択肢（`str`）。`options` はラベルのリスト、またはラベルから説明への辞書です（ローカルモデルでは 2〜16 個）。
+- `jev.choice(state, question, options)` → 最も確からしい選択肢（`str`）。`options` はラベルのリスト、またはラベルから説明への辞書です（汎用のローカルモデルでは 2〜16 個）。
 - `jev.probabilities(state, question, options)` → 各選択肢とその確率の `dict`（合計 `1.0`）。`options` は `jev.choice` と同じです。同じ引数で両方を呼んでも、問い合わせは 1 回です。
-- `jev.score(state, question, levels)` → 順序付きの段階の説明に対する期待値（`float`、`0`〜`len(levels) - 1`）（ローカルモデルでは 2〜16 段階）
+- `jev.score(state, question, levels)` → 順序付きの段階の説明に対する期待値（`float`、`0`〜`len(levels) - 1`）（汎用のローカルモデルでは 2〜16 段階）
 
 ```python
 # it1: a prompt text
@@ -195,8 +199,12 @@ client.system_one(state=it1, questions={"q0": Noul(instructions="q1"), "q1": Nou
 
 注意：
 - TypeSafe API は `state` と質問をインターネット経由で送信します。外部に出せないテキストには使わないでください。
-- ローカルモデルの質問は 1 回の順伝播で処理されます（GPU 上の 4B〜9B モデルで約 1〜1.5 秒）。モデルは実行ごとに最初の質問で読み込まれ（数秒）、実行が終わるとアンロードされて、ワークフローの残りのために VRAM を解放します。
+- 汎用のローカルモデルは、質問 1 つにつき 1 回の順伝播で処理します（GPU 上の 4B〜9B モデルで約 1〜1.5 秒）。Clef モデルは、1 回のリクエストに含まれる質問すべてを 1 回の順伝播で処理します（Intel Arc の内蔵 GPU 上の Clef-Flash Q8_0 で、300 トークンのリクエストが約 3 秒）。d1 モデルは、質問 1 つにつき 1 回の順伝播で処理します（同じ GPU 上の d1-3B Q8_0 で約 0.5 秒）。モデルは実行ごとに最初の質問で読み込まれ（数秒）、実行が終わるとアンロードされて、ワークフローの残りのために VRAM を解放します。
 - ローカルモデルの確率は較正されておらず、モデルによって変わります。しきい値は実際の入力で試して決めてください。
+- Qwen3.5-9B での試験では、汎用のローカルモデルは、テキストから答えが決まらない質問に「No」と答えました。`jev.noul` の低い値は、「起こりにくい」ではなく「真だとは言えない」を表すことがあります。この2つを区別したい場合は、主張とその否定形の両方について聞くか、`jev.score` でパーセントの段階を選ばせてください。測定の結果とスクリプトの例は、[汎用のローカルモデルは、不確かな問いにどう答えるか](docs/local-model-uncertainty/README.ja.md) にあります。
+- ローカルモデルのプロンプトは 8,192 トークンまでです。汎用モデルでは `state` と質問 1 つ、Clef では `state` と 1 回のリクエストの質問すべてがこの範囲に収まる必要があります。Clef はテキストのみに対応しています。
+- d1 モデルでは、`jev.choice` と `jev.probabilities` の選択肢は 2〜26 個、`jev.score` の段階は 2〜10 個です。画像には対応していません。
+- d1-3B での試験では、似た選択肢に確率が分かれ、1 つだけ性質の違う選択肢が最多になることがありました。アスペクト比の例では、縦長の 3 つの比率の合計が 0.49 だったのに対し、0.35 の `1:1` が選ばれました。選択肢がグループに分かれる場合は、`jev.probabilities` をグループごとに合計してから選んでください。
 - 実行ごとにコンソールへ `[ComfyUI-ScriptFlow] Jev requests: N, states: S, questions: Q, responses: M` を出力します。送ったリクエスト数、異なる `state` の数、質問数、返信があったリクエスト数です。返信数がリクエスト数より少なければ、途中のリクエストが失敗しています。同じ実行のキャッシュから答えた質問は数えません。
 - `typesafe-sdk` がインストールされていると、`httpx2`（SDK が使う HTTP クライアント）と `typesafe_sdk` のロガーを WARNING に設定するため、リクエストごとの INFO ログは表示されません。
 - `MultiOutputScript` はモデルを読み込まず、ネットワークにもアクセスしません。`jev` は使えません。
@@ -459,6 +467,274 @@ ot1 = (
 
 </details>
 
+### プロンプトエンハンサーの出力をシステムプロンプトと照らし合わせる
+
+<details>
+<summary>プロンプトエンハンサーのルール確認レシピを表示</summary>
+
+プロンプトエンハンサーは、短い依頼文を、システムプロンプトのルールに従って画像・動画モデル用のプロンプトに書き直す LLM です。
+このレシピは、そのシステムプロンプトをルールに分け、エンハンサーの出力が各ルールを破っていないかを Jev に尋ねて、
+スコアと、破っている疑いのあるルールを出力します。ルールは接続したシステムプロンプトから読み取るので、特定のエンハンサーに依存しません。
+下のスクリプトを `MultiOutputScript (Jev)` の `code` 欄に貼り付けてください。
+
+推奨するワークフロー：
+
+```text
+enhancer output            -> MultiOutputScript (Jev).in_text_1
+system prompt              -> MultiOutputScript (Jev).in_text_2
+user request (optional)    -> MultiOutputScript (Jev).in_text_3
+MultiOutputScript (Jev)
+  -> out_text_1   report
+  -> out_text_2   likely violations only
+  -> out_text_3   rules extracted from the system prompt
+  -> out_value_1  adherence score (0-100)
+  -> out_value_2  number of likely violations
+```
+
+次のコードを `MultiOutputScript (Jev).code` に貼り付けます。
+
+```python
+# エンハンサー出力がシステムプロンプトにどれだけ従っているかを評価する（モデル非依存）
+# it1: エンハンサーの出力
+# it2: システムプロンプト（テキストそのもの、または "system_prompt" を含む JSON 設定の中身）
+# it3: ユーザー文（任意。ルールがユーザー文を参照する場合の判定材料）
+# ot1: レポート（違反の疑いが強い順） / ot2: 違反の疑いがあるルールのみ / ot3: 抽出したルール一覧
+# ov1: 準拠スコア (0-100) / ov2: 違反の疑いがあるルール数
+# Jev リクエストは 2 回（全ルールで 1 回、総合評価で 1 回。OVERALL = False なら 1 回）。max_jev_calls は 2 以上
+
+MAX_RULES = 40          # これを超えるルールは評価しない
+THRESHOLD = 0.5         # 違反確率がこれ以上なら違反とみなす
+SKIP_HEADINGS = ["例", "example", "sample", "思考", "手順", "thinking", "reasoning"]   # この語を含む見出しの節は評価しない
+OVERALL = True          # システムプロンプト全体に対する総合評価も行う
+
+def has_any(s, words):
+    for w in words:
+        if w in s:
+            return True
+    return False
+
+output = str(it1 or "").strip()
+sp = str(it2 or "")
+user_text = str(it3 or "").strip()
+
+if sp.strip().startswith("{"):
+    k = sp.find('"system_prompt"')
+    if k < 0:
+        raise ValueError("JSON に system_prompt がありません")
+    tail = sp[sp.find('"', sp.find(":", k) + 1) + 1:]
+    tail = tail.replace("\\\\", "\x00").replace('\\"', "\x01")
+    sp = tail[:tail.find('"')].replace("\\n", "\n").replace("\\t", "\t").replace("\x01", '"').replace("\x00", "\\")
+
+# 見出しごとに、行頭の項目または段落を1ルールとし、字下げされた行は直前のルールに含める
+rules = []
+heading = ""
+skip = False
+for line in sp.splitlines():
+    s = line.strip()
+    if not s:
+        continue
+    if s[0] in "【#[-":
+        if s[:3] == "---":
+            continue
+        if s[0] != "-" and (s[0] != "[" or s[-1] == "]"):
+            heading = s
+            skip = has_any(s.lower(), SKIP_HEADINGS)
+            continue
+    if skip:
+        continue
+    if rules and line[:1] in " \t　":
+        rules[-1][1] = rules[-1][1] + "\n" + s
+    else:
+        rules.append([heading, s])
+
+truncated = len(rules) > MAX_RULES
+rules = rules[:MAX_RULES]
+
+# 全ルールで state を共通にし、ルールは質問文に入れる（1 リクエストにまとまる）
+state = {"output": output}
+if user_text:
+    state["user_input"] = user_text
+results = []
+for r in rules:
+    q = "Does the output clearly break the instruction below? Answer no if the instruction does not concern the output, or if judging it needs a reference image or anything else not given here."
+    if r[0]:
+        q += "\nSection: " + r[0]
+    p = jev.noul(state, q + "\nInstruction: " + r[1])
+    results.append([float(p), r[0], r[1]])
+
+# 総合評価はシステムプロンプト全体を state にするので別リクエスト。並べ替えの前に聞いてルールと同時に送る
+if OVERALL:
+    overall_state = {"system_prompt": sp, "output": output}
+    if user_text:
+        overall_state["user_input"] = user_text
+    level = jev.score(overall_state, "How faithfully does the output follow the system prompt as a whole?", ["ignores most instructions", "follows some instructions", "follows about half", "follows most instructions", "follows every instruction"])
+
+ranked = list(reversed(sorted(results)))
+lines = []
+issues = []
+total = 0.0
+for r in ranked:
+    total += r[0]
+    bad = r[0] >= THRESHOLD
+    line = ("[違反?] " if bad else "[OK]    ") + f"p={r[0]:.2f} {r[1]} " + r[2].splitlines()[0][:70]
+    lines.append(line)
+    if bad:
+        issues.append(line)
+
+score = round(100 * (1 - total / len(ranked))) if ranked else 0
+header = f"準拠スコア: {score}/100 (ルール {len(ranked)} 件中、違反の疑い {len(issues)} 件)"
+if truncated:
+    header += f"\n※ルールが {MAX_RULES} 件を超えたため、以降は未評価"
+if OVERALL:
+    header += f"\n総合評価: {level:.2f} / 4"
+
+ot1 = header + "\n" + "\n".join(lines)
+ot2 = header + "\n" + "\n".join(issues)
+rule_lines = []
+for i, r in enumerate(rules):
+    rule_lines.append(f"{i + 1}. {r[0]} {r[1]}")
+ot3 = "\n\n".join(rule_lines)
+ov1 = score
+ov2 = len(issues)
+```
+
+出力される `out_text_1` は、次のようなレポートです。
+
+```text
+準拠スコア: 71/100 (ルール 4 件中、違反の疑い 1 件)
+総合評価: 3.10 / 4
+[違反?] p=0.74 ## 出力形式 - 文章は書かない。
+[OK]    p=0.21  あなたは SDXL 用のプロンプトエンハンサーです。
+[OK]    p=0.12 ## 出力形式 - 先頭は「masterpiece, best quality」にする。
+[OK]    p=0.09 ## 出力形式 - カンマ区切りのタグを 1 行で出力する。
+```
+
+注意：
+
+- v1.3.2 以降が必要です。v1.3.2 以降では、同じ `state` についての質問を 1 回のリクエストにまとめて送ります。このスクリプトのリクエストは 2 回なので、`max_jev_calls` は 2 以上にしてください。
+- `p` は、出力がそのルールを破っていると Jev が判断した確率です。確実な検査ではありません。テストでは、守られているルールを違反と判定することも、破られているルールを見逃すこともあり、`THRESHOLD` 付近の値は実行のたびに変わりました。違反と判定されたルールは、確認すべき候補として扱ってください。
+- このレシピは、TypeSafe API の Jev で作成しました。テストでは、ローカルの Clef-Flash モデルは、出力が守っているルールも含めて大半のルールを違反と判定したため、このレシピには向きません。
+- 参照画像のように、スクリプトに渡していない情報が必要なルールは判定できません。ユーザーの依頼文に関するルールを判定するには、依頼文を `in_text_3` につないでください。
+- `in_text_2` には、システムプロンプトのテキストそのもの、または `"system_prompt"` キーを持つ JSON 設定のテキストを渡せます。JSON の `\uXXXX` エスケープは復元しません。
+- `#`、`【`、`[...]` で始まる行は見出しとして扱います。見出しの下では、行頭から始まる行を 1 つのルールとし、字下げされた行は直前のルールに含めます。システムプロンプトがどう分けられたかは `out_text_3` で確認できます。
+- 見出しに `SKIP_HEADINGS` の語を含む節は評価しません。例や思考の手順がルールとして判定されるのを防ぐためです。
+- 評価するのは先頭から `MAX_RULES` 件までです。システムプロンプトが非常に長いと、スクリプトのステップ上限を超えます。その場合は、下の[長いシステムプロンプトの場合](#長いシステムプロンプトの場合)を参照してください。
+- TypeSafe API を使う場合、システムプロンプト、エンハンサーの出力、ユーザーの依頼文はインターネット経由で送信されます。
+- `out_text_3` と数値出力は必要に応じて使ってください。`out_value_3` は使いません。
+
+#### 長いシステムプロンプトの場合
+
+上のスクリプトはシステムプロンプトを 1 行ずつ評価するため、数百行あるシステムプロンプトでは `Script exceeded step limit` で止まります。
+その場合は、下のスクリプトを使ってください。システムプロンプトを空行の位置で `PART_CHARS` 字程度の部分に区切り、
+出力がその部分の指示を破っていないかを、部分ごとに Jev に尋ねます。入力と出力の接続は同じです。
+
+```python
+# 長いシステムプロンプト向け：エンハンサー出力がシステムプロンプトにどれだけ従っているかを、部分ごとに評価する
+# システムプロンプトを空行の位置で PART_CHARS 字程度の部分に区切り、部分ごとに違反の有無を Jev に聞く
+# （1 行ずつ評価する prompt_rule_check.py がステップ上限を超える長さのシステムプロンプト用）
+# it1: エンハンサーの出力
+# it2: システムプロンプト（テキストそのもの、または "system_prompt" を含む JSON 設定の中身）
+# it3: ユーザー文（任意。指示がユーザー文を参照する場合の判定材料）
+# ot1: レポート（違反の疑いが強い順） / ot2: 違反の疑いがある部分のみ / ot3: 区切った部分の一覧
+# ov1: 準拠スコア (0-100) / ov2: 違反の疑いがある部分の数
+# Jev リクエストは 2 回（全部分で 1 回、総合評価で 1 回。OVERALL = False なら 1 回）。max_jev_calls は 2 以上
+
+PART_CHARS = 1000       # 1 つの部分のおおよその文字数。小さくすると細かく評価する（質問数が増える）
+MAX_PARTS = 60          # これを超える部分は評価しない
+THRESHOLD = 0.5         # 違反確率がこれ以上なら違反とみなす
+OVERALL = True          # システムプロンプト全体に対する総合評価も行う
+
+output = str(it1 or "").strip()
+sp = str(it2 or "")
+user_text = str(it3 or "").strip()
+
+if sp.strip().startswith("{"):
+    k = sp.find('"system_prompt"')
+    if k < 0:
+        raise ValueError("JSON に system_prompt がありません")
+    tail = sp[sp.find('"', sp.find(":", k) + 1) + 1:]
+    tail = tail.replace("\\\\", "\x00").replace('\\"', "\x01")
+    sp = tail[:tail.find('"')].replace("\\n", "\n").replace("\\t", "\t").replace("\x01", '"').replace("\x00", "\\")
+sp = sp.replace("\r\n", "\n").strip()
+
+# PART_CHARS 字を超えた後の最初の空行で区切る（段落の途中では切らない）
+parts = []
+pos = 0
+while pos < len(sp) and len(parts) < MAX_PARTS:
+    end = sp.find("\n\n", pos + PART_CHARS)
+    if end < 0:
+        end = len(sp)
+    parts.append(sp[pos:end].strip())
+    pos = end
+truncated = pos < len(sp)
+
+# 全部分で state を共通にし、部分は質問文に入れる（1 リクエストにまとまる）
+state = {"output": output}
+if user_text:
+    state["user_input"] = user_text
+results = []
+for i, part in enumerate(parts):
+    p = jev.noul(state, "Does the output clearly break any instruction in the part of the system prompt below? Answer no if the instructions do not concern the output, or if judging them needs a reference image or anything else not given here.\nPart of the system prompt:\n" + part)
+    results.append([float(p), i + 1])
+
+# 総合評価はシステムプロンプト全体を state にするので別リクエスト。並べ替えの前に聞いて部分と同時に送る
+if OVERALL:
+    overall_state = {"system_prompt": sp, "output": output}
+    if user_text:
+        overall_state["user_input"] = user_text
+    level = jev.score(overall_state, "How faithfully does the output follow the system prompt as a whole?", ["ignores most instructions", "follows some instructions", "follows about half", "follows most instructions", "follows every instruction"])
+
+lines = []
+issues = []
+total = 0.0
+for r in reversed(sorted(results)):
+    total += r[0]
+    bad = r[0] >= THRESHOLD
+    # 部分の先頭行を見出し代わりに表示する（区切り線だけの行は飛ばす）
+    title = parts[r[1] - 1].strip("=-_*# \n\t").splitlines()[0][:60]
+    line = ("[違反?] " if bad else "[OK]    ") + f"p={r[0]:.2f} 部分{r[1]}: {title}"
+    lines.append(line)
+    if bad:
+        issues.append(line)
+
+score = round(100 * (1 - total / len(parts))) if parts else 0
+header = f"準拠スコア: {score}/100 (部分 {len(parts)} 件中、違反の疑い {len(issues)} 件)"
+if truncated:
+    header += f"\n※部分が {MAX_PARTS} 件を超えたため、以降は未評価"
+if OVERALL:
+    header += f"\n総合評価: {level:.2f} / 4"
+
+ot1 = header + "\n" + "\n".join(lines)
+ot2 = header + "\n" + "\n".join(issues)
+part_texts = []
+for i, part in enumerate(parts):
+    part_texts.append(f"--- 部分{i + 1} ---\n{part}")
+ot3 = "\n\n".join(part_texts)
+ov1 = score
+ov2 = len(issues)
+```
+
+出力される `out_text_1` は、次のようなレポートです。
+
+```text
+準拠スコア: 74/100 (部分 19 件中、違反の疑い 2 件)
+総合評価: 2.90 / 4
+[違反?] p=0.78 部分7: 出来事はユーザーが指定した順序のままにする。
+[違反?] p=0.61 部分12: カメラの動きは、ユーザーが求めたときだけ書く。
+[OK]    p=0.34 部分1: あなたは動画モデル用のプロンプトエンハンサーです。
+[OK]    p=0.22 部分3: 長さはユーザーが指定した秒数にする。
+```
+
+注意：
+
+- `max_jev_calls`、`p`、`in_text_2`、TypeSafe API についての上の注意は、このスクリプトにも当てはまります。
+- 部分は、`PART_CHARS` 字を超えた後の最初の空行で区切るので、段落の途中では切れません。見出しや箇条書きは解釈しないので、どんな書き方のシステムプロンプトでも使えますが、例の節は飛ばしません。
+- レポートは、各部分をその先頭の行で示します。違反と判定された部分の全文は `out_text_3` で読み、どの指示が問題なのかを確認してください。
+- 1 つの質問に複数の指示が含まれるので、1 行ずつ評価するスクリプトより結果は粗くなります。`PART_CHARS` を小さくすると、部分が細かくなり、質問数が増えます。
+- テストでは、約 20,000 字のシステムプロンプトが 19 個の部分に区切られ、約 2,200 ステップで実行できました。ステップ上限には十分収まります。
+
+</details>
+
 ## ライセンス
 
 このプロジェクトは **GNU General Public License v3.0** の下でライセンスされています。
@@ -484,9 +760,6 @@ You should have received a copy of the GNU General Public License along with thi
 
 詳しいバージョン履歴は [CHANGELOG.md](CHANGELOG.md) を参照してください。
 
-### 現在のバージョン：1.3.2
-- `MultiOutputScript (Jev)` がスクリプトを変えずに質問をまとめて送るように：同じ `state` の質問は 1 回のリクエストで送り、`state` が違うリクエストは同時に送る
-- `max_jev_calls` が質問数ではなくリクエスト数を制限するように
-- 実行ごとに Jev のリクエスト数、`state` 数、質問数、返信数をコンソールに出力
-- TypeSafe SDK と HTTP クライアントのリクエストごとの INFO ログを非表示に
-- `datetime.datetime.now()` と `datetime.date.today()` が実行を開始した時刻を返すように
+### 現在のバージョン：1.3.5
+- `MultiOutputScript (Jev)` が、ローカルの d1 の GGUF モデル（Liquid AI のオープンウェイトの判断モデル）を動かせるように。d1 が学習したプロンプト形式で質問する
+- Jev、Clef-Flash、汎用のローカルモデルが、答えを知りようのない問いにどう答えるかを調べた技術レポートを追加

README.md

@@ -1,7 +1,7 @@
 # ComfyUI-ScriptFlow
 [en | [ja](README.ja.md)]
 
-**Version:** 1.3.2
+**Version:** 1.3.5
 **License:** GPL-3.0
 
 Safe Python-like script node for ComfyUI with a System One decision model.
@@ -53,6 +53,8 @@ The `MultiOutputScript` and `centi` nodes need no extra packages. `MultiOutputSc
 
 - Install [llama-cpp-python](https://github.com/JamePeng/llama-cpp-python) in the Python environment that runs ComfyUI. Tested with the JamePeng fork and the upstream [abetlen/llama-cpp-python](https://github.com/abetlen/llama-cpp-python) 0.3.35.
 - Put an instruction-tuned GGUF model in `ComfyUI/models/LLM` (or `models/text_encoders`). The GGUF must include a chat template (`tokenizer.chat_template` metadata), as most instruction-tuned GGUFs do. Tested with Qwen3.5-9B Q8_0 and Gemma-4-E4B Q8.
+- To use [Clef](https://huggingface.co/Cloudflare/clef-flash), Cloudflare's open-weight decision model, put a Clef GGUF in the llama.cpp native format, such as [ggml-org/Clef-Flash-GGUF](https://huggingface.co/ggml-org/Clef-Flash-GGUF), in the same folder. It needs the JamePeng fork of llama-cpp-python 0.4.2 or later. A Clef GGUF converted as an ordinary chat model is treated as a general model and does not use Clef's decision head. Tested with Clef-Flash Q8_0 on Windows (Intel Arc, SYCL build); Linux and macOS are untested.
+- To use [d1](https://huggingface.co/LiquidAI/d1-3B), Liquid AI's open-weight decision model, put a d1 GGUF, such as [LiquidAI/d1-3B-GGUF](https://huggingface.co/LiquidAI/d1-3B-GGUF), in the same folder. It needs a llama-cpp-python build that loads LFM2 models. The `mmproj` file is not needed. Tested with d1-3B Q8_0 and the JamePeng fork of llama-cpp-python 0.4.2 on Windows (Intel Arc, SYCL build); Linux and macOS are untested.
 
 ## Key Features
 - Multiple inputs: freely combine numeric and text inputs.
@@ -122,8 +124,10 @@ Same inputs, outputs, and script rules as `MultiOutputScript`, plus a `jev` name
 
 - **TypeSafe API** (primary): sends the questions to Jev over the internet.
 - **Local GGUF model** (fallback): for when the Jev API is not available, or when text must stay on your machine. It approximates Jev-style decisions with a general instruction-tuned model, reading the answer from next-token probabilities of lettered options in one forward pass, following [SemIf (OpenJev)](https://github.com/TheoLeeCJ/SemIf-OpenJev). No text is generated and nothing is sent over the network. Answers and probabilities differ from Jev's.
+- **Local Clef model**: [Clef](https://huggingface.co/Cloudflare/clef-flash) is an open-weight decision model with the same question types as Jev. A GGUF whose `general.architecture` metadata is `clef` is treated as a Clef model. Selecting one uses the model's decision head instead of the approximation above, and answers every question about one `state` in one forward pass: the `state` and the questions are laid out in Clef's input format, and the score the decision head gives each option is turned into probabilities with a softmax per question. No text is generated and nothing is sent over the network. Answers and probabilities differ from Jev's.
+- **Local d1 model**: [d1](https://huggingface.co/LiquidAI/d1-3B) is an open-weight decision model with the same question types as Jev. A GGUF whose `lfm2.decision.type` metadata is `lfm2-d1` is treated as a d1 model. Selecting one asks each question in the prompt format d1 was tuned on instead of the lettered options above, and reads the answer from the next-token probabilities of the reply d1 was tuned to give: yes or no for `jev.noul`, an option code for `jev.choice`, and a digit for `jev.score`. No text is generated and nothing is sent over the network. Answers and probabilities differ from Jev's.
 
-Scripts are the same for both. See [Installation](#installation) for the setup of each backend.
+Scripts are the same for all of them. See [Installation](#installation) for the setup of each backend.
 
 Extra inputs:
 - `model`: `TypeSafe API`, or a local GGUF model (`mmproj-*` files are hidden).
@@ -132,9 +136,9 @@ Extra inputs:
 Functions (`state` is a string, dict, or list; text only):
 - `jev.yes(state, question[, threshold])` → `bool` (yes-probability ≥ `threshold`, default `0.5`)
 - `jev.noul(state, question)` → yes-probability (`0.0`–`1.0`). Compare it with a threshold; using it directly as a condition raises an error.
-- `jev.choice(state, question, options)` → the most likely option (`str`). `options` is a list of labels, or a dict of labels to descriptions (2–16 options for local models).
+- `jev.choice(state, question, options)` → the most likely option (`str`). `options` is a list of labels, or a dict of labels to descriptions (2–16 options for general local models).
 - `jev.probabilities(state, question, options)` → `dict` of each option to its probability (sums to `1.0`). Same `options` as `jev.choice`; asking both with the same arguments asks the model once.
-- `jev.score(state, question, levels)` → expected level (`float`, `0` to `len(levels) - 1`) for ordered level descriptions (2–16 levels for local models)
+- `jev.score(state, question, levels)` → expected level (`float`, `0` to `len(levels) - 1`) for ordered level descriptions (2–16 levels for general local models)
 
 ```python
 # it1: a prompt text
@@ -193,8 +197,12 @@ client.system_one(state=it1, questions={"q0": Noul(instructions="q1"), "q1": Nou
 
 Notes:
 - The TypeSafe API sends `state` and questions over the internet. Do not use it with text you cannot share externally.
-- A local question takes one forward pass (about 1–1.5 s with a 4B–9B model on GPU). The model is loaded on the first question of each run (several seconds) and unloaded when the run ends to free VRAM for the rest of the workflow.
+- A general local model takes one forward pass per question (about 1–1.5 s with a 4B–9B model on GPU). A Clef model takes one forward pass per request, for all of its questions (about 3 s for a 300-token request with Clef-Flash Q8_0 on an Intel Arc integrated GPU). A d1 model takes one forward pass per question (about 0.5 s with d1-3B Q8_0 on the same GPU). The model is loaded on the first question of each run (several seconds) and unloaded when the run ends to free VRAM for the rest of the workflow.
 - Local probabilities are uncalibrated and depend on the model; test thresholds on your own inputs.
+- In tests with Qwen3.5-9B, a general local model answered "no" when the text did not settle the question, so a low `jev.noul` value can mean "cannot say it is true" rather than "unlikely". To tell the two apart, ask about both the statement and its negation, or have the model pick a percentage level with `jev.score`. See [How general local models answer uncertain questions](docs/local-model-uncertainty/README.md) for the measurements and example scripts.
+- A local prompt is limited to 8,192 tokens: the `state` and one question for a general model, or the `state` and all questions of a request for Clef. Clef is text only.
+- With a d1 model, `jev.choice` and `jev.probabilities` take 2 to 26 options and `jev.score` takes 2 to 10 levels. Images are not supported.
+- In tests with d1-3B, the probabilities of similar options were spread out, so a single different option could come first: in the aspect ratio example, three portrait ratios shared 0.49 and `1:1` was chosen with 0.35. When options fall into groups, add up `jev.probabilities` by group before picking one.
 - Each run prints `[ComfyUI-ScriptFlow] Jev requests: N, states: S, questions: Q, responses: M` to the console: requests sent, distinct states, questions asked, and requests answered. Fewer responses than requests means a request failed. Questions answered from the run's cache are not counted.
 - When `typesafe-sdk` is installed, the `httpx2` (the SDK's HTTP client) and `typesafe_sdk` loggers are set to WARNING, so per-request INFO lines are not shown.
 - `MultiOutputScript` never loads a model or makes network requests; `jev` is not available there.
@@ -457,6 +465,300 @@ Notes:
 
 </details>
 
+### Check a prompt enhancer's output against its system prompt
+
+<details>
+<summary>Show prompt enhancer rule check recipe</summary>
+
+A prompt enhancer is an LLM that rewrites a short request into a prompt for an image or video model,
+following the rules in its system prompt. This recipe splits that system prompt into rules, asks Jev
+whether the enhancer's output breaks each one, and reports a score and the rules that are likely broken.
+It reads the rules from the system prompt you connect, so it is not tied to one enhancer.
+Paste the script below into the `code` field of `MultiOutputScript (Jev)`.
+
+Recommended workflow:
+
+```text
+enhancer output            -> MultiOutputScript (Jev).in_text_1
+system prompt              -> MultiOutputScript (Jev).in_text_2
+user request (optional)    -> MultiOutputScript (Jev).in_text_3
+MultiOutputScript (Jev)
+  -> out_text_1   report
+  -> out_text_2   likely violations only
+  -> out_text_3   rules extracted from the system prompt
+  -> out_value_1  adherence score (0-100)
+  -> out_value_2  number of likely violations
+```
+
+Paste this code into `MultiOutputScript (Jev).code`:
+
+```python
+# Score how well a prompt enhancer's output follows its system prompt.
+#
+# Input:
+#   it1: the enhancer's output
+#   it2: the system prompt, as plain text or as the text of a JSON config
+#        that has a "system_prompt" key
+#   it3: the user's request (optional; used by rules that refer to it)
+#
+# Output:
+#   ot1: report, most likely violations first
+#   ot2: likely violations only
+#   ot3: the rules extracted from the system prompt
+#   ov1: adherence score (0-100)
+#   ov2: number of likely violations
+#
+# Jev requests: 2 (one for all rules, one for the overall rating),
+# or 1 with OVERALL = False.
+
+MAX_RULES = 40          # rules beyond this are not checked
+THRESHOLD = 0.5         # a rule counts as broken at this probability or above
+SKIP_HEADINGS = ["example", "sample", "thinking", "reasoning", "例", "思考", "手順"]   # sections whose heading has one of these are skipped
+OVERALL = True          # also rate the output against the whole system prompt
+
+def has_any(s, words):
+    for w in words:
+        if w in s:
+            return True
+    return False
+
+output = str(it1 or "").strip()
+sp = str(it2 or "")
+user_text = str(it3 or "").strip()
+
+if sp.strip().startswith("{"):
+    k = sp.find('"system_prompt"')
+    if k < 0:
+        raise ValueError("The JSON has no system_prompt key")
+    tail = sp[sp.find('"', sp.find(":", k) + 1) + 1:]
+    tail = tail.replace("\\\\", "\x00").replace('\\"', "\x01")
+    sp = tail[:tail.find('"')].replace("\\n", "\n").replace("\\t", "\t").replace("\x01", '"').replace("\x00", "\\")
+
+# Under each heading, a line that starts at the left margin is one rule.
+# Indented lines belong to the rule above them.
+rules = []
+heading = ""
+skip = False
+for line in sp.splitlines():
+    s = line.strip()
+    if not s:
+        continue
+    if s[0] in "【#[-":
+        if s[:3] == "---":
+            continue
+        if s[0] != "-" and (s[0] != "[" or s[-1] == "]"):
+            heading = s
+            skip = has_any(s.lower(), SKIP_HEADINGS)
+            continue
+    if skip:
+        continue
+    if rules and line[:1] in " \t　":
+        rules[-1][1] = rules[-1][1] + "\n" + s
+    else:
+        rules.append([heading, s])
+
+truncated = len(rules) > MAX_RULES
+rules = rules[:MAX_RULES]
+
+# Every rule shares one state and goes in the question text,
+# so all rules are asked in one request.
+state = {"output": output}
+if user_text:
+    state["user_input"] = user_text
+results = []
+for r in rules:
+    q = "Does the output clearly break the instruction below? Answer no if the instruction does not concern the output, or if judging it needs a reference image or anything else not given here."
+    if r[0]:
+        q += "\nSection: " + r[0]
+    p = jev.noul(state, q + "\nInstruction: " + r[1])
+    results.append([float(p), r[0], r[1]])
+
+# The overall rating needs the whole system prompt as state, so it is a second request.
+# Asking it before the sort below sends it together with the rules.
+if OVERALL:
+    overall_state = {"system_prompt": sp, "output": output}
+    if user_text:
+        overall_state["user_input"] = user_text
+    level = jev.score(overall_state, "How faithfully does the output follow the system prompt as a whole?", ["ignores most instructions", "follows some instructions", "follows about half", "follows most instructions", "follows every instruction"])
+
+ranked = list(reversed(sorted(results)))
+lines = []
+issues = []
+total = 0.0
+for r in ranked:
+    total += r[0]
+    bad = r[0] >= THRESHOLD
+    line = ("[BREAK?] " if bad else "[OK]     ") + f"p={r[0]:.2f} {r[1]} " + r[2].splitlines()[0][:70]
+    lines.append(line)
+    if bad:
+        issues.append(line)
+
+score = round(100 * (1 - total / len(ranked))) if ranked else 0
+header = f"Adherence: {score}/100 ({len(issues)} of {len(ranked)} rules likely broken)"
+if truncated:
+    header += f"\nOnly the first {MAX_RULES} rules were checked"
+if OVERALL:
+    header += f"\nOverall: {level:.2f} / 4"
+
+ot1 = header + "\n" + "\n".join(lines)
+ot2 = header + "\n" + "\n".join(issues)
+rule_lines = []
+for i, r in enumerate(rules):
+    rule_lines.append(f"{i + 1}. {r[0]} {r[1]}")
+ot3 = "\n\n".join(rule_lines)
+ov1 = score
+ov2 = len(issues)
+```
+
+The resulting `out_text_1` is a report like this:
+
+```text
+Adherence: 71/100 (1 of 4 rules likely broken)
+Overall: 3.10 / 4
+[BREAK?] p=0.74 ## Output format - Do not write sentences.
+[OK]     p=0.21  You are a prompt enhancer for SDXL.
+[OK]     p=0.12 ## Output format - Start with "masterpiece, best quality".
+[OK]     p=0.09 ## Output format - Output a single line of comma-separated tags.
+```
+
+Notes:
+
+- Requires v1.3.2 or later, where questions about the same `state` share one request. The script makes 2 requests, so set `max_jev_calls` to 2 or more.
+- `p` is Jev's probability that the output breaks the rule. It is a judgment, not a check: in testing it flagged some rules that were followed and missed some that were broken, and values near `THRESHOLD` changed between runs. Treat the flagged rules as candidates to review.
+- The recipe was developed with Jev through the TypeSafe API. In testing, a local Clef-Flash model flagged most rules, including rules the output followed, so it is not recommended for this recipe.
+- A rule that needs something the script does not have, such as a reference image, cannot be judged. Connect the user request to `in_text_3` for rules that refer to it.
+- `in_text_2` takes the system prompt as plain text, or the text of a JSON config with a `"system_prompt"` key. `\uXXXX` escapes in the JSON are not decoded.
+- Lines starting with `#`, `【`, or `[...]` are headings. Under a heading, each line at the left margin is one rule, and indented lines belong to the rule above. Check `out_text_3` to see how your system prompt was split.
+- Sections whose heading contains a word in `SKIP_HEADINGS` are skipped, so examples and thinking steps are not judged as rules.
+- Only the first `MAX_RULES` rules are checked. A very long system prompt exceeds the script step limit; see [Long system prompts](#long-system-prompts) below.
+- With the TypeSafe API, the system prompt, the enhancer output, and the user request are sent over the internet.
+- `out_text_3` and the numeric outputs are optional; `out_value_3` is unused.
+
+#### Long system prompts
+
+The script above checks the system prompt one line at a time, so a system prompt of several hundred lines stops with `Script exceeded step limit`.
+For those, use the script below. It cuts the system prompt into parts of about `PART_CHARS` characters at blank lines,
+and asks Jev whether the output breaks any instruction in each part. Connect the inputs and outputs the same way.
+
+```python
+# Score how well a prompt enhancer's output follows a long system prompt.
+# The system prompt is cut into parts of about PART_CHARS characters at blank lines,
+# and Jev is asked whether the output breaks any instruction in each part.
+#
+# Input:
+#   it1: the enhancer's output
+#   it2: the system prompt, as plain text or as the text of a JSON config
+#        that has a "system_prompt" key
+#   it3: the user's request (optional; used by instructions that refer to it)
+#
+# Output:
+#   ot1: report, most likely violations first
+#   ot2: likely violations only
+#   ot3: the parts the system prompt was cut into
+#   ov1: adherence score (0-100)
+#   ov2: number of parts with a likely violation
+#
+# Jev requests: 2 (one for all parts, one for the overall rating),
+# or 1 with OVERALL = False.
+
+PART_CHARS = 1000       # approximate size of one part; smaller parts mean more questions
+MAX_PARTS = 60          # parts beyond this are not checked
+THRESHOLD = 0.5         # a part counts as broken at this probability or above
+OVERALL = True          # also rate the output against the whole system prompt
+
+output = str(it1 or "").strip()
+sp = str(it2 or "")
+user_text = str(it3 or "").strip()
+
+if sp.strip().startswith("{"):
+    k = sp.find('"system_prompt"')
+    if k < 0:
+        raise ValueError("The JSON has no system_prompt key")
+    tail = sp[sp.find('"', sp.find(":", k) + 1) + 1:]
+    tail = tail.replace("\\\\", "\x00").replace('\\"', "\x01")
+    sp = tail[:tail.find('"')].replace("\\n", "\n").replace("\\t", "\t").replace("\x01", '"').replace("\x00", "\\")
+sp = sp.replace("\r\n", "\n").strip()
+
+# Cut at the first blank line after PART_CHARS characters, so no paragraph is split.
+parts = []
+pos = 0
+while pos < len(sp) and len(parts) < MAX_PARTS:
+    end = sp.find("\n\n", pos + PART_CHARS)
+    if end < 0:
+        end = len(sp)
+    parts.append(sp[pos:end].strip())
+    pos = end
+truncated = pos < len(sp)
+
+# Every part shares one state and goes in the question text,
+# so all parts are asked in one request.
+state = {"output": output}
+if user_text:
+    state["user_input"] = user_text
+results = []
+for i, part in enumerate(parts):
+    p = jev.noul(state, "Does the output clearly break any instruction in the part of the system prompt below? Answer no if the instructions do not concern the output, or if judging them needs a reference image or anything else not given here.\nPart of the system prompt:\n" + part)
+    results.append([float(p), i + 1])
+
+# The overall rating needs the whole system prompt as state, so it is a second request.
+# Asking it before the sort below sends it together with the parts.
+if OVERALL:
+    overall_state = {"system_prompt": sp, "output": output}
+    if user_text:
+        overall_state["user_input"] = user_text
+    level = jev.score(overall_state, "How faithfully does the output follow the system prompt as a whole?", ["ignores most instructions", "follows some instructions", "follows about half", "follows most instructions", "follows every instruction"])
+
+lines = []
+issues = []
+total = 0.0
+for r in reversed(sorted(results)):
+    total += r[0]
+    bad = r[0] >= THRESHOLD
+    # Show the first line of the part as its title, skipping divider lines.
+    title = parts[r[1] - 1].strip("=-_*# \n\t").splitlines()[0][:60]
+    line = ("[BREAK?] " if bad else "[OK]     ") + f"p={r[0]:.2f} part {r[1]}: {title}"
+    lines.append(line)
+    if bad:
+        issues.append(line)
+
+score = round(100 * (1 - total / len(parts))) if parts else 0
+header = f"Adherence: {score}/100 ({len(issues)} of {len(parts)} parts likely broken)"
+if truncated:
+    header += f"\nOnly the first {MAX_PARTS} parts were checked"
+if OVERALL:
+    header += f"\nOverall: {level:.2f} / 4"
+
+ot1 = header + "\n" + "\n".join(lines)
+ot2 = header + "\n" + "\n".join(issues)
+part_texts = []
+for i, part in enumerate(parts):
+    part_texts.append(f"--- part {i + 1} ---\n{part}")
+ot3 = "\n\n".join(part_texts)
+ov1 = score
+ov2 = len(issues)
+```
+
+The resulting `out_text_1` is a report like this:
+
+```text
+Adherence: 74/100 (2 of 19 parts likely broken)
+Overall: 2.90 / 4
+[BREAK?] p=0.78 part 7: Keep the events in the order the user gave them.
+[BREAK?] p=0.61 part 12: Describe the camera only when the user asks for it.
+[OK]     p=0.34 part 1: You are a prompt enhancer for a video model.
+[OK]     p=0.22 part 3: Use the requested duration.
+```
+
+Notes:
+
+- The notes above about `max_jev_calls`, `p`, `in_text_2`, and the TypeSafe API apply here too.
+- A part is cut at the first blank line after `PART_CHARS` characters, so no paragraph is split. Headings and bullets are not interpreted, so this works with any layout, but example sections are not skipped.
+- The report names each part by its first line. Read the full text of a flagged part in `out_text_3` to find the instruction in question.
+- One question covers several instructions, so the result is coarser than the line-by-line script. Lower `PART_CHARS` for smaller parts and more questions.
+- In testing, a system prompt of about 20,000 characters was cut into 19 parts and ran in about 2,200 script steps, well under the step limit.
+
+</details>
+
 ## License
 
 This project is licensed under the **GNU General Public License v3.0**.
@@ -482,9 +784,6 @@ You should have received a copy of the GNU General Public License along with thi
 
 See [CHANGELOG.md](CHANGELOG.md) for detailed version history.
 
-### Current Version: 1.3.2
-- `MultiOutputScript (Jev)` batches questions without script changes: questions about the same `state` are sent in one request, and requests for different states are sent in parallel
-- `max_jev_calls` now limits requests instead of questions
-- Added a console summary of Jev requests, states, questions, and responses after each run
-- Hid the per-request INFO log lines of the TypeSafe SDK and its HTTP client
-- `datetime.datetime.now()` and `datetime.date.today()` return the time the run started
+### Current Version: 1.3.5
+- `MultiOutputScript (Jev)` runs local d1 GGUF models (Liquid AI's open-weight decision model), asking each question in the prompt format d1 was tuned on
+- Added a technical report on how Jev, Clef-Flash, and a general local model answer questions whose answer cannot be known

__init__.py

@@ -17,11 +17,13 @@ from typing import Any, Dict
 
 from .jev_backend import (
     UNANSWERED,
+    ClefJev,
     JevBackend,
     LocalJev,
     NeedAnswers,
     TypeSafeClient,
     TypeSafeJev,
+    is_clef_model,
     list_local_models,
     unload_local_model,
 )
@@ -941,8 +943,9 @@ class MultiOutputScriptJev(MultiOutputScript):
         model_path = list_local_models().get(model)
         if model_path is None:
             raise ValueError(f"GGUF model not found under models/LLM or models/text_encoders: {model}")
+        engine = ClefJev(model_path) if is_clef_model(model_path) else LocalJev(model_path)
         try:
-            return _run_jev_script(code, JevBackend(LocalJev(model_path), max_jev_calls, False), inputs)
+            return _run_jev_script(code, JevBackend(engine, max_jev_calls, False), inputs)
         finally:
             unload_local_model()
 

docs/local-model-uncertainty/README.ja.md

@@ -0,0 +1,384 @@
+# 汎用のローカルモデルは、不確かな問いにどう答えるか：`jev.*` の挙動と聞き方の指針
+[[en](README.md) | ja]
+
+技術報告（2026年10月）
+
+## 要約
+
+`MultiOutputScript (Jev)` は、TypeSafe の Jev の代わりに、ローカルの GGUF モデルで判断を行える。この報告は、答えのわからない問いに対して、Jev、Clef-Flash、汎用モデル（Qwen3.5-9B）がどう答えるかを比べ、汎用モデルを使うときの聞き方を調べたものである。
+
+- 答えが確実な問いでは、3つのモデルの答えはほぼ一致した。
+- 答えが不確実な問いでは、振る舞いが分かれた。五分五分の問いに `jev.noul` で「真ですか？」と聞くと、平均は Jev が 0.44、Clef-Flash が 0.31、Qwen3.5-9B が 0.06 だった。
+- Qwen3.5-9B の低い値は、「起こる確率が低い」ではなく「真だとは言えない」を表している。確率が 0.92 の主張にも 0.02 と答えた。
+- 主張とその否定形の両方について聞き、2つの答えを合わせることで、Qwen3.5-9B でも五分五分の問いを 0.51 にすることができた。
+- 確率の段階を選ばせる（`jev.score`）と、確率の高さがある程度取り出せた。結果は、段階の作り方と並び順に左右された。
+- 「偽ですか？」と質問を反対向きにする方法は、否定形の代わりにならなかった。
+
+31組62文の小さな試験であり、段階で聞く方法は Qwen3.5-9B の1つの量子化でしか試していない。わかるのは傾向である。
+
+## 1. 目的
+
+判断モデルの出力は、しきい値で区切って分岐に使うことが多い（`jev.yes(state, question, 0.5)` など）。このとき、判断材料が足りない入力が混ざると、モデルが「わからない」をどの値で表すかによって、分岐の結果が変わる。
+
+この報告の目的は、次の2点を確かめることである。
+
+1. 汎用のローカルモデルは、不確かな問いに対して、Jev とどう違う答え方をするか。
+2. その違いを、スクリプトの書き方（聞き方）でどこまで埋められるか。
+
+確率で答えるモデルに期待する性質として、次の2つを基準にした。
+
+- **基準1**：対になる主張の確率を足すと、1 になる。
+- **基準2**：五分五分の問いには、0.5 と答える。
+
+## 2. 方法
+
+### 2.1 モデル
+
+| モデル | 種類 | 答えの出し方 |
+| --- | --- | --- |
+| Jev | TypeSafe の判断モデル（API） | API が確率を返す |
+| Clef-Flash | Qwen3.5-9B を判断用に追加学習したモデル（`ggml-org/Clef-Flash-GGUF`、Q8_0） | 判断専用の層が出す値を確率に直す |
+| Qwen3.5-9B | 汎用の LLM（`unsloth/Qwen3.5-9B-GGUF`、Q8_0） | 選択肢の記号が次に出る確率を読む |
+
+汎用モデルでは、選択肢に記号を付けて示し（`jev.noul` なら A：Yes、B：No）、記号1文字だけで答えるよう指示する。文章は生成させず、次のトークンとして各記号が出る確率を読み、（記号以外の出力は除き）記号どうしの合計が 1 になるように直す。この方式は SemIf (OpenJev) で提案されているものである。この報告の値は、較正をしていない値である。較正については、3.2 節と 5 節で触れる。なお、サンプリングをしないので、乱数のシードや温度の設定はない。
+
+指示文は次のとおりである。主張、質問、選択肢は、JSON にまとめて渡す。
+
+```text
+Apply the supplied criterion to the supplied evidence. Choose exactly one listed option. Respond with only its uppercase letter, with no explanation or reasoning.
+```
+
+3つとも、このノードの判定処理（v1.3.3）をそのまま使った。質問は1問ずつ送った。
+
+### 2.2 設問
+
+対になる主張を31組、合計62文用意した。主張と質問は英語である。一覧は [`data/items.json`](data/items.json) にある。
+
+| グループ | 記号 | 組数 | 内容 | 例 |
+| --- | --- | --- | --- | --- |
+| 既知の事実 | K | 6 | 1つめが真、2つめが偽 | Paris is the capital of France. / ... of Germany. |
+| ほぼ確実な未来 | N | 6 | 1つめがほぼ確実に起こる | The sun will rise tomorrow. / ... will not rise tomorrow. |
+| 傾向のある未来 | T | 6 | 1つめが起こりやすいが、確実ではない | The S&P 500 index will be higher one year from now ... / lower ... |
+| 五分五分の問い | U | 8 | どちらも約50% | The next coin I flip will land heads. / ... tails. |
+| 確率のわかっている問い | P | 5 | 正しい確率が計算でわかる | The next roll of a fair six-sided die will show a 1.（1/6） / ... a number other than 1.（5/6） |
+
+K、N、T、U の26組は、3つのモデルすべてに聞いた。P の5組は、あとから Qwen3.5-9B だけに聞いた。
+
+### 2.3 質問
+
+| 記号 | 関数 | 質問 | 選択肢 |
+| --- | --- | --- | --- |
+| Q1 | `jev.noul` | Is this statement true? | Yes / No |
+| Q2 | `jev.noul` | Is this statement more likely to be true than false? | Yes / No |
+| F1 | `jev.noul` | Is this statement false? | Yes / No |
+| S6 | `jev.score` | How likely is this statement to be true? | 0-10%、10-30%、30-50%、50-70%、70-90%、90-100% |
+| S11 | `jev.score` | How likely is this statement to be true? | 0%、10%、…、100% |
+| F6 | `jev.score` | How likely is this statement to be false? | S6 と同じ6段階 |
+
+Q1 と Q2 は3つのモデルに聞いた。F1、S6、S11、F6 は Qwen3.5-9B だけに聞いた。段階で聞く質問は、選択肢を低い順に並べた場合と、高い順に並べた場合の両方で聞いた。
+
+対照として、Qwen3.5-9B に確率を数字で書かせた（20文）。指示は「What is the probability that the following statement is true? Answer with a single number from 0 to 100 (percent) and nothing else.」で、最も確率の高いトークンを順に選んで生成した。
+
+### 2.4 指標
+
+- **対の合計**：対の2つの主張への値の合計（基準1）。
+- **0.5 からのずれ**：五分五分の問い16文での、値と 0.5 の差の絶対値の平均（基準2）。
+- **誤差の平均**：確率のわかっている問い10文での、値と正しい確率の差の絶対値の平均。
+- **順位相関**：同じ10文での、値と正しい確率のスピアマンの順位相関。
+- **否定形と合わせた値**：（主張への値 ＋ 1 − 否定形への値）÷ 2。対の合計が当然 1 になる。
+- **段階からの推定値**：各段階の値（S6 と F6 は段階の中央の値である 0.05、0.20、0.40、0.60、0.80、0.95、S11 はその点の値である 0、0.1、…、1.0）を、モデルが各段階に付けた確率で重み付けした値。
+
+否定形と合わせた値は、新しく測ったものではなく、測定済みの値から計算したものである。
+
+## 3. 結果
+
+### 3.1 3つのモデルの比較（Q1）
+
+値は Yes の確率の平均で、左が対の1つめ、右が2つめの主張である。特に、太字の斜体で示した Qwen3.5-9B の「傾向のある未来」と「五分五分の問い」の結果は、1 節の基準1・基準2 と乖離しており、3.2 節以降でさらに検証を行う。
+
+| グループ | Jev | Clef-Flash | Qwen3.5-9B |
+| --- | --- | --- | --- |
+| 既知の事実 | 0.92 / 0.03 | 0.95 / 0.02 | 0.99 / 0.01 |
+| ほぼ確実な未来 | 0.88 / 0.07 | 0.86 / 0.11 | 0.85 / 0.13 |
+| 傾向のある未来 | 0.58 / 0.37 | 0.62 / 0.36 | ***0.07 / 0.08*** |
+| 五分五分の問い | 0.44 / 0.44 | 0.36 / 0.26 | ***0.07 / 0.04*** |
+
+| 指標 | Jev | Clef-Flash | Qwen3.5-9B |
+| --- | --- | --- | --- |
+| 対の合計：既知の事実 | 0.95 | 0.97 | 1.00 |
+| 対の合計：ほぼ確実な未来 | 0.95 | 0.96 | 0.98 |
+| 対の合計：傾向のある未来 | 0.95 | 0.98 | ***0.15*** |
+| 対の合計：五分五分の問い | 0.88 | 0.62 | ***0.11*** |
+| 0.5 からのずれ（五分五分の問い） | 0.06 | 0.22 | ***0.44*** |
+
+- 既知の事実は、3つとも12文すべてに正しい側の値を返した。
+- Jev は、五分五分の問い16文すべてに 0.34〜0.51 を返した。
+- Qwen3.5-9B は、不確かな主張（傾向のある未来、五分五分の問い）のほぼすべてに No と答えた。未来のことをすべて No とするわけではなく、「明日、太陽は昇る」（ほぼ確実な未来）には 0.99 を返している。
+- Clef-Flash は、傾向のある未来では対の合計が 0.98 で、調整前の Qwen3.5-9B（0.15）と大きく違う。コインやサイコロのような問いでは 0.62 にとどまった（コイン投げは 0.16 / 0.10）。
+
+Clef-Flash と Qwen3.5-9B では、答えの出し方も違う。2つの差が、重みの調整によるものか、答えの出し方によるものかは、この試験では分けられない。
+
+Clef-Flash は、傾向のある未来で片方をはっきり選んだ（S&P 500 は 0.80 / 0.06。Jev は 0.53 / 0.31）。ただし、「来年の今日、東京で雨が降らない / 降る」には 0.09 / 0.81 と、実際の頻度とは逆向きの値を返し、「次の米大統領は就任時に50歳より上 / 下」には 0.81 / 0.71 と、両方に Yes と答えた。
+
+### 3.2 Qwen3.5-9B の No は、何を表しているか
+
+ここから、Qwen3.5-9B の「傾向のある未来」と「五分五分の問い」の結果について検証する。
+
+まず確認したのは、Qwen3.5-9B の No が **記号以外のトークンに確率が流れた結果ではない** ということである。語彙全体の確率のうち、A と B が占める割合（SemIf の `allowed_token_mass`）を、62文 × Q1・Q2 の124問で記録した。
+
+| | 最小 | 中央 | 最大 |
+| --- | --- | --- | --- |
+| A と B が占める割合 | 0.9971 | 0.9992 | 0.9999 |
+
+A・B 以外で最も確率が高かったトークンは「C」で、最大でも 0.15% だった。断りや説明の書き出しにあたるトークンは、上位10個に現れなかった。モデルは、B（No）をはっきり選んでいる。
+
+**「わからない」と「ほぼ確実に起こらない」が、同じ値になる。** A と B のロジットの差（正なら Yes 寄り）の平均は、次のとおりである。
+
+| 主張の種類 | ロジットの差 | 確率に直すと |
+| --- | --- | --- |
+| 真の事実 | ＋5.6 | 約 1.00 |
+| ほぼ確実に起こる未来 | ＋2.9 | 約 0.95 |
+| 傾向のある未来 | −3.0 | 約 0.05 |
+| 五分五分の問い | −3.3 | 約 0.04 |
+| ほぼ確実に起こらない未来 | −3.3 | 約 0.04 |
+| 偽の事実 | −6.2 | 約 0.002 |
+
+**値は、確率の高さの順に並ばない。** 確率のわかっている問いへの Q1 の答えは、次のとおりである。10文での順位相関は −0.38 だった。
+
+| 主張 | 正しい確率 | Q1 の値 |
+| --- | --- | --- |
+| トランプでエースを引く | 0.08 | 0.03 |
+| サイコロで1が出る | 0.17 | 0.02 |
+| トランプでハートを引く | 0.25 | 0.03 |
+| サイコロで3〜6が出る | 0.67 | 0.35 |
+| トランプでハート以外を引く | 0.75 | 0.02 |
+| サイコロで1以外が出る | 0.83 | 0.01 |
+| トランプでエース以外を引く | 0.92 | 0.02 |
+
+Qwen3.5-9B は、「真ですか？」に対して、「確実に真だと言えるか」という基準で答えていると考えられる。出てくる値は、出来事の確率ではなく、No という答えへの自信である。値の中に確率の高さの情報がないので、Q1 の値を後から直して確率にすること（較正）は難しい。
+
+### 3.3 質問文を変える（Q2）
+
+質問を「偽であるより真である可能性のほうが高いですか？」に変えると、次のようになった。
+
+- Jev と Clef-Flash は、Q1 とほぼ同じ傾向だった。
+- Qwen3.5-9B は、「すべて No」ではなくなった。確率のわかっている問いでは、10文中8文で正しい側（0.5 より上か下か）を選んだ。
+- ただし、Qwen3.5-9B の値は 0 か 1 に近く、確率の程度を表さない。五分五分の問いは、問いごとに Yes か No のどちらかに振れた（0.02〜0.99、0.5 からのずれは 0.36）。
+
+汎用モデルの答えは、質問の言い回しに強く左右される。
+
+### 3.4 主張と否定形の両方について聞く
+
+Qwen3.5-9B は、Q1 で「ほぼ確実に起こる」には Yes、「わからない」には No と答える。区別できないのは、「わからない」と「ほぼ確実に起こらない」である。後者の否定形は「ほぼ確実に起こる」なので、否定形についても聞けば区別できる。
+
+| 主張の種類 | 主張への答え | 否定形への答え |
+| --- | --- | --- |
+| ほぼ確実に起こる | Yes | No |
+| ほぼ確実に起こらない | No | Yes |
+| わからない | No | No |
+
+設問は対で作ってあるので、対の相手を否定形とみなして、否定形と合わせた値を計算した（Q1）。かっこ内は、主張だけを聞いたときの値である。太字の斜体は、3.1 節で基準から外れていた項目である。
+
+| グループ | Jev | Clef-Flash | Qwen3.5-9B |
+| --- | --- | --- | --- |
+| 既知の事実 | 0.95（0.92） | 0.96（0.95） | 0.99（0.99） |
+| ほぼ確実な未来 | 0.90（0.88） | 0.88（0.86） | 0.86（0.85） |
+| 傾向のある未来 | 0.60（0.58） | 0.63（0.62） | ***0.49（0.07）*** |
+| 五分五分の問い | 0.50（0.44） | 0.55（0.36） | ***0.51（0.07）*** |
+
+| 0.5 からのずれ（五分五分の問い） | Jev | Clef-Flash | Qwen3.5-9B |
+| --- | --- | --- | --- |
+| 主張だけ | 0.06 | 0.22 | ***0.44*** |
+| 否定形と合わせる | 0.006 | 0.053 | ***0.015*** |
+
+Qwen3.5-9B について、2つの答えを 0.5 で区切って「真 / 偽 / わからない」に分けると、31組中29組が期待どおりになった。
+
+- 既知の事実と、ほぼ確実な未来（12組）：10組が「真」。
+- 傾向のある未来、五分五分の問い、確率のわかっている問い（19組）：19組すべてが「わからない」。
+
+外れた2組は、「来年1月に札幌で雪が降る / 降らない」（両方に No）と、「皆既日食が5年以内に起こる / 起こらない」（両方に Yes）である。両方に Yes という答えは、その答えが信頼できないことの目印になる。
+
+この方法では、確率の程度は取り出せない。確率が 0.08 の主張も 0.92 の主張も、否定形と合わせた値は 0.5 付近になる（確率のわかっている問いでの誤差の平均は 0.244、順位相関は −0.12）。
+
+### 3.5 確率の段階を選ばせる
+
+以下は、Qwen3.5-9B だけの結果である。
+
+**対照：数字で書かせると、ほぼ正確である。** 確率のわかっている10文での誤差の平均は 0.003 だった（たとえば、正しい確率 7.7% に「9」、16.7% に「16.67」、92.3% に「92」）。コイン投げには「50」と答えた。モデルは確率の知識を持っている。Q1 で確率の高さが出ないのは、聞き方による。
+
+| 主張 | 正しい確率 | 書かれた数字 |
+| --- | --- | --- |
+| トランプでエースを引く | 7.7% | 9 |
+| サイコロで1が出る | 16.7% | 16.67 |
+| トランプでハートを引く | 25% | 25 |
+| コイン2回とも表 | 25% | 25 |
+| サイコロで1か2が出る | 33.3% | 33 |
+| サイコロで3〜6が出る | 66.7% | 66 |
+| トランプでハート以外を引く | 75% | 75 |
+| コイン2回とも表、にはならない | 75% | 75 |
+| サイコロで1以外が出る | 83.3% | 83 |
+| トランプでエース以外を引く | 92.3% | 92 |
+| 五分五分の問い（コイン、サイコロ、生まれた時期、砂粒の4文） | 50% | すべて 50 |
+| 明日、太陽は昇る / 昇らない | — | 100 / 0 |
+| パリはフランスの首都である / ドイツの首都である | — | 100 / 0 |
+| S&P 500 は1年後に高い / 低い | — | 65 / 50 |
+
+**段階で聞くと、確率の高さがある程度出る。** 確率のわかっている10文での結果は、次のとおりである。
+
+| 聞き方 | 順位相関 | 誤差の平均 |
+| --- | --- | --- |
+| Q1（Yes / No） | −0.38 | — |
+| S6、低い順 | ＋0.88 | 0.095 |
+| S6、高い順 | ＋0.76 | 0.208 |
+| S11、低い順 | ＋0.77 | 0.202 |
+| S11、高い順 | ＋0.80 | 0.203 |
+| 数字で書かせる | ＋1.00 | 0.003 |
+
+段階で聞く2通りの質問を、もう一度示す。どちらも質問文は「How likely is this statement to be true?」で、選択肢だけが違う。
+
+- **S6（6段階の幅）**：0-10%、10-30%、30-50%、50-70%、70-90%、90-100%。推定値には、各段階の中央の値（0.05、0.20、0.40、0.60、0.80、0.95）を使う。
+- **S11（11段階の点）**：0%、10%、20%、…、100%。推定値には、その点の値（0、0.1、…、1.0）を使う。
+
+**6段階の幅（S6）と11段階の点（S11）は、得意な点が違う。**
+
+| 指標 | S6 | S11 |
+| --- | --- | --- |
+| 0.5 からのずれ（五分五分の問い、低い順 / 高い順） | 0.118 / 0.140 | 0.027 / 0.029 |
+| 並び順による推定値の差（62文の平均） | 0.087 | 0.042 |
+| 対の合計：既知の事実（並び順2通りの平均） | 1.07 | 1.03 |
+| 対の合計：ほぼ確実な未来（同上） | 1.14 | 1.04 |
+
+- S6 は、五分五分の問いが約 0.60 になった。50% が「30-50%」と「50-70%」の境目にあたり、モデルが16文中14文で「50-70%」を選んだためである。
+- S6 は、並び順に強く左右された。「ハート以外を引く」（正しくは 0.75）は、低い順で 0.75、高い順で 0.24 だった。
+- S11 は、五分五分の問い16文中14文が 0.50 になり、対の合計も 1 に近い。主張だけを聞いても、基準1と基準2をほぼ満たす。
+- S11 は、確率の程度がほとんど出ない。モデルが、確実でない主張のほとんどに「50%」を選ぶためである。確率が 0.25 の「ハートを引く」にも、0.75 の「ハート以外を引く」にも、「50%」を選んだ。
+
+「50%」という選択肢があると、五分五分の問いには正しく答えられるが、確実でない問いがそこに集まる。S11 は、「真 / 偽 / わからない」の3つに分ける方法に近い働きをする。
+
+**6段階の幅で、主張と否定形の両方について聞くと、両方に対応できる。**
+
+| 聞き方（S6） | 0.5 からのずれ | 誤差の平均 | 順位相関 |
+| --- | --- | --- | --- |
+| 低い順、主張だけ | 0.118 | 0.095 | ＋0.88 |
+| 低い順、否定形と合わせる | 0.027 | 0.069 | ＋0.90 |
+| 高い順、否定形と合わせる | 0.034 | 0.148 | ＋0.92 |
+
+五分五分の問いのずれは、主張にも否定形にも同じ向きに出るので、合わせると打ち消し合う。段階は低い順に並べるほうがよい。
+
+| 主張 | 正しい確率 | 主張だけ | 否定形と合わせる |
+| --- | --- | --- | --- |
+| トランプでエースを引く | 0.08 | 0.06 | 0.13 |
+| サイコロで1が出る | 0.17 | 0.05 | 0.19 |
+| トランプでハートを引く | 0.25 | 0.32 | 0.29 |
+| コイン2回とも表 | 0.25 | 0.06 | 0.12 |
+| サイコロで1か2が出る | 0.33 | 0.53 | 0.44 |
+
+### 3.6 「偽ですか？」は、否定形の代わりにならない
+
+否定形を書く代わりに、主張はそのままにして、質問を反対向きにする方法（F1、F6）を試した。合わせた値は、「（Q1 の値 ＋ 1 − F1 の値）÷ 2」で計算した。
+
+| グループ | 否定形の主張と合わせる | F1 と合わせる |
+| --- | --- | --- |
+| 既知の事実 | 0.99 | 0.99 |
+| ほぼ確実な未来 | 0.86 | 0.87 |
+| 傾向のある未来 | 0.49 | 0.36 |
+| 五分五分の問い | 0.51 | 0.42 |
+
+| 指標 | 否定形の主張と合わせる | 「偽」の質問と合わせる |
+| --- | --- | --- |
+| 0.5 からのずれ（Yes / No） | 0.015 | 0.109 |
+| 0.5 からのずれ（6段階、低い順） | 0.027 | 0.072 |
+| 誤差の平均（6段階、低い順） | 0.069 | 0.146 |
+
+確実な問いでは差がない。崩れるのは、不確かな問いである。Qwen3.5-9B は、不確かな主張に「真ですか？」と聞くとほぼ必ず No と答えるが、「偽ですか？」には Yes と答えることがあった（「コインは裏が出る」に、Q1 は 0.03、F1 は 0.60）。不確かな主張を「真だ」とは言わないが、「偽だ」とは言いやすい。
+
+否定形の主張が効くのは、同じ質問文で聞くので、質問文による偏りが両方に同じようにかかり、打ち消し合うためと考えられる。
+
+### 3.7 聞き方のまとめ（Qwen3.5-9B）
+
+| 聞き方 | 五分五分を 0.5 に | 確率の程度 | 否定形を書く | 質問の数 |
+| --- | --- | --- | --- | --- |
+| Yes / No、主張だけ | できない | 出ない | 不要 | 1 |
+| Yes / No、否定形と合わせる | できる | 出ない | 必要 | 2 |
+| 11段階の点、主張だけ | できる | ほぼ出ない | 不要 | 1 |
+| 6段階の幅、主張だけ | ずれる（約 0.60） | ある程度出る | 不要 | 1 |
+| 6段階の幅、否定形と合わせる | できる | ある程度出る | 必要 | 2 |
+| 「偽」の質問と合わせる | ずれる | 出ない | 不要 | 2 |
+| 数字で書かせる | できる | 出る | 不要 | 文章の生成が必要 |
+
+## 4. 汎用モデルを使うときの指針
+
+以上から、汎用のローカルモデルで `jev.*` を使うスクリプトについて、次のように言える。
+
+1. **`jev.noul` や `jev.yes` の低い値を、「起こりにくい」と読まない。** 「真だとは言えない」を含む。判断材料が足りない入力は、ほぼ常に No 側に分岐する。
+2. **「真 / 偽 / わからない」がわかればよいなら、11段階の点で1回聞く。**
+3. **Yes / No で聞くなら、主張と否定形の両方について聞く。** 両方が No なら「わからない」、両方が Yes なら「信頼できない答え」として扱う。
+4. **確率の程度も知りたいなら、6段階の幅を低い順に並べ、主張と否定形の両方について聞く。**
+5. **否定形は、主張として書く。** 質問を「偽ですか？」に変える方法は、代わりにならない。そのため、否定形の質問をノードの側で自動的に作る方法は、いまのところない。
+6. **聞きたいことを、そのまま質問文にする。** 「真か」と「起こりやすいほうか」では、答えが大きく変わる。
+7. **モデルを替えたら、しきい値を見直す。** 同じスクリプトでも、Jev、Clef-Flash、汎用モデルで、不確かな入力の分岐が変わる。
+
+主張と否定形の両方について聞く例。数値の出力（`ov1` など）は整数なので、否定形と合わせた値は 100 倍して 0〜100 で出力する。
+
+```python
+# it1: 主張、it2: その否定形
+# ov1: 否定形と合わせた値（0〜100）、ot1: 分類
+a = jev.noul(it1, "Is this statement true?")
+b = jev.noul(it2, "Is this statement true?")
+ov1 = (a + 1 - b) / 2 * 100
+
+if a < 0.5 and b < 0.5:
+    ot1 = "unsure"
+elif a >= 0.5 and b >= 0.5:
+    ot1 = "contradiction"
+elif a >= 0.5:
+    ot1 = "true"
+else:
+    ot1 = "false"
+```
+
+11段階の点で聞く例。`jev.score` は選ばれた段階の番号の期待値（0〜10）を返すので、10 倍すると 0〜100 の値になる。
+
+```python
+levels = ["0%", "10%", "20%", "30%", "40%", "50%", "60%", "70%", "80%", "90%", "100%"]
+ov1 = jev.score(it1, "How likely is this statement to be true?", levels) * 10
+```
+
+6段階の幅では、段階の中央の値が等間隔でないので、`jev.score` の値を 20 倍しても、3.5 節の推定値（の 100 倍）とは同じにならない。近似として使う場合は、その点に注意する。
+
+## 5. 限界
+
+- 31組62文の小さな試験である。確率のわかっている問いは5組10文だけで、聞き方の比較はこの少ない問いにもとづく。
+- モデルの優劣を評価したものではない。未来の問いには正解がなく、予測の正しさは測っていない。
+- 否定形と合わせる方法は、このデータを見て思いついたもので、同じデータで効果を確かめている。別の設問で確かめてはいない。
+- 設問の対は、厳密な否定になっていないものがある（「高い」と「低い」など）。否定形の作り方で、結果は変わりうる。
+- 段階で聞く方法（S6、S11）、否定形と合わせる方法の段階での効果、「偽」の質問（F1、F6）は、Qwen3.5-9B でしか試していない。Jev と Clef-Flash には聞いていない。
+- 段階の作り方は、2通りしか試していない。
+- 汎用モデルは Qwen3.5-9B の Q8_0 だけである。別のモデル、別の量子化、別の判定方式では、違う結果になりうる。
+- 汎用モデルの値は、較正をしていない。SemIf (OpenJev) は、用途ごとに較正して使うよう勧めているが、今回は較正を試していない。3.2 節のとおり、Yes / No の値には確率の高さの情報がなく、較正に使える設問も用意できていない。
+- どの質問も1回ずつの測定である。ローカルのモデルの値は、サンプリングをしないので、同じ条件なら変わらない。Jev は、同じ質問でも値が少し変わる。五分五分の問い16文を Q1 で3回聞いたところ、値の幅は最大で 0.04 だった。
+- 主張と質問は英語である。ほかの言語では試していない。
+
+## 6. データ
+
+[`data/`](data/) に、設問と結果を置いている。主張の ID は、対の記号に `a`（1つめ）か `b`（2つめ）を付けたものである（例：`U1a`）。
+
+| ファイル | 内容 |
+| --- | --- |
+| `items.json` | 設問（`pairs`）、質問文（`questions`）、グループの説明（`groups`）、確率のわかっている問いの正しい確率（`probabilities`） |
+| `results_jev.json` | Jev。Q1・Q2 の `p_yes`。五分五分の問いの Q1 は、`run` 1〜3 の3回分 |
+| `results_clef_flash.json` | Clef-Flash。Q1・Q2 の `p_yes` |
+| `results_qwen35.json` | Qwen3.5-9B。Q1・Q2 の `p_yes`（26組） |
+| `results_qwen35_mass.json` | Qwen3.5-9B。Q1・Q2（31組）の `p_yes`、A と B のロジット、`allowed_token_mass`、確率の高い上位10トークン |
+| `results_qwen35_score.json` | Qwen3.5-9B。S6 の段階ごとの確率（`score`）と、数字で書かせた対照の答え（`text`） |
+| `results_qwen35_score11.json` | Qwen3.5-9B。S11 の段階ごとの確率 |
+| `results_qwen35_false.json` | Qwen3.5-9B。F1 の `p_yes`（`noul`）と、F6 の段階ごとの確率（`score`） |
+
+段階ごとの確率（`p`）は、選択肢をどちらの順で示した場合も、低い段階から順に並べて保存している。`order` は、示した順である。
+
+## 7. 測定の条件
+
+- 実施日：2026年10月5日（3つのモデルの比較）、10月6日〜8日（Qwen3.5-9B の追加の測定）
+- ノード：ComfyUI-ScriptFlow v1.3.3
+- Jev：TypeSafe API（typesafe-sdk 0.7.1）。API が返すモデル名は、2026年10月8日に確認した時点で `jev-1.13.0` だった。測定時（10月5日）のモデル名は記録していない。
+- ローカルのモデルの実行：llama-cpp-python 0.4.2（SYCL ビルド）、Windows 11

docs/local-model-uncertainty/README.md

@@ -0,0 +1,384 @@
+# How general local models answer uncertain questions: `jev.*` behavior and how to ask
+[en | [ja](README.ja.md)]
+
+Technical report (October 2026)
+
+## Summary
+
+`MultiOutputScript (Jev)` can make decisions with a local GGUF model instead of TypeSafe's Jev. This report compares how Jev, Clef-Flash, and a general model (Qwen3.5-9B) answer questions whose answer cannot be known, and looks at how to ask when a general model is used.
+
+- On questions with a certain answer, the three models gave almost the same answers.
+- On questions with an uncertain answer, they behaved differently. When 50/50 statements were asked "Is this statement true?" with `jev.noul`, the mean was 0.44 for Jev, 0.31 for Clef-Flash, and 0.06 for Qwen3.5-9B.
+- A low value from Qwen3.5-9B means "cannot say it is true", not "unlikely to happen". It answered 0.02 for a statement whose probability is 0.92.
+- Asking about both a statement and its negation, and combining the two answers, brought the 50/50 statements to 0.51 with Qwen3.5-9B as well.
+- Having the model pick a probability level (`jev.score`) recovered the size of the probability to some degree. The result depended on how the levels were defined and on their order.
+- Reversing the question to "Is this statement false?" did not work as a substitute for the negation.
+
+This is a small test of 31 pairs (62 statements), and the level-based questions were tried with one quantization of Qwen3.5-9B only. It shows tendencies.
+
+## 1. Purpose
+
+The output of a decision model is often cut at a threshold and used to branch (for example `jev.yes(state, question, 0.5)`). When some inputs do not give enough to decide, the branch taken depends on which value the model uses to express "I cannot tell".
+
+This report checks two things:
+
+1. How a general local model answers uncertain questions differently from Jev.
+2. How much of that difference can be closed by how the script asks.
+
+Two properties expected of a model that answers with probabilities were used as criteria:
+
+- **Criterion 1**: the probabilities of the two statements of a pair add up to 1.
+- **Criterion 2**: a 50/50 statement gets 0.5.
+
+## 2. Method
+
+### 2.1 Models
+
+| Model | Kind | How the answer is produced |
+| --- | --- | --- |
+| Jev | TypeSafe's decision model (API) | The API returns probabilities |
+| Clef-Flash | Qwen3.5-9B further trained for decisions (`ggml-org/Clef-Flash-GGUF`, Q8_0) | The values from the decision head are turned into probabilities |
+| Qwen3.5-9B | General LLM (`unsloth/Qwen3.5-9B-GGUF`, Q8_0) | The probabilities of the option letters as the next token are read |
+
+For a general model, the options are shown with letters (for `jev.noul`, A: Yes and B: No) and the model is told to answer with one letter only. No text is generated. The probability of each letter as the next token is read, and the values are rescaled so that the letters add up to 1 (any other output is left out). This is the method proposed by SemIf (OpenJev). The values in this report are uncalibrated. Calibration is discussed in sections 3.2 and 5. Because nothing is sampled, there is no random seed or temperature setting.
+
+The instruction is as follows. The statement, the question, and the options are passed together as JSON.
+
+```text
+Apply the supplied criterion to the supplied evidence. Choose exactly one listed option. Respond with only its uppercase letter, with no explanation or reasoning.
+```
+
+All three models were run through this node's own decision code (v1.3.3). Questions were sent one at a time.
+
+### 2.2 Statements
+
+There are 31 pairs of statements, 62 statements in total. Statements and questions are in English. The full list is in [`data/items.json`](data/items.json).
+
+| Group | Code | Pairs | Content | Example |
+| --- | --- | --- | --- | --- |
+| Known facts | K | 6 | The first is true, the second false | Paris is the capital of France. / ... of Germany. |
+| Near-certain future | N | 6 | The first is almost sure to happen | The sun will rise tomorrow. / ... will not rise tomorrow. |
+| Future with a tendency | T | 6 | The first is more likely, but not certain | The S&P 500 index will be higher one year from now ... / lower ... |
+| 50/50 | U | 8 | Both are about 50% | The next coin I flip will land heads. / ... tails. |
+| Known probability | P | 5 | The exact probability can be calculated | The next roll of a fair six-sided die will show a 1. (1/6) / ... a number other than 1. (5/6) |
+
+The 26 pairs of K, N, T, and U were asked to all three models. The 5 pairs of P were added later and asked to Qwen3.5-9B only.
+
+### 2.3 Questions
+
+| Code | Function | Question | Options |
+| --- | --- | --- | --- |
+| Q1 | `jev.noul` | Is this statement true? | Yes / No |
+| Q2 | `jev.noul` | Is this statement more likely to be true than false? | Yes / No |
+| F1 | `jev.noul` | Is this statement false? | Yes / No |
+| S6 | `jev.score` | How likely is this statement to be true? | 0-10%, 10-30%, 30-50%, 50-70%, 70-90%, 90-100% |
+| S11 | `jev.score` | How likely is this statement to be true? | 0%, 10%, ..., 100% |
+| F6 | `jev.score` | How likely is this statement to be false? | The same six levels as S6 |
+
+Q1 and Q2 were asked to all three models. F1, S6, S11, and F6 were asked to Qwen3.5-9B only. The level-based questions were asked twice, with the options in ascending and in descending order.
+
+As a control, Qwen3.5-9B was also asked to write the probability as a number (20 statements). The instruction was "What is the probability that the following statement is true? Answer with a single number from 0 to 100 (percent) and nothing else.", and the text was generated by taking the most likely token at each step.
+
+### 2.4 Measures
+
+- **Pair sum**: the sum of the values for the two statements of a pair (criterion 1).
+- **Distance from 0.5**: the mean absolute difference between the value and 0.5 over the 16 50/50 statements (criterion 2).
+- **Mean error**: the mean absolute difference between the value and the true probability over the 10 known-probability statements.
+- **Rank correlation**: Spearman's rank correlation between the value and the true probability over the same 10 statements.
+- **Negation-combined value**: (value for the statement + 1 − value for the negation) / 2. The pair sum is 1 by construction.
+- **Estimate from levels**: the value of each level (for S6 and F6 the midpoints 0.05, 0.20, 0.40, 0.60, 0.80, 0.95; for S11 the points 0, 0.1, ..., 1.0) weighted by the probability the model gave that level.
+
+Negation-combined values were not measured separately. They are calculated from the values already measured.
+
+## 3. Results
+
+### 3.1 The three models compared (Q1)
+
+Values are mean yes-probabilities, with the first statement of the pair on the left and the second on the right. The results of Qwen3.5-9B for "Future with a tendency" and "50/50", shown in bold italics, are far from criteria 1 and 2 of section 1, and are examined further from section 3.2 on.
+
+| Group | Jev | Clef-Flash | Qwen3.5-9B |
+| --- | --- | --- | --- |
+| Known facts | 0.92 / 0.03 | 0.95 / 0.02 | 0.99 / 0.01 |
+| Near-certain future | 0.88 / 0.07 | 0.86 / 0.11 | 0.85 / 0.13 |
+| Future with a tendency | 0.58 / 0.37 | 0.62 / 0.36 | ***0.07 / 0.08*** |
+| 50/50 | 0.44 / 0.44 | 0.36 / 0.26 | ***0.07 / 0.04*** |
+
+| Measure | Jev | Clef-Flash | Qwen3.5-9B |
+| --- | --- | --- | --- |
+| Pair sum: known facts | 0.95 | 0.97 | 1.00 |
+| Pair sum: near-certain future | 0.95 | 0.96 | 0.98 |
+| Pair sum: future with a tendency | 0.95 | 0.98 | ***0.15*** |
+| Pair sum: 50/50 | 0.88 | 0.62 | ***0.11*** |
+| Distance from 0.5 (50/50) | 0.06 | 0.22 | ***0.44*** |
+
+- For known facts, all three models gave a value on the correct side for all 12 statements.
+- Jev returned 0.34 to 0.51 for all 16 50/50 statements.
+- Qwen3.5-9B answered No to almost every uncertain statement (future with a tendency, 50/50). It does not answer No to everything about the future: it returned 0.99 for "The sun will rise tomorrow." (near-certain future).
+- Clef-Flash had a pair sum of 0.98 for the future with a tendency, very different from Qwen3.5-9B before tuning (0.15). For questions like coins and dice it stayed at 0.62 (the coin flip was 0.16 / 0.10).
+
+Clef-Flash and Qwen3.5-9B also differ in how the answer is produced. This test cannot separate whether the difference between the two comes from the tuned weights or from the way the answer is produced.
+
+Clef-Flash chose one side clearly for the future with a tendency (0.80 / 0.06 for the S&P 500, where Jev gave 0.53 / 0.31). However, it returned 0.09 / 0.81 for "It will not rain / will rain in Tokyo on this day next year", the opposite of the actual frequency, and 0.81 / 0.71 for "The next US president will be older / younger than 50 when taking office", answering Yes to both.
+
+### 3.2 What the No of Qwen3.5-9B represents
+
+From here on, the results of Qwen3.5-9B for "Future with a tendency" and "50/50" are examined.
+
+The first check was that the No of Qwen3.5-9B is **not the result of probability going to tokens other than the letters**. The share of the whole vocabulary's probability taken by A and B (`allowed_token_mass` in SemIf) was recorded for 124 questions: 62 statements × Q1 and Q2.
+
+| | Minimum | Median | Maximum |
+| --- | --- | --- | --- |
+| Share taken by A and B | 0.9971 | 0.9992 | 0.9999 |
+
+The most likely token other than A and B was "C", at 0.15% at most. Tokens that would begin a refusal or an explanation did not appear among the top 10. The model chooses B (No) clearly.
+
+**"Cannot tell" and "almost surely will not happen" get the same value.** The mean difference between the logits of A and B (positive means toward Yes) is as follows.
+
+| Kind of statement | Logit difference | As a probability |
+| --- | --- | --- |
+| True fact | +5.6 | about 1.00 |
+| Near-certain to happen | +2.9 | about 0.95 |
+| Future with a tendency | −3.0 | about 0.05 |
+| 50/50 | −3.3 | about 0.04 |
+| Near-certain not to happen | −3.3 | about 0.04 |
+| False fact | −6.2 | about 0.002 |
+
+**The values do not follow the order of the probabilities.** The Q1 answers for known-probability statements are below. The rank correlation over the 10 statements was −0.38.
+
+| Statement | True probability | Q1 value |
+| --- | --- | --- |
+| Drawing an ace from a deck | 0.08 | 0.03 |
+| Rolling a 1 | 0.17 | 0.02 |
+| Drawing a heart | 0.25 | 0.03 |
+| Rolling a 3, 4, 5, or 6 | 0.67 | 0.35 |
+| Drawing a card that is not a heart | 0.75 | 0.02 |
+| Rolling a number other than 1 | 0.83 | 0.01 |
+| Drawing a card that is not an ace | 0.92 | 0.02 |
+
+Qwen3.5-9B appears to answer "Is this statement true?" by the standard "can it be said to be true for certain?". The value that comes out is not the probability of the event but the confidence in the answer No. Because the value carries no information about the size of the probability, turning Q1 values into probabilities afterwards (calibration) is difficult.
+
+### 3.3 Changing the question (Q2)
+
+With the question changed to "Is this statement more likely to be true than false?":
+
+- Jev and Clef-Flash showed almost the same tendency as with Q1.
+- Qwen3.5-9B no longer answered No to everything. For the known-probability statements it chose the correct side (above or below 0.5) for 8 of 10.
+- However, the values of Qwen3.5-9B were close to 0 or 1 and did not express the size of the probability. The 50/50 statements swung to Yes or No depending on the statement (0.02 to 0.99; distance from 0.5 was 0.36).
+
+The answers of a general model depend strongly on the wording of the question.
+
+### 3.4 Asking about both the statement and its negation
+
+With Q1, Qwen3.5-9B answers Yes to "near-certain to happen" and No to "cannot tell". What it does not separate is "cannot tell" from "near-certain not to happen". The negation of the latter is "near-certain to happen", so asking about the negation as well separates them.
+
+| Kind of statement | Answer for the statement | Answer for the negation |
+| --- | --- | --- |
+| Near-certain to happen | Yes | No |
+| Near-certain not to happen | No | Yes |
+| Cannot tell | No | No |
+
+The statements were written as pairs, so the other statement of the pair was taken as the negation and the negation-combined value was calculated (Q1). The value in parentheses is from asking about the statement alone. Bold italics mark the items that were far from the criteria in section 3.1.
+
+| Group | Jev | Clef-Flash | Qwen3.5-9B |
+| --- | --- | --- | --- |
+| Known facts | 0.95 (0.92) | 0.96 (0.95) | 0.99 (0.99) |
+| Near-certain future | 0.90 (0.88) | 0.88 (0.86) | 0.86 (0.85) |
+| Future with a tendency | 0.60 (0.58) | 0.63 (0.62) | ***0.49 (0.07)*** |
+| 50/50 | 0.50 (0.44) | 0.55 (0.36) | ***0.51 (0.07)*** |
+
+| Distance from 0.5 (50/50) | Jev | Clef-Flash | Qwen3.5-9B |
+| --- | --- | --- | --- |
+| Statement alone | 0.06 | 0.22 | ***0.44*** |
+| Combined with the negation | 0.006 | 0.053 | ***0.015*** |
+
+For Qwen3.5-9B, cutting the two answers at 0.5 and sorting each pair into "true / false / cannot tell" gave the expected result for 29 of 31 pairs.
+
+- Known facts and near-certain future (12 pairs): 10 pairs were "true".
+- Future with a tendency, 50/50, and known probability (19 pairs): all 19 pairs were "cannot tell".
+
+The two pairs that missed were "It will snow / will not snow in Sapporo next January" (No to both) and "A total solar eclipse will / will not occur within the next five years" (Yes to both). A Yes to both is a sign that the answer cannot be trusted.
+
+This method does not recover the size of the probability. A statement with probability 0.08 and one with 0.92 both get a negation-combined value near 0.5 (for the known-probability statements the mean error was 0.244 and the rank correlation −0.12).
+
+### 3.5 Having the model pick a probability level
+
+The results below are for Qwen3.5-9B only.
+
+**Control: written as a number, the answer is almost exact.** The mean error over the 10 known-probability statements was 0.003 (for example "9" for a true probability of 7.7%, "16.67" for 16.7%, and "92" for 92.3%). For the coin flip it answered "50". The model has the knowledge of the probabilities. That the size of the probability does not show with Q1 comes from how the question is asked.
+
+| Statement | True probability | Number written |
+| --- | --- | --- |
+| Drawing an ace from a deck | 7.7% | 9 |
+| Rolling a 1 | 16.7% | 16.67 |
+| Drawing a heart | 25% | 25 |
+| Two heads in two coin flips | 25% | 25 |
+| Rolling a 1 or a 2 | 33.3% | 33 |
+| Rolling a 3, 4, 5, or 6 | 66.7% | 66 |
+| Drawing a card that is not a heart | 75% | 75 |
+| Not two heads in two coin flips | 75% | 75 |
+| Rolling a number other than 1 | 83.3% | 83 |
+| Drawing a card that is not an ace | 92.3% | 92 |
+| 50/50 statements (4: coin, die, time of birth, grains of sand) | 50% | 50 for all |
+| The sun will rise / will not rise tomorrow | — | 100 / 0 |
+| Paris is the capital of France / of Germany | — | 100 / 0 |
+| The S&P 500 will be higher / lower in one year | — | 65 / 50 |
+
+**Asked by level, the size of the probability shows to some degree.** The results for the 10 known-probability statements are as follows.
+
+| How it was asked | Rank correlation | Mean error |
+| --- | --- | --- |
+| Q1 (Yes / No) | −0.38 | — |
+| S6, ascending | +0.88 | 0.095 |
+| S6, descending | +0.76 | 0.208 |
+| S11, ascending | +0.77 | 0.202 |
+| S11, descending | +0.80 | 0.203 |
+| Written as a number | +1.00 | 0.003 |
+
+The two level-based questions are shown again here. Both use the question "How likely is this statement to be true?", and only the options differ.
+
+- **S6 (six ranges)**: 0-10%, 10-30%, 30-50%, 50-70%, 70-90%, 90-100%. The estimate uses the midpoint of each range (0.05, 0.20, 0.40, 0.60, 0.80, 0.95).
+- **S11 (eleven points)**: 0%, 10%, 20%, ..., 100%. The estimate uses the value of each point (0, 0.1, ..., 1.0).
+
+**Six ranges (S6) and eleven points (S11) are good at different things.**
+
+| Measure | S6 | S11 |
+| --- | --- | --- |
+| Distance from 0.5 (50/50; ascending / descending) | 0.118 / 0.140 | 0.027 / 0.029 |
+| Difference in the estimate between the two orders (mean of 62 statements) | 0.087 | 0.042 |
+| Pair sum: known facts (mean of the two orders) | 1.07 | 1.03 |
+| Pair sum: near-certain future (mean of the two orders) | 1.14 | 1.04 |
+
+- With S6, the 50/50 statements came out at about 0.60. 50% lies on the border between "30-50%" and "50-70%", and the model chose "50-70%" for 14 of 16 statements.
+- S6 depended strongly on the order. "Drawing a card that is not a heart" (true probability 0.75) was 0.75 in ascending order and 0.24 in descending order.
+- With S11, 14 of the 16 50/50 statements came out at 0.50, and the pair sums are close to 1. Asking about the statement alone almost meets criteria 1 and 2.
+- With S11, the size of the probability hardly shows, because the model chooses "50%" for most statements that are not certain. It chose "50%" both for "Drawing a heart" (0.25) and for "Drawing a card that is not a heart" (0.75).
+
+When there is a "50%" option, 50/50 statements are answered correctly, but uncertain statements gather there. S11 works much like sorting into "true / false / cannot tell".
+
+**Six ranges asked about both the statement and its negation cover both.**
+
+| How it was asked (S6) | Distance from 0.5 | Mean error | Rank correlation |
+| --- | --- | --- | --- |
+| Ascending, statement alone | 0.118 | 0.095 | +0.88 |
+| Ascending, combined with the negation | 0.027 | 0.069 | +0.90 |
+| Descending, combined with the negation | 0.034 | 0.148 | +0.92 |
+
+The shift of the 50/50 statements goes in the same direction for the statement and for the negation, so it cancels when the two are combined. The levels are better listed in ascending order.
+
+| Statement | True probability | Statement alone | Combined with the negation |
+| --- | --- | --- | --- |
+| Drawing an ace from a deck | 0.08 | 0.06 | 0.13 |
+| Rolling a 1 | 0.17 | 0.05 | 0.19 |
+| Drawing a heart | 0.25 | 0.32 | 0.29 |
+| Two heads in two coin flips | 0.25 | 0.06 | 0.12 |
+| Rolling a 1 or a 2 | 0.33 | 0.53 | 0.44 |
+
+### 3.6 "Is this statement false?" is not a substitute for the negation
+
+Instead of writing a negation, the statement was kept as it is and the question was reversed (F1, F6). The combined value was calculated as (Q1 value + 1 − F1 value) / 2.
+
+| Group | Combined with the negated statement | Combined with F1 |
+| --- | --- | --- |
+| Known facts | 0.99 | 0.99 |
+| Near-certain future | 0.86 | 0.87 |
+| Future with a tendency | 0.49 | 0.36 |
+| 50/50 | 0.51 | 0.42 |
+
+| Measure | Combined with the negated statement | Combined with the "false" question |
+| --- | --- | --- |
+| Distance from 0.5 (Yes / No) | 0.015 | 0.109 |
+| Distance from 0.5 (six ranges, ascending) | 0.027 | 0.072 |
+| Mean error (six ranges, ascending) | 0.069 | 0.146 |
+
+There is no difference for certain statements. What breaks is the uncertain ones. Asked "Is this statement true?" about an uncertain statement, Qwen3.5-9B almost always answers No, but asked "Is this statement false?" it sometimes answered Yes (for "The next coin I flip will land tails.", Q1 was 0.03 and F1 was 0.60). It does not say an uncertain statement is "true", but it says "false" more easily.
+
+The negated statement likely works because it is asked with the same question, so the bias of the question applies to both in the same way and cancels.
+
+### 3.7 Summary of the ways of asking (Qwen3.5-9B)
+
+| How it was asked | 50/50 at 0.5 | Size of the probability | Writing a negation | Questions |
+| --- | --- | --- | --- | --- |
+| Yes / No, statement alone | No | Does not show | Not needed | 1 |
+| Yes / No, combined with the negation | Yes | Does not show | Needed | 2 |
+| Eleven points, statement alone | Yes | Hardly shows | Not needed | 1 |
+| Six ranges, statement alone | Shifted (about 0.60) | Shows to some degree | Not needed | 1 |
+| Six ranges, combined with the negation | Yes | Shows to some degree | Needed | 2 |
+| Combined with the "false" question | Shifted | Does not show | Not needed | 2 |
+| Written as a number | Yes | Shows | Not needed | Needs text generation |
+
+## 4. Guidelines for using a general model
+
+From the above, the following can be said for scripts that use `jev.*` with a general local model.
+
+1. **Do not read a low `jev.noul` or `jev.yes` value as "unlikely".** It includes "cannot say it is true". Inputs that do not give enough to decide almost always take the No branch.
+2. **If "true / false / cannot tell" is enough, ask once with eleven points.**
+3. **If you ask Yes / No, ask about both the statement and its negation.** Treat No to both as "cannot tell", and Yes to both as an answer that cannot be trusted.
+4. **If you also want the size of the probability, list six ranges in ascending order and ask about both the statement and its negation.**
+5. **Write the negation as a statement.** Changing the question to "Is this statement false?" is not a substitute. For this reason, there is currently no way for the node to create the negated question automatically.
+6. **Put what you want to know directly into the question.** "Is it true" and "is it the more likely side" give very different answers.
+7. **Review thresholds when you change the model.** With the same script, the branch taken for uncertain inputs differs between Jev, Clef-Flash, and a general model.
+
+Asking about both the statement and its negation. Numeric outputs (`ov1` and so on) are integers, so the negation-combined value is multiplied by 100 and output as 0 to 100.
+
+```python
+# it1: the statement, it2: its negation
+# ov1: negation-combined value (0-100), ot1: classification
+a = jev.noul(it1, "Is this statement true?")
+b = jev.noul(it2, "Is this statement true?")
+ov1 = (a + 1 - b) / 2 * 100
+
+if a < 0.5 and b < 0.5:
+    ot1 = "unsure"
+elif a >= 0.5 and b >= 0.5:
+    ot1 = "contradiction"
+elif a >= 0.5:
+    ot1 = "true"
+else:
+    ot1 = "false"
+```
+
+Asking with eleven points. `jev.score` returns the expected index of the chosen level (0 to 10), so multiplying by 10 gives a value from 0 to 100.
+
+```python
+levels = ["0%", "10%", "20%", "30%", "40%", "50%", "60%", "70%", "80%", "90%", "100%"]
+ov1 = jev.score(it1, "How likely is this statement to be true?", levels) * 10
+```
+
+With six ranges, the midpoints are not evenly spaced, so multiplying the `jev.score` value by 20 does not give the same value as 100 times the estimate in section 3.5. Keep this in mind when using it as an approximation.
+
+## 5. Limitations
+
+- This is a small test of 31 pairs (62 statements). There are only 5 known-probability pairs (10 statements), and the comparison of the ways of asking rests on these few statements.
+- It does not rank the models. Questions about the future have no correct answer, and the accuracy of the predictions was not measured.
+- Combining with the negation was thought of while looking at this data, and its effect was checked on the same data. It has not been checked on other statements.
+- Some pairs are not strict negations (such as "higher" and "lower"). The result can change with how the negation is written.
+- The level-based questions (S6, S11), the effect of combining with the negation for levels, and the "false" questions (F1, F6) were tried with Qwen3.5-9B only. They were not asked to Jev or Clef-Flash.
+- Only two ways of defining the levels were tried.
+- The only general model is Qwen3.5-9B at Q8_0. Another model, another quantization, or another decision method may give different results.
+- The values of the general model are uncalibrated. SemIf (OpenJev) recommends calibrating for each use, but calibration was not tried here. As section 3.2 shows, the Yes / No values carry no information about the size of the probability, and no statements suitable for calibration have been prepared.
+- Every question was measured once. The values of a local model do not change under the same conditions, because nothing is sampled. Jev's values vary a little for the same question: when the 16 50/50 statements were asked three times with Q1, the largest range was 0.04.
+- Statements and questions are in English. Other languages were not tried.
+
+## 6. Data
+
+The statements and the results are in [`data/`](data/). A statement ID is the pair code followed by `a` (first statement) or `b` (second statement), for example `U1a`.
+
+| File | Content |
+| --- | --- |
+| `items.json` | Statements (`pairs`), question texts (`questions`), group descriptions (`groups`), and the true probabilities of the known-probability statements (`probabilities`) |
+| `results_jev.json` | Jev. `p_yes` for Q1 and Q2. Q1 of the 50/50 statements has three runs (`run` 1 to 3) |
+| `results_clef_flash.json` | Clef-Flash. `p_yes` for Q1 and Q2 |
+| `results_qwen35.json` | Qwen3.5-9B. `p_yes` for Q1 and Q2 (26 pairs) |
+| `results_qwen35_mass.json` | Qwen3.5-9B. For Q1 and Q2 (31 pairs): `p_yes`, the logits of A and B, `allowed_token_mass`, and the 10 most likely tokens |
+| `results_qwen35_score.json` | Qwen3.5-9B. The probability of each level for S6 (`score`) and the answers of the written-number control (`text`) |
+| `results_qwen35_score11.json` | Qwen3.5-9B. The probability of each level for S11 |
+| `results_qwen35_false.json` | Qwen3.5-9B. `p_yes` for F1 (`noul`) and the probability of each level for F6 (`score`) |
+
+The level probabilities (`p`) are stored from the lowest level to the highest, whichever order the options were shown in. `order` is the order shown.
+
+## 7. Measurement conditions
+
+- Dates: October 5, 2026 (comparison of the three models), October 6 to 8, 2026 (additional measurements with Qwen3.5-9B)
+- Node: ComfyUI-ScriptFlow v1.3.3
+- Jev: TypeSafe API (typesafe-sdk 0.7.1). The model name returned by the API was `jev-1.13.0` when checked on October 8, 2026. The model name at the time of measurement (October 5) was not recorded.
+- Local models: llama-cpp-python 0.4.2 (SYCL build), Windows 11

docs/local-model-uncertainty/data/items.json

@@ -0,0 +1,182 @@
+{
+ "questions": {
+  "Q1": "Is this statement true?",
+  "Q2": "Is this statement more likely to be true than false?"
+ },
+ "groups": {
+  "U": "Unknowable, symmetric: each statement of a pair should be about 50%",
+  "T": "Future with a tendency: one statement of a pair is more likely, but not certain",
+  "N": "Future, near-certain: the first statement of a pair is almost sure",
+  "K": "Known facts (control): the first statement of a pair is true, the second false",
+  "P": "Known probability: the statement has an exact probability, listed under probabilities"
+ },
+ "pairs": [
+  [
+   "U1",
+   "The next coin I flip will land heads.",
+   "The next coin I flip will land tails."
+  ],
+  [
+   "U2",
+   "The next roll of a fair six-sided die will show an even number.",
+   "The next roll of a fair six-sided die will show an odd number."
+  ],
+  [
+   "U3",
+   "The next card I draw from a well-shuffled standard deck will be red.",
+   "The next card I draw from a well-shuffled standard deck will be black."
+  ],
+  [
+   "U4",
+   "A fair spinner with two equal halves labelled A and B will land on A on its next spin.",
+   "A fair spinner with two equal halves labelled A and B will land on B on its next spin."
+  ],
+  [
+   "U5",
+   "The last digit of the next winning lottery number will be even.",
+   "The last digit of the next winning lottery number will be odd."
+  ],
+  [
+   "U6",
+   "A person chosen at random from the world's population was born in the first half of the year.",
+   "A person chosen at random from the world's population was born in the second half of the year."
+  ],
+  [
+   "U7",
+   "The number of grains of sand on the nearest beach is even.",
+   "The number of grains of sand on the nearest beach is odd."
+  ],
+  [
+   "U8",
+   "The next ball drawn from a bag holding five white and five black balls will be white.",
+   "The next ball drawn from a bag holding five white and five black balls will be black."
+  ],
+  [
+   "T1",
+   "The S&P 500 index will be higher one year from now than it is today.",
+   "The S&P 500 index will be lower one year from now than it is today."
+  ],
+  [
+   "T2",
+   "The price of Bitcoin will be higher one year from now than it is today.",
+   "The price of Bitcoin will be lower one year from now than it is today."
+  ],
+  [
+   "T3",
+   "It will not rain in Tokyo on this day next year.",
+   "It will rain in Tokyo on this day next year."
+  ],
+  [
+   "T4",
+   "The next FIFA World Cup will be won by a European team.",
+   "The next FIFA World Cup will be won by a team from outside Europe."
+  ],
+  [
+   "T5",
+   "The men's 100 metres world record will not be broken within the next year.",
+   "The men's 100 metres world record will be broken within the next year."
+  ],
+  [
+   "T6",
+   "The next person elected president of the United States will be older than 50 when taking office.",
+   "The next person elected president of the United States will be younger than 50 when taking office."
+  ],
+  [
+   "N1",
+   "The sun will rise tomorrow.",
+   "The sun will not rise tomorrow."
+  ],
+  [
+   "N2",
+   "The world's population will be larger one year from now than it is today.",
+   "The world's population will be smaller one year from now than it is today."
+  ],
+  [
+   "N3",
+   "An earthquake of magnitude 6 or greater will occur somewhere in the world during the next year.",
+   "No earthquake of magnitude 6 or greater will occur anywhere in the world during the next year."
+  ],
+  [
+   "N4",
+   "It will snow in Sapporo at some point next January.",
+   "It will not snow in Sapporo at any point next January."
+  ],
+  [
+   "N5",
+   "Next year will have 365 or 366 days.",
+   "Next year will have 400 days."
+  ],
+  [
+   "N6",
+   "The next total solar eclipse visible from Earth will occur within the next five years.",
+   "No total solar eclipse will be visible from Earth during the next five years."
+  ],
+  [
+   "K1",
+   "Paris is the capital of France.",
+   "Paris is the capital of Germany."
+  ],
+  [
+   "K2",
+   "Water boils at 100 degrees Celsius at sea level.",
+   "Water boils at 50 degrees Celsius at sea level."
+  ],
+  [
+   "K3",
+   "In April 1986, a reactor at the Chernobyl nuclear power plant exploded, and people were killed.",
+   "No one was killed in the accident at the Chernobyl nuclear power plant in April 1986."
+  ],
+  [
+   "K4",
+   "The potato blight of the 1840s led to a famine in Ireland in which about one million people died.",
+   "No famine occurred in Ireland during the 1840s."
+  ],
+  [
+   "K5",
+   "A 1964 report by the US Surgeon General found that smoking causes lung cancer.",
+   "The 1964 report by the US Surgeon General on smoking found no evidence that smoking harms health."
+  ],
+  [
+   "K6",
+   "The Earth orbits the Sun.",
+   "The Sun orbits the Earth."
+  ],
+  [
+   "P1",
+   "The next roll of a fair six-sided die will show a 1.",
+   "The next roll of a fair six-sided die will show a number other than 1."
+  ],
+  [
+   "P2",
+   "The next card I draw from a well-shuffled standard deck will be a heart.",
+   "The next card I draw from a well-shuffled standard deck will not be a heart."
+  ],
+  [
+   "P3",
+   "The next card I draw from a well-shuffled standard deck will be an ace.",
+   "The next card I draw from a well-shuffled standard deck will not be an ace."
+  ],
+  [
+   "P4",
+   "The next roll of a fair six-sided die will show a 1 or a 2.",
+   "The next roll of a fair six-sided die will show a 3, 4, 5, or 6."
+  ],
+  [
+   "P5",
+   "The next two flips of a fair coin will both land heads.",
+   "The next two flips of a fair coin will not both land heads."
+  ]
+ ],
+ "probabilities": {
+  "P1a": 0.1667,
+  "P1b": 0.8333,
+  "P2a": 0.25,
+  "P2b": 0.75,
+  "P3a": 0.0769,
+  "P3b": 0.9231,
+  "P4a": 0.3333,
+  "P4b": 0.6667,
+  "P5a": 0.25,
+  "P5b": 0.75
+ }
+}

docs/local-model-uncertainty/data/results_clef_flash.json

@@ -0,0 +1,626 @@
+[
+{
+"id": "U1a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.1563546622523346
+},
+{
+"id": "U1b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.10298498607890458
+},
+{
+"id": "U2a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.27531802064465766
+},
+{
+"id": "U2b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.29130223232773783
+},
+{
+"id": "U3a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.596886783643548
+},
+{
+"id": "U3b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.3328182831349319
+},
+{
+"id": "U4a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.4135071008052303
+},
+{
+"id": "U4b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.25353891261846645
+},
+{
+"id": "U5a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.3597190663096565
+},
+{
+"id": "U5b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.3477136050354511
+},
+{
+"id": "U6a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.6314669640969492
+},
+{
+"id": "U6b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.3741538921045834
+},
+{
+"id": "U7a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.31978838456545733
+},
+{
+"id": "U7b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.2546097220836936
+},
+{
+"id": "U8a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.15263059327321835
+},
+{
+"id": "U8b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.13091564626756208
+},
+{
+"id": "T1a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.803736698937114
+},
+{
+"id": "T1b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.056274844577240885
+},
+{
+"id": "T2a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.3678579990777839
+},
+{
+"id": "T2b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.37329297956124446
+},
+{
+"id": "T3a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.08927474404407679
+},
+{
+"id": "T3b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.8137230093194368
+},
+{
+"id": "T4a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.790477390492118
+},
+{
+"id": "T4b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.08047192678047338
+},
+{
+"id": "T5a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.8520500870846421
+},
+{
+"id": "T5b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.13444005790013824
+},
+{
+"id": "T6a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.8111855757100763
+},
+{
+"id": "T6b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.7050209843722118
+},
+{
+"id": "N1a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.9356080934390207
+},
+{
+"id": "N1b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.032338866863635074
+},
+{
+"id": "N2a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.9559256289426079
+},
+{
+"id": "N2b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.016569419950235302
+},
+{
+"id": "N3a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.908919745901481
+},
+{
+"id": "N3b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.026298370171733743
+},
+{
+"id": "N4a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.9392960484183409
+},
+{
+"id": "N4b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.02327075306277671
+},
+{
+"id": "N5a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.9076664209994167
+},
+{
+"id": "N5b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.035031695022713166
+},
+{
+"id": "N6a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.49885577163170747
+},
+{
+"id": "N6b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.5087399222304279
+},
+{
+"id": "K1a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.9774759389069352
+},
+{
+"id": "K1b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.012810059886409647
+},
+{
+"id": "K2a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.9532845343591053
+},
+{
+"id": "K2b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.025426653952945435
+},
+{
+"id": "K3a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.9683758009733633
+},
+{
+"id": "K3b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.034370633898392955
+},
+{
+"id": "K4a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.8849612771088546
+},
+{
+"id": "K4b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.018067256873252214
+},
+{
+"id": "K5a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.9283748177780613
+},
+{
+"id": "K5b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.031002559742295537
+},
+{
+"id": "K6a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.9763295039553002
+},
+{
+"id": "K6b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.022602225223762408
+},
+{
+"id": "U1a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.24064914137892165
+},
+{
+"id": "U1b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.1549050607030875
+},
+{
+"id": "U2a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.7114077534383975
+},
+{
+"id": "U2b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.6654411859926993
+},
+{
+"id": "U3a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.6207051703754343
+},
+{
+"id": "U3b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.5670477720065128
+},
+{
+"id": "U4a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.6319509842446309
+},
+{
+"id": "U4b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.6359247882034883
+},
+{
+"id": "U5a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.5442487038788544
+},
+{
+"id": "U5b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.3274244257434919
+},
+{
+"id": "U6a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.728263732364755
+},
+{
+"id": "U6b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.4342110118229748
+},
+{
+"id": "U7a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.27081585373424377
+},
+{
+"id": "U7b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.18296151484483048
+},
+{
+"id": "U8a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.47336041183055105
+},
+{
+"id": "U8b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.30528242405046324
+},
+{
+"id": "T1a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.8882269104060471
+},
+{
+"id": "T1b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.04396967521678573
+},
+{
+"id": "T2a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.3013701517706645
+},
+{
+"id": "T2b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.32712378007033704
+},
+{
+"id": "T3a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.08464050383877865
+},
+{
+"id": "T3b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.8699422179938108
+},
+{
+"id": "T4a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.8827551196528663
+},
+{
+"id": "T4b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.05284938359932525
+},
+{
+"id": "T5a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.899715889250761
+},
+{
+"id": "T5b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.1015997172090742
+},
+{
+"id": "T6a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.8614926406864682
+},
+{
+"id": "T6b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.7546528239764947
+},
+{
+"id": "N1a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9389274955639704
+},
+{
+"id": "N1b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.02653183601637322
+},
+{
+"id": "N2a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9424760623004863
+},
+{
+"id": "N2b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.022103155252441102
+},
+{
+"id": "N3a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.921112184919618
+},
+{
+"id": "N3b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.033456409653188746
+},
+{
+"id": "N4a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9354006228802139
+},
+{
+"id": "N4b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.027246941111788263
+},
+{
+"id": "N5a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.8264708812070534
+},
+{
+"id": "N5b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.03676608855142873
+},
+{
+"id": "N6a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.48855409348700174
+},
+{
+"id": "N6b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.5130744566996128
+},
+{
+"id": "K1a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9696001534468888
+},
+{
+"id": "K1b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.017463957778993393
+},
+{
+"id": "K2a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9438410895630032
+},
+{
+"id": "K2b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.04037044384111732
+},
+{
+"id": "K3a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9591383366948559
+},
+{
+"id": "K3b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.0316748786508726
+},
+{
+"id": "K4a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.8854235311543147
+},
+{
+"id": "K4b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.02242497296914693
+},
+{
+"id": "K5a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.941702509009343
+},
+{
+"id": "K5b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.026032544145669488
+},
+{
+"id": "K6a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9674552732131715
+},
+{
+"id": "K6b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.022730758403467992
+}
+]

docs/local-model-uncertainty/data/results_jev.json

@@ -0,0 +1,818 @@
+[
+{
+"id": "U1a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.46
+},
+{
+"id": "U1b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.44
+},
+{
+"id": "U2a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.44
+},
+{
+"id": "U2b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.45
+},
+{
+"id": "U3a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.46
+},
+{
+"id": "U3b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.45
+},
+{
+"id": "U4a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.34
+},
+{
+"id": "U4b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.37
+},
+{
+"id": "U5a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.5
+},
+{
+"id": "U5b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.51
+},
+{
+"id": "U6a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.49
+},
+{
+"id": "U6b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.49
+},
+{
+"id": "U7a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.45
+},
+{
+"id": "U7b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.46
+},
+{
+"id": "U8a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.38
+},
+{
+"id": "U8b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.37
+},
+{
+"id": "T1a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.53
+},
+{
+"id": "T1b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.31
+},
+{
+"id": "T2a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.49
+},
+{
+"id": "T2b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.39
+},
+{
+"id": "T3a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.62
+},
+{
+"id": "T3b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.53
+},
+{
+"id": "T4a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.49
+},
+{
+"id": "T4b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.5
+},
+{
+"id": "T5a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.57
+},
+{
+"id": "T5b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.31
+},
+{
+"id": "T6a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.76
+},
+{
+"id": "T6b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.19
+},
+{
+"id": "N1a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.91
+},
+{
+"id": "N1b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.04
+},
+{
+"id": "N2a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.86
+},
+{
+"id": "N2b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.09
+},
+{
+"id": "N3a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.95
+},
+{
+"id": "N3b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.04
+},
+{
+"id": "N4a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.89
+},
+{
+"id": "N4b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.08
+},
+{
+"id": "N5a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.97
+},
+{
+"id": "N5b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.06
+},
+{
+"id": "N6a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.69
+},
+{
+"id": "N6b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.13
+},
+{
+"id": "K1a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.98
+},
+{
+"id": "K1b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.01
+},
+{
+"id": "K2a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.96
+},
+{
+"id": "K2b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.01
+},
+{
+"id": "K3a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.89
+},
+{
+"id": "K3b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.09
+},
+{
+"id": "K4a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.87
+},
+{
+"id": "K4b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.01
+},
+{
+"id": "K5a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.84
+},
+{
+"id": "K5b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.03
+},
+{
+"id": "K6a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.98
+},
+{
+"id": "K6b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.01
+},
+{
+"id": "U1a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.32
+},
+{
+"id": "U1b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.28
+},
+{
+"id": "U2a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.21
+},
+{
+"id": "U2b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.26
+},
+{
+"id": "U3a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.29
+},
+{
+"id": "U3b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.29
+},
+{
+"id": "U4a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.31
+},
+{
+"id": "U4b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.28
+},
+{
+"id": "U5a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.48
+},
+{
+"id": "U5b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.49
+},
+{
+"id": "U6a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.46
+},
+{
+"id": "U6b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.41
+},
+{
+"id": "U7a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.51
+},
+{
+"id": "U7b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.54
+},
+{
+"id": "U8a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.36
+},
+{
+"id": "U8b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.36
+},
+{
+"id": "T1a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.57
+},
+{
+"id": "T1b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.33
+},
+{
+"id": "T2a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.5
+},
+{
+"id": "T2b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.4
+},
+{
+"id": "T3a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.7
+},
+{
+"id": "T3b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.47
+},
+{
+"id": "T4a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.52
+},
+{
+"id": "T4b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.48
+},
+{
+"id": "T5a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.61
+},
+{
+"id": "T5b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.32
+},
+{
+"id": "T6a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.82
+},
+{
+"id": "T6b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.18
+},
+{
+"id": "N1a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.97
+},
+{
+"id": "N1b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.03
+},
+{
+"id": "N2a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.89
+},
+{
+"id": "N2b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.09
+},
+{
+"id": "N3a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.97
+},
+{
+"id": "N3b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.04
+},
+{
+"id": "N4a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.96
+},
+{
+"id": "N4b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.06
+},
+{
+"id": "N5a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.98
+},
+{
+"id": "N5b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.07
+},
+{
+"id": "N6a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.74
+},
+{
+"id": "N6b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.17
+},
+{
+"id": "K1a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.99
+},
+{
+"id": "K1b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.01
+},
+{
+"id": "K2a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.98
+},
+{
+"id": "K2b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.02
+},
+{
+"id": "K3a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.95
+},
+{
+"id": "K3b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.2
+},
+{
+"id": "K4a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.88
+},
+{
+"id": "K4b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.02
+},
+{
+"id": "K5a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.94
+},
+{
+"id": "K5b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.03
+},
+{
+"id": "K6a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.99
+},
+{
+"id": "K6b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.02
+},
+{
+"id": "U1a",
+"q": "Q1",
+"run": 2,
+"p_yes": 0.46
+},
+{
+"id": "U1b",
+"q": "Q1",
+"run": 2,
+"p_yes": 0.43
+},
+{
+"id": "U2a",
+"q": "Q1",
+"run": 2,
+"p_yes": 0.44
+},
+{
+"id": "U2b",
+"q": "Q1",
+"run": 2,
+"p_yes": 0.46
+},
+{
+"id": "U3a",
+"q": "Q1",
+"run": 2,
+"p_yes": 0.46
+},
+{
+"id": "U3b",
+"q": "Q1",
+"run": 2,
+"p_yes": 0.45
+},
+{
+"id": "U4a",
+"q": "Q1",
+"run": 2,
+"p_yes": 0.33
+},
+{
+"id": "U4b",
+"q": "Q1",
+"run": 2,
+"p_yes": 0.36
+},
+{
+"id": "U5a",
+"q": "Q1",
+"run": 2,
+"p_yes": 0.51
+},
+{
+"id": "U5b",
+"q": "Q1",
+"run": 2,
+"p_yes": 0.5
+},
+{
+"id": "U6a",
+"q": "Q1",
+"run": 2,
+"p_yes": 0.47
+},
+{
+"id": "U6b",
+"q": "Q1",
+"run": 2,
+"p_yes": 0.49
+},
+{
+"id": "U7a",
+"q": "Q1",
+"run": 2,
+"p_yes": 0.41
+},
+{
+"id": "U7b",
+"q": "Q1",
+"run": 2,
+"p_yes": 0.46
+},
+{
+"id": "U8a",
+"q": "Q1",
+"run": 2,
+"p_yes": 0.38
+},
+{
+"id": "U8b",
+"q": "Q1",
+"run": 2,
+"p_yes": 0.35
+},
+{
+"id": "U1a",
+"q": "Q1",
+"run": 3,
+"p_yes": 0.47
+},
+{
+"id": "U1b",
+"q": "Q1",
+"run": 3,
+"p_yes": 0.44
+},
+{
+"id": "U2a",
+"q": "Q1",
+"run": 3,
+"p_yes": 0.44
+},
+{
+"id": "U2b",
+"q": "Q1",
+"run": 3,
+"p_yes": 0.44
+},
+{
+"id": "U3a",
+"q": "Q1",
+"run": 3,
+"p_yes": 0.45
+},
+{
+"id": "U3b",
+"q": "Q1",
+"run": 3,
+"p_yes": 0.45
+},
+{
+"id": "U4a",
+"q": "Q1",
+"run": 3,
+"p_yes": 0.34
+},
+{
+"id": "U4b",
+"q": "Q1",
+"run": 3,
+"p_yes": 0.36
+},
+{
+"id": "U5a",
+"q": "Q1",
+"run": 3,
+"p_yes": 0.51
+},
+{
+"id": "U5b",
+"q": "Q1",
+"run": 3,
+"p_yes": 0.5
+},
+{
+"id": "U6a",
+"q": "Q1",
+"run": 3,
+"p_yes": 0.49
+},
+{
+"id": "U6b",
+"q": "Q1",
+"run": 3,
+"p_yes": 0.5
+},
+{
+"id": "U7a",
+"q": "Q1",
+"run": 3,
+"p_yes": 0.45
+},
+{
+"id": "U7b",
+"q": "Q1",
+"run": 3,
+"p_yes": 0.47
+},
+{
+"id": "U8a",
+"q": "Q1",
+"run": 3,
+"p_yes": 0.37
+},
+{
+"id": "U8b",
+"q": "Q1",
+"run": 3,
+"p_yes": 0.36
+}
+]

docs/local-model-uncertainty/data/results_qwen35.json

@@ -0,0 +1,626 @@
+[
+{
+"id": "U1a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.04293589372706846
+},
+{
+"id": "U1b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.029438944319826802
+},
+{
+"id": "U2a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.012875717935652332
+},
+{
+"id": "U2b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.015392259115030355
+},
+{
+"id": "U3a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.04062919162805074
+},
+{
+"id": "U3b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.02698182131824937
+},
+{
+"id": "U4a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.00848915157377681
+},
+{
+"id": "U4b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.017836672013940468
+},
+{
+"id": "U5a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.0243622668628212
+},
+{
+"id": "U5b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.024844143004769465
+},
+{
+"id": "U6a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.28326839905182954
+},
+{
+"id": "U6b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.10557380048752749
+},
+{
+"id": "U7a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.10408252745000653
+},
+{
+"id": "U7b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.11869719610087492
+},
+{
+"id": "U8a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.015351063883996655
+},
+{
+"id": "U8b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.016817785729793713
+},
+{
+"id": "T1a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.022057347437745733
+},
+{
+"id": "T1b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.021683283520289053
+},
+{
+"id": "T2a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.025373198109493362
+},
+{
+"id": "T2b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.027161821769643532
+},
+{
+"id": "T3a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.03047279286820563
+},
+{
+"id": "T3b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.035284298468013904
+},
+{
+"id": "T4a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.08476684431458896
+},
+{
+"id": "T4b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.343042922540492
+},
+{
+"id": "T5a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.12621128760152997
+},
+{
+"id": "T5b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.021762038393210035
+},
+{
+"id": "T6a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.10113807328739997
+},
+{
+"id": "T6b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.049815864450550015
+},
+{
+"id": "N1a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.9923621500632038
+},
+{
+"id": "N1b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.015732907919588063
+},
+{
+"id": "N2a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.9824827957993699
+},
+{
+"id": "N2b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.0075136385868251965
+},
+{
+"id": "N3a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.868971045455636
+},
+{
+"id": "N3b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.0381296967843491
+},
+{
+"id": "N4a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.30421726759545153
+},
+{
+"id": "N4b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.04969070102240233
+},
+{
+"id": "N5a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.9886396837495554
+},
+{
+"id": "N5b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.004316507429744988
+},
+{
+"id": "N6a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.9430432488085654
+},
+{
+"id": "N6b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.6918752290584557
+},
+{
+"id": "K1a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.9983674985865608
+},
+{
+"id": "K1b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.001096284565803176
+},
+{
+"id": "K2a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.9968625419382113
+},
+{
+"id": "K2b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.0007492928963817307
+},
+{
+"id": "K3a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.9973426193681626
+},
+{
+"id": "K3b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.00564343885975838
+},
+{
+"id": "K4a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.9799021310424756
+},
+{
+"id": "K4b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.0005369441215446696
+},
+{
+"id": "K5a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.9905241203741747
+},
+{
+"id": "K5b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.021604404187287096
+},
+{
+"id": "K6a",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.9987152563331709
+},
+{
+"id": "K6b",
+"q": "Q1",
+"run": 1,
+"p_yes": 0.0010153053389281193
+},
+{
+"id": "U1a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.036065371680301876
+},
+{
+"id": "U1b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.016015495356012023
+},
+{
+"id": "U2a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9928237567830527
+},
+{
+"id": "U2b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9819637611648699
+},
+{
+"id": "U3a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9911519256833594
+},
+{
+"id": "U3b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9896628364737204
+},
+{
+"id": "U4a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.07382253819152373
+},
+{
+"id": "U4b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.1958787322666148
+},
+{
+"id": "U5a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.8139829254666326
+},
+{
+"id": "U5b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.31925609067679434
+},
+{
+"id": "U6a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.5005578992435681
+},
+{
+"id": "U6b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.17424720135095165
+},
+{
+"id": "U7a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.2632437468400511
+},
+{
+"id": "U7b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.2088569763616402
+},
+{
+"id": "U8a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9238114271132076
+},
+{
+"id": "U8b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.8458922188739306
+},
+{
+"id": "T1a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.06844525246176938
+},
+{
+"id": "T1b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.03048891340927842
+},
+{
+"id": "T2a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.06271505803458369
+},
+{
+"id": "T2b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.10705744486969046
+},
+{
+"id": "T3a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.10345688735641251
+},
+{
+"id": "T3b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.09088666758694891
+},
+{
+"id": "T4a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.5041784267436615
+},
+{
+"id": "T4b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.5448751840860677
+},
+{
+"id": "T5a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.8486552233735357
+},
+{
+"id": "T5b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.03331734516287631
+},
+{
+"id": "T6a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.5903050903416717
+},
+{
+"id": "T6b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.06163418042691574
+},
+{
+"id": "N1a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9952257610652505
+},
+{
+"id": "N1b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.003253875244862294
+},
+{
+"id": "N2a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9945098325502285
+},
+{
+"id": "N2b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.01492617814078072
+},
+{
+"id": "N3a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9879770039907161
+},
+{
+"id": "N3b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.03849597359112111
+},
+{
+"id": "N4a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9700146589847463
+},
+{
+"id": "N4b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.2949851143799287
+},
+{
+"id": "N5a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9854057133971014
+},
+{
+"id": "N5b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.004827952746193422
+},
+{
+"id": "N6a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9450633776994529
+},
+{
+"id": "N6b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.8507882764263268
+},
+{
+"id": "K1a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9987219591117057
+},
+{
+"id": "K1b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.0018607377876475697
+},
+{
+"id": "K2a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9986368018314127
+},
+{
+"id": "K2b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.0021420849777845407
+},
+{
+"id": "K3a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9983703033039442
+},
+{
+"id": "K3b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.026889236926969314
+},
+{
+"id": "K4a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9958700895758017
+},
+{
+"id": "K4b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.0007353601143941215
+},
+{
+"id": "K5a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9974361652933591
+},
+{
+"id": "K5b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.046099363839834075
+},
+{
+"id": "K6a",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.9990418145507582
+},
+{
+"id": "K6b",
+"q": "Q2",
+"run": 1,
+"p_yes": 0.0009381008392334215
+}
+]

jev_backend.py

@@ -11,10 +11,12 @@
 
 from __future__ import annotations
 
+import ctypes
 import gc
 import json
 import logging
 import os
+import struct
 from concurrent.futures import ThreadPoolExecutor
 from typing import Any
 
@@ -23,6 +25,9 @@ import folder_paths
 import numpy as np
 
 try:
+    # The Clef path reads its newer functions from this module at call time,
+    # so llama-cpp-python builds without them still serve the other local models.
+    import llama_cpp.llama_cpp as llama_lib
     from llama_cpp import (
         Llama,
         llama_batch_free,
@@ -52,6 +57,33 @@ _DIRECT_SYSTEM = (
 )
 _N_CTX = 8192
 
+# Clef (Cloudflare's decision model) prompt, as in joint_schema_model.py of Cloudflare/clef-flash (Apache-2.0)
+# and the "systemone" template of its GGUF.
+_CLEF_SYSTEM = (
+    "Read the complete state and schema. Decide every field jointly. "
+    "Each answer must be exactly one of that field's allowed options."
+)
+_CLEF_NOUL_OPTIONS = [
+    ("true", "The proposition is true or the answer is yes."),
+    ("false", "The proposition is false or the answer is no."),
+]
+# enum llama_decision_order: which tokens the decision head reads as a question of each type, or as an option
+_CLEF_ORDER_QUESTION = {"noul": 1, "choice": 2, "score": 3}
+_CLEF_ORDER_OPTION = 4
+# llama_batch_ext_set_decision_order is exported with C++ linkage: MSVC name, then Itanium (macOS, Linux)
+_CLEF_SET_ORDER_SYMBOLS = (
+    "?llama_batch_ext_set_decision_order@@YA_NPEAUllama_batch_ext@@HW4llama_decision_order@@@Z",
+    "__Z34llama_batch_ext_set_decision_orderP15llama_batch_exti20llama_decision_order",
+    "_Z34llama_batch_ext_set_decision_orderP15llama_batch_exti20llama_decision_order",
+)
+_LLAMA_PROCESS_TYPE_DECODE = 1
+_GGUF_TYPE_STRING = 8
+
+# d1 (Liquid AI's decision model) prompt and answer words, as in prompt.py of LiquidAI/d1-3B
+# and the "systemone" template of its GGUF.
+_D1_CODES = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+_D1_NOUL_FORMS = [("yes", "Yes", "YES"), ("no", "No", "NO")]
+
 _loaded_model: tuple[str, Any, Any] | None = None
 
 
@@ -157,12 +189,18 @@ class JevBackend:
 
 
 class LocalJev:
-    """Answers with a local GGUF model, one forward pass per question. Noul is asked as Yes/No options."""
+    """Answers with a local GGUF model, one forward pass per question.
+
+    A d1 model is asked in the prompt format it was tuned on. Other models get lettered options, noul as Yes/No.
+    """
 
     def __init__(self, model_path: str):
         self.model_path = model_path
 
     def ask(self, state: Any, questions: list[tuple]) -> list[list[float]]:
+        llm, _ = load_local_model(self.model_path)
+        if llm.metadata.get("lfm2.decision.type") == "lfm2-d1":
+            return [_d1_probabilities(llm, state, *question) for question in questions]
         answers = []
         for kind, question, criteria in questions:
             if kind == "noul":
@@ -175,6 +213,109 @@ class LocalJev:
         return answers
 
 
+class ClefJev:
+    """Answers with a local Clef GGUF: every question about one state in one forward pass.
+
+    The model's joint decision head scores each option of each question; no text is generated.
+    """
+
+    def __init__(self, model_path: str):
+        self.model_path = model_path
+
+    def ask(self, state: Any, questions: list[tuple]) -> list[list[float]]:
+        llm = load_clef_model(self.model_path)
+        options = [_clef_options(kind, criteria) for kind, _, criteria in questions]
+
+        # The pieces are tokenized one by one, as in training.
+        tokens: list[int] = []
+        orders: list[int] = []
+
+        def add(text: str, order: int = 0) -> None:
+            piece = llm.tokenize(text.encode("utf-8"), add_bos=False, special=True)
+            tokens.extend(piece)
+            orders.extend([order] * len(piece))
+
+        add(f"<|im_start|>system\n{_CLEF_SYSTEM}<|im_end|>\n<|im_start|>user\nSTATE:\n")
+        add(_clef_render(state))
+        add("\n\nSCHEMA FIELDS:\n")
+        for i, (kind, question, _) in enumerate(questions):
+            add(f"\nFIELD {i + 1}\nID: q{i}\nTYPE: {kind}\nINSTRUCTION: ")
+            add(_clef_render(question), _CLEF_ORDER_QUESTION[kind])
+            add("\nALLOWED OPTIONS:\n")
+            for j, (key, description) in enumerate(options[i]):
+                add(f"OPTION {j + 1}: ")
+                semantics = {"option_id": key}
+                if description is not None:
+                    semantics["description"] = description
+                add(_clef_render(semantics), _CLEF_ORDER_OPTION)
+                add("\n")
+            add("END FIELD\n")
+        add("\n<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\nJOINT SCHEMA DECISIONS:")
+        if len(tokens) > llm.n_ctx():
+            raise ValueError(f"Jev prompt is {len(tokens)} tokens, over the {llm.n_ctx()} token context")
+
+        set_order = _clef_set_order_function()
+        llama_memory_clear(llama_get_memory(llm.ctx), True)
+        batch = llama_lib.llama_batch_ext_init(llm.ctx)
+        try:
+            for index, (token, order) in enumerate(zip(tokens, orders)):
+                llama_lib.llama_batch_ext_add_token(batch, 0, token)
+                llama_lib.llama_batch_ext_set_pos(batch, index, ctypes.byref(llama_lib.llama_pos(index)))
+                llama_lib.llama_batch_ext_set_output_embd(batch, index, True)
+                if order:
+                    set_order(batch, index, order)
+            if llama_lib.llama_process(llm.ctx, _LLAMA_PROCESS_TYPE_DECODE, batch) != 0:
+                raise RuntimeError("llama_process failed for the Jev prompt")
+        finally:
+            llama_lib.llama_batch_ext_free(batch)
+
+        # Row i of the embeddings output holds the score of option i, counted across the questions.
+        results = []
+        row = 0
+        for (kind, _, criteria), question_options in zip(questions, options):
+            scores = np.array(
+                [llama_lib.llama_get_embeddings_ith(llm.ctx, row + i)[0] for i in range(len(question_options))],
+                dtype=np.float64,
+            )
+            row += len(question_options)
+            if np.isnan(scores).any():
+                raise RuntimeError("The Clef model could not evaluate the questions")
+            weights = np.exp(scores - scores.max())
+            probabilities = dict(zip((key for key, _ in question_options), (weights / weights.sum()).tolist()))
+            if kind == "choice":
+                # Clef takes the options sorted by label; answer in the order they were given.
+                results.append([probabilities[str(label)] for label in criteria])
+            else:
+                results.append(list(probabilities.values()))
+        return results
+
+
+def _clef_render(value: Any) -> str:
+    if isinstance(value, str):
+        return value
+    return json.dumps(value, ensure_ascii=False, separators=(",", ":"), sort_keys=True)
+
+
+def _clef_options(kind: str, criteria: Any) -> list[tuple[str, Any]]:
+    if kind == "noul":
+        return _CLEF_NOUL_OPTIONS
+    if kind == "choice":
+        return sorted((str(label), description) for label, description in criteria.items())
+    return [(str(level), description) for level, description in enumerate(criteria)]
+
+
+def _clef_set_order_function() -> Any:
+    for symbol in _CLEF_SET_ORDER_SYMBOLS:
+        try:
+            function = getattr(llama_lib._lib, symbol)
+        except AttributeError:
+            continue
+        function.argtypes = [ctypes.c_void_p, ctypes.c_int32, ctypes.c_int]
+        function.restype = ctypes.c_bool
+        return function
+    raise RuntimeError("This llama-cpp-python build cannot run Clef models. See the README for the required version.")
+
+
 class TypeSafeJev:
     """Answers with TypeSafe's Jev API, all questions about one state in one request.
 
@@ -221,15 +362,67 @@ def _option_probabilities(model_path: str, state: Any, question: Any, options: l
         {"role": "user", "content": json.dumps(payload, ensure_ascii=False)},
     ]
     prompt = formatter(messages=messages, enable_thinking=False).prompt
-    tokens = llm.tokenize(prompt.encode("utf-8"), add_bos=False, special=True)
-    if len(tokens) >= llm.n_ctx():
-        raise ValueError(f"Jev prompt is {len(tokens)} tokens, over the {llm.n_ctx()} token context")
     slots = []
     for letter in _LETTERS[: len(options)]:
         encoded = llm.tokenize(letter.encode("utf-8"), add_bos=False, special=False)
         if len(encoded) != 1:
             raise ValueError(f"Model tokenizer does not encode {letter!r} as one token")
         slots.append(encoded[0])
+    logits = _next_token_logits(llm, prompt)[slots].astype(np.float64)
+    weights = np.exp(logits - logits.max())
+    return (weights / weights.sum()).tolist()
+
+
+def _d1_text(value: Any) -> str:
+    return value if isinstance(value, str) else json.dumps(value, ensure_ascii=False)
+
+
+def _d1_probabilities(llm: Any, state: Any, kind: str, question: Any, criteria: Any) -> list[float]:
+    """Reads one decision from the next-token logits of the answer words d1 was tuned to reply with."""
+
+    def token_ids(forms: Any) -> list[int]:
+        encoded = [llm.tokenize(form.encode("utf-8"), add_bos=False, special=False) for form in forms]
+        return [tokens[0] for tokens in encoded if len(tokens) == 1]
+
+    if kind == "noul":
+        body = "\n\nReply with yes or no only."
+        groups = [token_ids(forms) for forms in _D1_NOUL_FORMS]
+    elif kind == "choice":
+        if not 2 <= len(criteria) <= len(_D1_CODES):
+            raise ValueError(f"d1 choice questions need 2-{len(_D1_CODES)} options, got {len(criteria)}")
+        labels = [str(label).strip() for label in criteria]
+        # Labels that are single letters are their own codes.
+        codes = labels if all(len(label) == 1 and label.isalpha() for label in labels) else _D1_CODES[: len(labels)]
+        lines = [
+            f"{code} {_d1_text(description) if description else label.replace('_', ' ')}"
+            for code, label, description in zip(codes, labels, criteria.values())
+        ]
+        body = "\n\nOptions:\n" + "\n".join(lines) + "\n\nReply with the option code only."
+        groups = [token_ids((code, f" {code}")) for code in codes]
+    else:
+        if not 2 <= len(criteria) <= 10:
+            raise ValueError(f"d1 score questions need 2-10 levels, got {len(criteria)}")
+        legend = "".join(f"{level} {_d1_text(description)}\n" for level, description in enumerate(criteria))
+        body = f"\n\n{legend}\nReply with a single digit 0-{len(criteria) - 1} only."
+        groups = [token_ids([str(level)]) for level in range(len(criteria))]
+    if not all(groups) or len({group[0] for group in groups}) != len(groups):
+        raise ValueError("Model tokenizer does not encode each d1 answer as one distinct token")
+    state_text = state if isinstance(state, str) else json.dumps(state, ensure_ascii=False, indent=2)
+    prompt = (
+        f"<|startoftext|><|im_start|>user\n{state_text}\n\n\nQUESTION:\n"
+        f"{_d1_text(question)}{body}<|im_end|>\n<|im_start|>assistant\n"
+    )
+    vocabulary = _next_token_logits(llm, prompt)
+    # Each option scores its best form.
+    scores = np.array([vocabulary[group].max() for group in groups], dtype=np.float64)
+    weights = np.exp(scores - scores.max())
+    return (weights / weights.sum()).tolist()
+
+
+def _next_token_logits(llm: Any, prompt: str) -> Any:
+    tokens = llm.tokenize(prompt.encode("utf-8"), add_bos=False, special=True)
+    if len(tokens) >= llm.n_ctx():
+        raise ValueError(f"Jev prompt is {len(tokens)} tokens, over the {llm.n_ctx()} token context")
     # Each question starts from empty memory, including the recurrent state of hybrid models such as Qwen3.5.
     # Decode with the llama.cpp API directly instead of the Llama evaluation helper,
     # whose method name trips the ComfyUI Registry dynamic-execution scanner.
@@ -247,10 +440,7 @@ def _option_probabilities(model_path: str, state: Any, question: Any, options: l
             raise RuntimeError("llama_decode failed for the Jev prompt")
     finally:
         llama_batch_free(batch)
-    vocabulary = np.ctypeslib.as_array(llama_get_logits_ith(llm.ctx, -1), shape=(llm.n_vocab(),))
-    logits = vocabulary[slots].astype(np.float64)
-    weights = np.exp(logits - logits.max())
-    return (weights / weights.sum()).tolist()
+    return np.ctypeslib.as_array(llama_get_logits_ith(llm.ctx, -1), shape=(llm.n_vocab(),))
 
 
 def _model_dirs() -> list[str]:
@@ -287,17 +477,45 @@ def _display_name(path: str) -> str:
     return path if rel.startswith("..") else rel.replace("\\", "/")
 
 
-def load_local_model(model_path: str) -> tuple[Any, Any]:
-    global _loaded_model
-    if _loaded_model is not None and _loaded_model[0] == model_path:
-        return _loaded_model[1], _loaded_model[2]
+def is_clef_model(model_path: str) -> bool:
+    """Reads general.architecture, which GGUF writers store as the first metadata entry."""
+    with open(model_path, "rb") as f:
+        # magic, version, tensor count, metadata count, then the first key: length and bytes
+        header = f.read(32)
+        if len(header) < 32 or header[:4] != b"GGUF":
+            return False
+        key_length = struct.unpack("<Q", header[24:])[0]
+        if key_length != len(b"general.architecture") or f.read(key_length) != b"general.architecture":
+            return False
+        value_type, value_length = struct.unpack("<IQ", f.read(12))
+        return value_type == _GGUF_TYPE_STRING and value_length == len(b"clef") and f.read(value_length) == b"clef"
+
+
+def _open_model(model_path: str, **kwargs: Any) -> Any:
     if Llama is None:
         raise RuntimeError("llama-cpp-python is not installed. See the README for installation.")
     unload_local_model()
     device = comfy.model_management.get_torch_device()
     comfy.model_management.free_memory(os.path.getsize(model_path) * 1.2, device)
-    # n_batch = n_ctx so one llama_decode call can take the whole prompt
-    llm = Llama(model_path=model_path, n_ctx=_N_CTX, n_batch=_N_CTX, n_gpu_layers=-1, verbose=False)
+    # n_batch = n_ctx so one decode call can take the whole prompt
+    return Llama(model_path=model_path, n_ctx=_N_CTX, n_batch=_N_CTX, n_gpu_layers=-1, verbose=False, **kwargs)
+
+
+def load_clef_model(model_path: str) -> Any:
+    global _loaded_model
+    if _loaded_model is not None and _loaded_model[0] == model_path:
+        return _loaded_model[1]
+    # The decision head reads the whole prompt in one physical batch, and its scores come out as embeddings.
+    llm = _open_model(model_path, n_ubatch=_N_CTX, embeddings=True)
+    _loaded_model = (model_path, llm, None)
+    return llm
+
+
+def load_local_model(model_path: str) -> tuple[Any, Any]:
+    global _loaded_model
+    if _loaded_model is not None and _loaded_model[0] == model_path:
+        return _loaded_model[1], _loaded_model[2]
+    llm = _open_model(model_path)
     template = llm.metadata.get("tokenizer.chat_template")
     if not template:
         llm.close()

pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "ComfyUI-ScriptFlow"
 description = "Safe Python-like script node with a System One decision model: ask yes/no, choice, and score questions via TypeSafe AI's Jev API, with local GGUF LLMs as a fallback. Multiple inputs and outputs let one node handle math, logic, text editing, conditional routing, resolution selection, filename generation, and more."
-version = "1.3.2"
+version = "1.3.5"
 license = {file = "LICENSE"} 
 # classifiers = [
 #     # For OS-independent nodes (works on all operating systems)