Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 kantan-kanto
|
d782fa5be3 | ||
|
kantan-kanto
|
bb582fd373 | ||
|
kantan-kanto
|
1285924f0a | ||
|
kantan-kanto
|
3e1ecf6807 | ||
|
kantan-kanto
|
be86cc8ce3 | ||
|
kantan-kanto
|
6422d27eb5 |
+10
-1
@@ -3,8 +3,17 @@
|
||||
All notable changes to ComfyUI-ScriptFlow will be documented in this file.
|
||||
|
||||
|
||||
## [Unreleased]
|
||||
## [1.3.4] - 2026-10-09
|
||||
|
||||
- `MultiOutputScript (Jev)` runs local d1 GGUF models, Liquid AI's open-weight decision model
|
||||
- A GGUF whose `lfm2.decision.type` metadata is `lfm2-d1` (such as `LiquidAI/d1-3B-GGUF`) is asked in the prompt format d1 was tuned on, and the answer is read from the next-token probabilities of its yes/no, option code, or digit reply, instead of the lettered-option approximation used for general local models
|
||||
- One forward pass per question; `jev.choice` and `jev.probabilities` take 2 to 26 options and `jev.score` takes 2 to 10 levels
|
||||
- Text only, with the `state` and one question limited to 8,192 tokens
|
||||
- Tested with d1-3B Q8_0 and the JamePeng fork of llama-cpp-python 0.4.2 on Windows (Intel Arc, SYCL build); Linux and macOS are untested
|
||||
- Added a technical report, `docs/local-model-uncertainty/` (English and Japanese), on how Jev, Clef-Flash, and a general local model (Qwen3.5-9B) answer questions whose answer cannot be known
|
||||
- With a general local model, a low `jev.noul` value can mean "cannot say it is true" rather than "unlikely"
|
||||
- Asking about both a statement and its negation, or picking a percentage level with `jev.score`, separates the two; the report has the measurements, example scripts, and the statements and results as JSON
|
||||
- `README.md` and `README.ja.md` link to the report from the notes of `MultiOutputScript (Jev)`
|
||||
|
||||
## [1.3.3] - 2026-10-05
|
||||
|
||||
|
||||
+10
-5
@@ -3,7 +3,7 @@
|
||||
|
||||
---
|
||||
|
||||
**Version:** 1.3.3
|
||||
**Version:** 1.3.4
|
||||
**License:** GPL-3.0
|
||||
|
||||
System One の判断モデルを使える、安全な Python 風スクリプトノードです。
|
||||
@@ -56,6 +56,7 @@ System One モデルはテキストを生成せず、1 回の高速な処理で
|
||||
- ComfyUI を実行している Python 環境に [llama-cpp-python](https://github.com/JamePeng/llama-cpp-python) をインストールします。JamePeng 版と、本家の [abetlen/llama-cpp-python](https://github.com/abetlen/llama-cpp-python) 0.3.35 で動作を確認しています。
|
||||
- 指示追従型（instruction-tuned）の GGUF モデルを `ComfyUI/models/LLM`（または `models/text_encoders`）に置きます。GGUF にはチャットテンプレート（`tokenizer.chat_template` メタデータ）が含まれている必要があります。ほとんどの指示追従型 GGUF には含まれています。Qwen3.5-9B Q8_0 と Gemma-4-E4B Q8 で動作を確認しています。
|
||||
- Cloudflare のオープンウェイトの判断モデル [Clef](https://huggingface.co/Cloudflare/clef-flash) を使う場合は、llama.cpp ネイティブ形式の Clef の GGUF（[ggml-org/Clef-Flash-GGUF](https://huggingface.co/ggml-org/Clef-Flash-GGUF) など）を同じフォルダに置きます。JamePeng 版の llama-cpp-python 0.4.2 以降が必要です。通常のチャットモデルとして変換された Clef の GGUF は汎用モデルとして扱われ、Clef の判断ヘッドは使われません。Windows（Intel Arc、SYCL ビルド）の Clef-Flash Q8_0 で動作を確認しています。Linux と macOS では未確認です。
|
||||
- Liquid AI のオープンウェイトの判断モデル [d1](https://huggingface.co/LiquidAI/d1-3B) を使う場合は、d1 の GGUF（[LiquidAI/d1-3B-GGUF](https://huggingface.co/LiquidAI/d1-3B-GGUF) など）を同じフォルダに置きます。LFM2 モデルを読み込める llama-cpp-python が必要です。`mmproj` ファイルは不要です。Windows（Intel Arc、SYCL ビルド）の d1-3B Q8_0 と JamePeng 版の llama-cpp-python 0.4.2 で動作を確認しています。Linux と macOS では未確認です。
|
||||
|
||||
## 主な機能
|
||||
- 複数の入力：数値とテキストの入力を自由に組み合わせられます。
|
||||
@@ -126,6 +127,7 @@ ov1, ov2 = (512, 384) if w >= h else (384, 512)
|
||||
- **TypeSafe API**（基本）：質問をインターネット経由で Jev に送ります。
|
||||
- **ローカル GGUF モデル**（フォールバック）：Jev API を使えない場合や、テキストを手元のマシンから出したくない場合に使います。汎用の指示追従型モデルで Jev 風の判断を近似するもので、[SemIf (OpenJev)](https://github.com/TheoLeeCJ/SemIf-OpenJev) の方式に従い、文字を割り当てた選択肢の次トークン確率を 1 回の順伝播で読み取ります。テキストは生成せず、ネットワークにも何も送りません。答えや確率は Jev とは異なります。
|
||||
- **ローカルの Clef モデル**：[Clef](https://huggingface.co/Cloudflare/clef-flash) は、Jev と同じ種類の質問に答えるオープンウェイトの判断モデルです。GGUF のメタデータ `general.architecture` が `clef` のモデルを、Clef として扱います。Clef の GGUF を選ぶと、上の近似の代わりにモデルの判断ヘッドを使い、同じ `state` についての質問すべてに 1 回の順伝播で答えます。`state` と質問を Clef の入力書式に並べ、判断ヘッドが出す選択肢ごとのスコアを、質問ごとに softmax で確率にします。文章は生成せず、ネットワークにも何も送信しません。答えと確率は Jev とは異なります。
|
||||
- **ローカルの d1 モデル**：[d1](https://huggingface.co/LiquidAI/d1-3B) は、Jev と同じ種類の質問に答えるオープンウェイトの判断モデルです。GGUF のメタデータ `lfm2.decision.type` が `lfm2-d1` のモデルを、d1 として扱います。d1 の GGUF を選ぶと、上の文字を割り当てた選択肢の代わりに、d1 が学習したプロンプト形式で質問します。答えは、d1 が学習した返答の次トークン確率から読み取ります。`jev.noul` では yes か no、`jev.choice` では選択肢のコード、`jev.score` では数字です。文章は生成せず、ネットワークにも何も送信しません。答えと確率は Jev とは異なります。
|
||||
|
||||
スクリプトはどれでも同じです。各バックエンドのセットアップは[インストール](#インストール)を参照してください。
|
||||
|
||||
@@ -197,9 +199,12 @@ client.system_one(state=it1, questions={"q0": Noul(instructions="q1"), "q1": Nou
|
||||
|
||||
注意：
|
||||
- TypeSafe API は `state` と質問をインターネット経由で送信します。外部に出せないテキストには使わないでください。
|
||||
- 汎用のローカルモデルは、質問 1 つにつき 1 回の順伝播で処理します（GPU 上の 4B〜9B モデルで約 1〜1.5 秒）。Clef モデルは、1 回のリクエストに含まれる質問すべてを 1 回の順伝播で処理します（Intel Arc の内蔵 GPU 上の Clef-Flash Q8_0 で、300 トークンのリクエストが約 3 秒）。モデルは実行ごとに最初の質問で読み込まれ（数秒）、実行が終わるとアンロードされて、ワークフローの残りのために VRAM を解放します。
|
||||
- 汎用のローカルモデルは、質問 1 つにつき 1 回の順伝播で処理します（GPU 上の 4B〜9B モデルで約 1〜1.5 秒）。Clef モデルは、1 回のリクエストに含まれる質問すべてを 1 回の順伝播で処理します（Intel Arc の内蔵 GPU 上の Clef-Flash Q8_0 で、300 トークンのリクエストが約 3 秒）。d1 モデルは、質問 1 つにつき 1 回の順伝播で処理します（同じ GPU 上の d1-3B Q8_0 で約 0.5 秒）。モデルは実行ごとに最初の質問で読み込まれ（数秒）、実行が終わるとアンロードされて、ワークフローの残りのために VRAM を解放します。
|
||||
- ローカルモデルの確率は較正されておらず、モデルによって変わります。しきい値は実際の入力で試して決めてください。
|
||||
- Qwen3.5-9B での試験では、汎用のローカルモデルは、テキストから答えが決まらない質問に「No」と答えました。`jev.noul` の低い値は、「起こりにくい」ではなく「真だとは言えない」を表すことがあります。この2つを区別したい場合は、主張とその否定形の両方について聞くか、`jev.score` でパーセントの段階を選ばせてください。測定の結果とスクリプトの例は、[汎用のローカルモデルは、不確かな問いにどう答えるか](docs/local-model-uncertainty/README.ja.md) にあります。
|
||||
- ローカルモデルのプロンプトは 8,192 トークンまでです。汎用モデルでは `state` と質問 1 つ、Clef では `state` と 1 回のリクエストの質問すべてがこの範囲に収まる必要があります。Clef はテキストのみに対応しています。
|
||||
- d1 モデルでは、`jev.choice` と `jev.probabilities` の選択肢は 2〜26 個、`jev.score` の段階は 2〜10 個です。画像には対応していません。
|
||||
- d1-3B での試験では、似た選択肢に確率が分かれ、1 つだけ性質の違う選択肢が最多になることがありました。アスペクト比の例では、縦長の 3 つの比率の合計が 0.49 だったのに対し、0.35 の `1:1` が選ばれました。選択肢がグループに分かれる場合は、`jev.probabilities` をグループごとに合計してから選んでください。
|
||||
- 実行ごとにコンソールへ `[ComfyUI-ScriptFlow] Jev requests: N, states: S, questions: Q, responses: M` を出力します。送ったリクエスト数、異なる `state` の数、質問数、返信があったリクエスト数です。返信数がリクエスト数より少なければ、途中のリクエストが失敗しています。同じ実行のキャッシュから答えた質問は数えません。
|
||||
- `typesafe-sdk` がインストールされていると、`httpx2`（SDK が使う HTTP クライアント）と `typesafe_sdk` のロガーを WARNING に設定するため、リクエストごとの INFO ログは表示されません。
|
||||
- `MultiOutputScript` はモデルを読み込まず、ネットワークにもアクセスしません。`jev` は使えません。
|
||||
@@ -755,6 +760,6 @@ You should have received a copy of the GNU General Public License along with thi
|
||||
|
||||
詳しいバージョン履歴は [CHANGELOG.md](CHANGELOG.md) を参照してください。
|
||||
|
||||
### 現在のバージョン：1.3.3
|
||||
- `MultiOutputScript (Jev)` が、ローカルの Clef の GGUF モデル（Cloudflare のオープンウェイトの判断モデル）をモデルの判断ヘッドで動かせるように。同じ `state` についての質問すべてに 1 回の順伝播で答える
|
||||
- プロンプトエンハンサーの出力をシステムプロンプトと照らし合わせる応用レシピを追加。長いシステムプロンプト用の 2 つめのスクリプトも含む
|
||||
### 現在のバージョン：1.3.4
|
||||
- `MultiOutputScript (Jev)` が、ローカルの d1 の GGUF モデル（Liquid AI のオープンウェイトの判断モデル）を動かせるように。d1 が学習したプロンプト形式で質問する
|
||||
- Jev、Clef-Flash、汎用のローカルモデルが、答えを知りようのない問いにどう答えるかを調べた技術レポートを追加
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# ComfyUI-ScriptFlow
|
||||
[en | [ja](README.ja.md)]
|
||||
|
||||
**Version:** 1.3.3
|
||||
**Version:** 1.3.4
|
||||
**License:** GPL-3.0
|
||||
|
||||
Safe Python-like script node for ComfyUI with a System One decision model.
|
||||
@@ -54,6 +54,7 @@ The `MultiOutputScript` and `centi` nodes need no extra packages. `MultiOutputSc
|
||||
- Install [llama-cpp-python](https://github.com/JamePeng/llama-cpp-python) in the Python environment that runs ComfyUI. Tested with the JamePeng fork and the upstream [abetlen/llama-cpp-python](https://github.com/abetlen/llama-cpp-python) 0.3.35.
|
||||
- Put an instruction-tuned GGUF model in `ComfyUI/models/LLM` (or `models/text_encoders`). The GGUF must include a chat template (`tokenizer.chat_template` metadata), as most instruction-tuned GGUFs do. Tested with Qwen3.5-9B Q8_0 and Gemma-4-E4B Q8.
|
||||
- To use [Clef](https://huggingface.co/Cloudflare/clef-flash), Cloudflare's open-weight decision model, put a Clef GGUF in the llama.cpp native format, such as [ggml-org/Clef-Flash-GGUF](https://huggingface.co/ggml-org/Clef-Flash-GGUF), in the same folder. It needs the JamePeng fork of llama-cpp-python 0.4.2 or later. A Clef GGUF converted as an ordinary chat model is treated as a general model and does not use Clef's decision head. Tested with Clef-Flash Q8_0 on Windows (Intel Arc, SYCL build); Linux and macOS are untested.
|
||||
- To use [d1](https://huggingface.co/LiquidAI/d1-3B), Liquid AI's open-weight decision model, put a d1 GGUF, such as [LiquidAI/d1-3B-GGUF](https://huggingface.co/LiquidAI/d1-3B-GGUF), in the same folder. It needs a llama-cpp-python build that loads LFM2 models. The `mmproj` file is not needed. Tested with d1-3B Q8_0 and the JamePeng fork of llama-cpp-python 0.4.2 on Windows (Intel Arc, SYCL build); Linux and macOS are untested.
|
||||
|
||||
## Key Features
|
||||
- Multiple inputs: freely combine numeric and text inputs.
|
||||
@@ -124,6 +125,7 @@ Same inputs, outputs, and script rules as `MultiOutputScript`, plus a `jev` name
|
||||
- **TypeSafe API** (primary): sends the questions to Jev over the internet.
|
||||
- **Local GGUF model** (fallback): for when the Jev API is not available, or when text must stay on your machine. It approximates Jev-style decisions with a general instruction-tuned model, reading the answer from next-token probabilities of lettered options in one forward pass, following [SemIf (OpenJev)](https://github.com/TheoLeeCJ/SemIf-OpenJev). No text is generated and nothing is sent over the network. Answers and probabilities differ from Jev's.
|
||||
- **Local Clef model**: [Clef](https://huggingface.co/Cloudflare/clef-flash) is an open-weight decision model with the same question types as Jev. A GGUF whose `general.architecture` metadata is `clef` is treated as a Clef model. Selecting one uses the model's decision head instead of the approximation above, and answers every question about one `state` in one forward pass: the `state` and the questions are laid out in Clef's input format, and the score the decision head gives each option is turned into probabilities with a softmax per question. No text is generated and nothing is sent over the network. Answers and probabilities differ from Jev's.
|
||||
- **Local d1 model**: [d1](https://huggingface.co/LiquidAI/d1-3B) is an open-weight decision model with the same question types as Jev. A GGUF whose `lfm2.decision.type` metadata is `lfm2-d1` is treated as a d1 model. Selecting one asks each question in the prompt format d1 was tuned on instead of the lettered options above, and reads the answer from the next-token probabilities of the reply d1 was tuned to give: yes or no for `jev.noul`, an option code for `jev.choice`, and a digit for `jev.score`. No text is generated and nothing is sent over the network. Answers and probabilities differ from Jev's.
|
||||
|
||||
Scripts are the same for all of them. See [Installation](#installation) for the setup of each backend.
|
||||
|
||||
@@ -195,9 +197,12 @@ client.system_one(state=it1, questions={"q0": Noul(instructions="q1"), "q1": Nou
|
||||
|
||||
Notes:
|
||||
- The TypeSafe API sends `state` and questions over the internet. Do not use it with text you cannot share externally.
|
||||
- A general local model takes one forward pass per question (about 1–1.5 s with a 4B–9B model on GPU). A Clef model takes one forward pass per request, for all of its questions (about 3 s for a 300-token request with Clef-Flash Q8_0 on an Intel Arc integrated GPU). The model is loaded on the first question of each run (several seconds) and unloaded when the run ends to free VRAM for the rest of the workflow.
|
||||
- A general local model takes one forward pass per question (about 1–1.5 s with a 4B–9B model on GPU). A Clef model takes one forward pass per request, for all of its questions (about 3 s for a 300-token request with Clef-Flash Q8_0 on an Intel Arc integrated GPU). A d1 model takes one forward pass per question (about 0.5 s with d1-3B Q8_0 on the same GPU). The model is loaded on the first question of each run (several seconds) and unloaded when the run ends to free VRAM for the rest of the workflow.
|
||||
- Local probabilities are uncalibrated and depend on the model; test thresholds on your own inputs.
|
||||
- In tests with Qwen3.5-9B, a general local model answered "no" when the text did not settle the question, so a low `jev.noul` value can mean "cannot say it is true" rather than "unlikely". To tell the two apart, ask about both the statement and its negation, or have the model pick a percentage level with `jev.score`. See [How general local models answer uncertain questions](docs/local-model-uncertainty/README.md) for the measurements and example scripts.
|
||||
- A local prompt is limited to 8,192 tokens: the `state` and one question for a general model, or the `state` and all questions of a request for Clef. Clef is text only.
|
||||
- With a d1 model, `jev.choice` and `jev.probabilities` take 2 to 26 options and `jev.score` takes 2 to 10 levels. Images are not supported.
|
||||
- In tests with d1-3B, the probabilities of similar options were spread out, so a single different option could come first: in the aspect ratio example, three portrait ratios shared 0.49 and `1:1` was chosen with 0.35. When options fall into groups, add up `jev.probabilities` by group before picking one.
|
||||
- Each run prints `[ComfyUI-ScriptFlow] Jev requests: N, states: S, questions: Q, responses: M` to the console: requests sent, distinct states, questions asked, and requests answered. Fewer responses than requests means a request failed. Questions answered from the run's cache are not counted.
|
||||
- When `typesafe-sdk` is installed, the `httpx2` (the SDK's HTTP client) and `typesafe_sdk` loggers are set to WARNING, so per-request INFO lines are not shown.
|
||||
- `MultiOutputScript` never loads a model or makes network requests; `jev` is not available there.
|
||||
@@ -779,6 +784,6 @@ You should have received a copy of the GNU General Public License along with thi
|
||||
|
||||
See [CHANGELOG.md](CHANGELOG.md) for detailed version history.
|
||||
|
||||
### Current Version: 1.3.3
|
||||
- `MultiOutputScript (Jev)` runs local Clef GGUF models (Cloudflare's open-weight decision model) with the model's decision head, answering every question about one `state` in one forward pass
|
||||
- Added an application recipe that checks a prompt enhancer's output against its system prompt, with a second script for long system prompts
|
||||
### Current Version: 1.3.4
|
||||
- `MultiOutputScript (Jev)` runs local d1 GGUF models (Liquid AI's open-weight decision model), asking each question in the prompt format d1 was tuned on
|
||||
- Added a technical report on how Jev, Clef-Flash, and a general local model answer questions whose answer cannot be known
|
||||
|
||||
@@ -0,0 +1,384 @@
|
||||
# 汎用のローカルモデルは、不確かな問いにどう答えるか：`jev.*` の挙動と聞き方の指針
|
||||
[[en](README.md) | ja]
|
||||
|
||||
技術報告（2026年10月）
|
||||
|
||||
## 要約
|
||||
|
||||
`MultiOutputScript (Jev)` は、TypeSafe の Jev の代わりに、ローカルの GGUF モデルで判断を行える。この報告は、答えのわからない問いに対して、Jev、Clef-Flash、汎用モデル（Qwen3.5-9B）がどう答えるかを比べ、汎用モデルを使うときの聞き方を調べたものである。
|
||||
|
||||
- 答えが確実な問いでは、3つのモデルの答えはほぼ一致した。
|
||||
- 答えが不確実な問いでは、振る舞いが分かれた。五分五分の問いに `jev.noul` で「真ですか？」と聞くと、平均は Jev が 0.44、Clef-Flash が 0.31、Qwen3.5-9B が 0.06 だった。
|
||||
- Qwen3.5-9B の低い値は、「起こる確率が低い」ではなく「真だとは言えない」を表している。確率が 0.92 の主張にも 0.02 と答えた。
|
||||
- 主張とその否定形の両方について聞き、2つの答えを合わせることで、Qwen3.5-9B でも五分五分の問いを 0.51 にすることができた。
|
||||
- 確率の段階を選ばせる（`jev.score`）と、確率の高さがある程度取り出せた。結果は、段階の作り方と並び順に左右された。
|
||||
- 「偽ですか？」と質問を反対向きにする方法は、否定形の代わりにならなかった。
|
||||
|
||||
31組62文の小さな試験であり、段階で聞く方法は Qwen3.5-9B の1つの量子化でしか試していない。わかるのは傾向である。
|
||||
|
||||
## 1. 目的
|
||||
|
||||
判断モデルの出力は、しきい値で区切って分岐に使うことが多い（`jev.yes(state, question, 0.5)` など）。このとき、判断材料が足りない入力が混ざると、モデルが「わからない」をどの値で表すかによって、分岐の結果が変わる。
|
||||
|
||||
この報告の目的は、次の2点を確かめることである。
|
||||
|
||||
1. 汎用のローカルモデルは、不確かな問いに対して、Jev とどう違う答え方をするか。
|
||||
2. その違いを、スクリプトの書き方（聞き方）でどこまで埋められるか。
|
||||
|
||||
確率で答えるモデルに期待する性質として、次の2つを基準にした。
|
||||
|
||||
- **基準1**：対になる主張の確率を足すと、1 になる。
|
||||
- **基準2**：五分五分の問いには、0.5 と答える。
|
||||
|
||||
## 2. 方法
|
||||
|
||||
### 2.1 モデル
|
||||
|
||||
| モデル | 種類 | 答えの出し方 |
|
||||
| --- | --- | --- |
|
||||
| Jev | TypeSafe の判断モデル（API） | API が確率を返す |
|
||||
| Clef-Flash | Qwen3.5-9B を判断用に追加学習したモデル（`ggml-org/Clef-Flash-GGUF`、Q8_0） | 判断専用の層が出す値を確率に直す |
|
||||
| Qwen3.5-9B | 汎用の LLM（`unsloth/Qwen3.5-9B-GGUF`、Q8_0） | 選択肢の記号が次に出る確率を読む |
|
||||
|
||||
汎用モデルでは、選択肢に記号を付けて示し（`jev.noul` なら A：Yes、B：No）、記号1文字だけで答えるよう指示する。文章は生成させず、次のトークンとして各記号が出る確率を読み、（記号以外の出力は除き）記号どうしの合計が 1 になるように直す。この方式は SemIf (OpenJev) で提案されているものである。この報告の値は、較正をしていない値である。較正については、3.2 節と 5 節で触れる。なお、サンプリングをしないので、乱数のシードや温度の設定はない。
|
||||
|
||||
指示文は次のとおりである。主張、質問、選択肢は、JSON にまとめて渡す。
|
||||
|
||||
```text
|
||||
Apply the supplied criterion to the supplied evidence. Choose exactly one listed option. Respond with only its uppercase letter, with no explanation or reasoning.
|
||||
```
|
||||
|
||||
3つとも、このノードの判定処理（v1.3.3）をそのまま使った。質問は1問ずつ送った。
|
||||
|
||||
### 2.2 設問
|
||||
|
||||
対になる主張を31組、合計62文用意した。主張と質問は英語である。一覧は [`data/items.json`](data/items.json) にある。
|
||||
|
||||
| グループ | 記号 | 組数 | 内容 | 例 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 既知の事実 | K | 6 | 1つめが真、2つめが偽 | Paris is the capital of France. / ... of Germany. |
|
||||
| ほぼ確実な未来 | N | 6 | 1つめがほぼ確実に起こる | The sun will rise tomorrow. / ... will not rise tomorrow. |
|
||||
| 傾向のある未来 | T | 6 | 1つめが起こりやすいが、確実ではない | The S&P 500 index will be higher one year from now ... / lower ... |
|
||||
| 五分五分の問い | U | 8 | どちらも約50% | The next coin I flip will land heads. / ... tails. |
|
||||
| 確率のわかっている問い | P | 5 | 正しい確率が計算でわかる | The next roll of a fair six-sided die will show a 1.（1/6） / ... a number other than 1.（5/6） |
|
||||
|
||||
K、N、T、U の26組は、3つのモデルすべてに聞いた。P の5組は、あとから Qwen3.5-9B だけに聞いた。
|
||||
|
||||
### 2.3 質問
|
||||
|
||||
| 記号 | 関数 | 質問 | 選択肢 |
|
||||
| --- | --- | --- | --- |
|
||||
| Q1 | `jev.noul` | Is this statement true? | Yes / No |
|
||||
| Q2 | `jev.noul` | Is this statement more likely to be true than false? | Yes / No |
|
||||
| F1 | `jev.noul` | Is this statement false? | Yes / No |
|
||||
| S6 | `jev.score` | How likely is this statement to be true? | 0-10%、10-30%、30-50%、50-70%、70-90%、90-100% |
|
||||
| S11 | `jev.score` | How likely is this statement to be true? | 0%、10%、…、100% |
|
||||
| F6 | `jev.score` | How likely is this statement to be false? | S6 と同じ6段階 |
|
||||
|
||||
Q1 と Q2 は3つのモデルに聞いた。F1、S6、S11、F6 は Qwen3.5-9B だけに聞いた。段階で聞く質問は、選択肢を低い順に並べた場合と、高い順に並べた場合の両方で聞いた。
|
||||
|
||||
対照として、Qwen3.5-9B に確率を数字で書かせた（20文）。指示は「What is the probability that the following statement is true? Answer with a single number from 0 to 100 (percent) and nothing else.」で、最も確率の高いトークンを順に選んで生成した。
|
||||
|
||||
### 2.4 指標
|
||||
|
||||
- **対の合計**：対の2つの主張への値の合計（基準1）。
|
||||
- **0.5 からのずれ**：五分五分の問い16文での、値と 0.5 の差の絶対値の平均（基準2）。
|
||||
- **誤差の平均**：確率のわかっている問い10文での、値と正しい確率の差の絶対値の平均。
|
||||
- **順位相関**：同じ10文での、値と正しい確率のスピアマンの順位相関。
|
||||
- **否定形と合わせた値**：（主張への値 ＋ 1 − 否定形への値）÷ 2。対の合計が当然 1 になる。
|
||||
- **段階からの推定値**：各段階の値（S6 と F6 は段階の中央の値である 0.05、0.20、0.40、0.60、0.80、0.95、S11 はその点の値である 0、0.1、…、1.0）を、モデルが各段階に付けた確率で重み付けした値。
|
||||
|
||||
否定形と合わせた値は、新しく測ったものではなく、測定済みの値から計算したものである。
|
||||
|
||||
## 3. 結果
|
||||
|
||||
### 3.1 3つのモデルの比較（Q1）
|
||||
|
||||
値は Yes の確率の平均で、左が対の1つめ、右が2つめの主張である。特に、太字の斜体で示した Qwen3.5-9B の「傾向のある未来」と「五分五分の問い」の結果は、1 節の基準1・基準2 と乖離しており、3.2 節以降でさらに検証を行う。
|
||||
|
||||
| グループ | Jev | Clef-Flash | Qwen3.5-9B |
|
||||
| --- | --- | --- | --- |
|
||||
| 既知の事実 | 0.92 / 0.03 | 0.95 / 0.02 | 0.99 / 0.01 |
|
||||
| ほぼ確実な未来 | 0.88 / 0.07 | 0.86 / 0.11 | 0.85 / 0.13 |
|
||||
| 傾向のある未来 | 0.58 / 0.37 | 0.62 / 0.36 | ***0.07 / 0.08*** |
|
||||
| 五分五分の問い | 0.44 / 0.44 | 0.36 / 0.26 | ***0.07 / 0.04*** |
|
||||
|
||||
| 指標 | Jev | Clef-Flash | Qwen3.5-9B |
|
||||
| --- | --- | --- | --- |
|
||||
| 対の合計：既知の事実 | 0.95 | 0.97 | 1.00 |
|
||||
| 対の合計：ほぼ確実な未来 | 0.95 | 0.96 | 0.98 |
|
||||
| 対の合計：傾向のある未来 | 0.95 | 0.98 | ***0.15*** |
|
||||
| 対の合計：五分五分の問い | 0.88 | 0.62 | ***0.11*** |
|
||||
| 0.5 からのずれ（五分五分の問い） | 0.06 | 0.22 | ***0.44*** |
|
||||
|
||||
- 既知の事実は、3つとも12文すべてに正しい側の値を返した。
|
||||
- Jev は、五分五分の問い16文すべてに 0.34〜0.51 を返した。
|
||||
- Qwen3.5-9B は、不確かな主張（傾向のある未来、五分五分の問い）のほぼすべてに No と答えた。未来のことをすべて No とするわけではなく、「明日、太陽は昇る」（ほぼ確実な未来）には 0.99 を返している。
|
||||
- Clef-Flash は、傾向のある未来では対の合計が 0.98 で、調整前の Qwen3.5-9B（0.15）と大きく違う。コインやサイコロのような問いでは 0.62 にとどまった（コイン投げは 0.16 / 0.10）。
|
||||
|
||||
Clef-Flash と Qwen3.5-9B では、答えの出し方も違う。2つの差が、重みの調整によるものか、答えの出し方によるものかは、この試験では分けられない。
|
||||
|
||||
Clef-Flash は、傾向のある未来で片方をはっきり選んだ（S&P 500 は 0.80 / 0.06。Jev は 0.53 / 0.31）。ただし、「来年の今日、東京で雨が降らない / 降る」には 0.09 / 0.81 と、実際の頻度とは逆向きの値を返し、「次の米大統領は就任時に50歳より上 / 下」には 0.81 / 0.71 と、両方に Yes と答えた。
|
||||
|
||||
### 3.2 Qwen3.5-9B の No は、何を表しているか
|
||||
|
||||
ここから、Qwen3.5-9B の「傾向のある未来」と「五分五分の問い」の結果について検証する。
|
||||
|
||||
まず確認したのは、Qwen3.5-9B の No が **記号以外のトークンに確率が流れた結果ではない** ということである。語彙全体の確率のうち、A と B が占める割合（SemIf の `allowed_token_mass`）を、62文 × Q1・Q2 の124問で記録した。
|
||||
|
||||
| | 最小 | 中央 | 最大 |
|
||||
| --- | --- | --- | --- |
|
||||
| A と B が占める割合 | 0.9971 | 0.9992 | 0.9999 |
|
||||
|
||||
A・B 以外で最も確率が高かったトークンは「C」で、最大でも 0.15% だった。断りや説明の書き出しにあたるトークンは、上位10個に現れなかった。モデルは、B（No）をはっきり選んでいる。
|
||||
|
||||
**「わからない」と「ほぼ確実に起こらない」が、同じ値になる。** A と B のロジットの差（正なら Yes 寄り）の平均は、次のとおりである。
|
||||
|
||||
| 主張の種類 | ロジットの差 | 確率に直すと |
|
||||
| --- | --- | --- |
|
||||
| 真の事実 | ＋5.6 | 約 1.00 |
|
||||
| ほぼ確実に起こる未来 | ＋2.9 | 約 0.95 |
|
||||
| 傾向のある未来 | −3.0 | 約 0.05 |
|
||||
| 五分五分の問い | −3.3 | 約 0.04 |
|
||||
| ほぼ確実に起こらない未来 | −3.3 | 約 0.04 |
|
||||
| 偽の事実 | −6.2 | 約 0.002 |
|
||||
|
||||
**値は、確率の高さの順に並ばない。** 確率のわかっている問いへの Q1 の答えは、次のとおりである。10文での順位相関は −0.38 だった。
|
||||
|
||||
| 主張 | 正しい確率 | Q1 の値 |
|
||||
| --- | --- | --- |
|
||||
| トランプでエースを引く | 0.08 | 0.03 |
|
||||
| サイコロで1が出る | 0.17 | 0.02 |
|
||||
| トランプでハートを引く | 0.25 | 0.03 |
|
||||
| サイコロで3〜6が出る | 0.67 | 0.35 |
|
||||
| トランプでハート以外を引く | 0.75 | 0.02 |
|
||||
| サイコロで1以外が出る | 0.83 | 0.01 |
|
||||
| トランプでエース以外を引く | 0.92 | 0.02 |
|
||||
|
||||
Qwen3.5-9B は、「真ですか？」に対して、「確実に真だと言えるか」という基準で答えていると考えられる。出てくる値は、出来事の確率ではなく、No という答えへの自信である。値の中に確率の高さの情報がないので、Q1 の値を後から直して確率にすること（較正）は難しい。
|
||||
|
||||
### 3.3 質問文を変える（Q2）
|
||||
|
||||
質問を「偽であるより真である可能性のほうが高いですか？」に変えると、次のようになった。
|
||||
|
||||
- Jev と Clef-Flash は、Q1 とほぼ同じ傾向だった。
|
||||
- Qwen3.5-9B は、「すべて No」ではなくなった。確率のわかっている問いでは、10文中8文で正しい側（0.5 より上か下か）を選んだ。
|
||||
- ただし、Qwen3.5-9B の値は 0 か 1 に近く、確率の程度を表さない。五分五分の問いは、問いごとに Yes か No のどちらかに振れた（0.02〜0.99、0.5 からのずれは 0.36）。
|
||||
|
||||
汎用モデルの答えは、質問の言い回しに強く左右される。
|
||||
|
||||
### 3.4 主張と否定形の両方について聞く
|
||||
|
||||
Qwen3.5-9B は、Q1 で「ほぼ確実に起こる」には Yes、「わからない」には No と答える。区別できないのは、「わからない」と「ほぼ確実に起こらない」である。後者の否定形は「ほぼ確実に起こる」なので、否定形についても聞けば区別できる。
|
||||
|
||||
| 主張の種類 | 主張への答え | 否定形への答え |
|
||||
| --- | --- | --- |
|
||||
| ほぼ確実に起こる | Yes | No |
|
||||
| ほぼ確実に起こらない | No | Yes |
|
||||
| わからない | No | No |
|
||||
|
||||
設問は対で作ってあるので、対の相手を否定形とみなして、否定形と合わせた値を計算した（Q1）。かっこ内は、主張だけを聞いたときの値である。太字の斜体は、3.1 節で基準から外れていた項目である。
|
||||
|
||||
| グループ | Jev | Clef-Flash | Qwen3.5-9B |
|
||||
| --- | --- | --- | --- |
|
||||
| 既知の事実 | 0.95（0.92） | 0.96（0.95） | 0.99（0.99） |
|
||||
| ほぼ確実な未来 | 0.90（0.88） | 0.88（0.86） | 0.86（0.85） |
|
||||
| 傾向のある未来 | 0.60（0.58） | 0.63（0.62） | ***0.49（0.07）*** |
|
||||
| 五分五分の問い | 0.50（0.44） | 0.55（0.36） | ***0.51（0.07）*** |
|
||||
|
||||
| 0.5 からのずれ（五分五分の問い） | Jev | Clef-Flash | Qwen3.5-9B |
|
||||
| --- | --- | --- | --- |
|
||||
| 主張だけ | 0.06 | 0.22 | ***0.44*** |
|
||||
| 否定形と合わせる | 0.006 | 0.053 | ***0.015*** |
|
||||
|
||||
Qwen3.5-9B について、2つの答えを 0.5 で区切って「真 / 偽 / わからない」に分けると、31組中29組が期待どおりになった。
|
||||
|
||||
- 既知の事実と、ほぼ確実な未来（12組）：10組が「真」。
|
||||
- 傾向のある未来、五分五分の問い、確率のわかっている問い（19組）：19組すべてが「わからない」。
|
||||
|
||||
外れた2組は、「来年1月に札幌で雪が降る / 降らない」（両方に No）と、「皆既日食が5年以内に起こる / 起こらない」（両方に Yes）である。両方に Yes という答えは、その答えが信頼できないことの目印になる。
|
||||
|
||||
この方法では、確率の程度は取り出せない。確率が 0.08 の主張も 0.92 の主張も、否定形と合わせた値は 0.5 付近になる（確率のわかっている問いでの誤差の平均は 0.244、順位相関は −0.12）。
|
||||
|
||||
### 3.5 確率の段階を選ばせる
|
||||
|
||||
以下は、Qwen3.5-9B だけの結果である。
|
||||
|
||||
**対照：数字で書かせると、ほぼ正確である。** 確率のわかっている10文での誤差の平均は 0.003 だった（たとえば、正しい確率 7.7% に「9」、16.7% に「16.67」、92.3% に「92」）。コイン投げには「50」と答えた。モデルは確率の知識を持っている。Q1 で確率の高さが出ないのは、聞き方による。
|
||||
|
||||
| 主張 | 正しい確率 | 書かれた数字 |
|
||||
| --- | --- | --- |
|
||||
| トランプでエースを引く | 7.7% | 9 |
|
||||
| サイコロで1が出る | 16.7% | 16.67 |
|
||||
| トランプでハートを引く | 25% | 25 |
|
||||
| コイン2回とも表 | 25% | 25 |
|
||||
| サイコロで1か2が出る | 33.3% | 33 |
|
||||
| サイコロで3〜6が出る | 66.7% | 66 |
|
||||
| トランプでハート以外を引く | 75% | 75 |
|
||||
| コイン2回とも表、にはならない | 75% | 75 |
|
||||
| サイコロで1以外が出る | 83.3% | 83 |
|
||||
| トランプでエース以外を引く | 92.3% | 92 |
|
||||
| 五分五分の問い（コイン、サイコロ、生まれた時期、砂粒の4文） | 50% | すべて 50 |
|
||||
| 明日、太陽は昇る / 昇らない | — | 100 / 0 |
|
||||
| パリはフランスの首都である / ドイツの首都である | — | 100 / 0 |
|
||||
| S&P 500 は1年後に高い / 低い | — | 65 / 50 |
|
||||
|
||||
**段階で聞くと、確率の高さがある程度出る。** 確率のわかっている10文での結果は、次のとおりである。
|
||||
|
||||
| 聞き方 | 順位相関 | 誤差の平均 |
|
||||
| --- | --- | --- |
|
||||
| Q1（Yes / No） | −0.38 | — |
|
||||
| S6、低い順 | ＋0.88 | 0.095 |
|
||||
| S6、高い順 | ＋0.76 | 0.208 |
|
||||
| S11、低い順 | ＋0.77 | 0.202 |
|
||||
| S11、高い順 | ＋0.80 | 0.203 |
|
||||
| 数字で書かせる | ＋1.00 | 0.003 |
|
||||
|
||||
段階で聞く2通りの質問を、もう一度示す。どちらも質問文は「How likely is this statement to be true?」で、選択肢だけが違う。
|
||||
|
||||
- **S6（6段階の幅）**：0-10%、10-30%、30-50%、50-70%、70-90%、90-100%。推定値には、各段階の中央の値（0.05、0.20、0.40、0.60、0.80、0.95）を使う。
|
||||
- **S11（11段階の点）**：0%、10%、20%、…、100%。推定値には、その点の値（0、0.1、…、1.0）を使う。
|
||||
|
||||
**6段階の幅（S6）と11段階の点（S11）は、得意な点が違う。**
|
||||
|
||||
| 指標 | S6 | S11 |
|
||||
| --- | --- | --- |
|
||||
| 0.5 からのずれ（五分五分の問い、低い順 / 高い順） | 0.118 / 0.140 | 0.027 / 0.029 |
|
||||
| 並び順による推定値の差（62文の平均） | 0.087 | 0.042 |
|
||||
| 対の合計：既知の事実（並び順2通りの平均） | 1.07 | 1.03 |
|
||||
| 対の合計：ほぼ確実な未来（同上） | 1.14 | 1.04 |
|
||||
|
||||
- S6 は、五分五分の問いが約 0.60 になった。50% が「30-50%」と「50-70%」の境目にあたり、モデルが16文中14文で「50-70%」を選んだためである。
|
||||
- S6 は、並び順に強く左右された。「ハート以外を引く」（正しくは 0.75）は、低い順で 0.75、高い順で 0.24 だった。
|
||||
- S11 は、五分五分の問い16文中14文が 0.50 になり、対の合計も 1 に近い。主張だけを聞いても、基準1と基準2をほぼ満たす。
|
||||
- S11 は、確率の程度がほとんど出ない。モデルが、確実でない主張のほとんどに「50%」を選ぶためである。確率が 0.25 の「ハートを引く」にも、0.75 の「ハート以外を引く」にも、「50%」を選んだ。
|
||||
|
||||
「50%」という選択肢があると、五分五分の問いには正しく答えられるが、確実でない問いがそこに集まる。S11 は、「真 / 偽 / わからない」の3つに分ける方法に近い働きをする。
|
||||
|
||||
**6段階の幅で、主張と否定形の両方について聞くと、両方に対応できる。**
|
||||
|
||||
| 聞き方（S6） | 0.5 からのずれ | 誤差の平均 | 順位相関 |
|
||||
| --- | --- | --- | --- |
|
||||
| 低い順、主張だけ | 0.118 | 0.095 | ＋0.88 |
|
||||
| 低い順、否定形と合わせる | 0.027 | 0.069 | ＋0.90 |
|
||||
| 高い順、否定形と合わせる | 0.034 | 0.148 | ＋0.92 |
|
||||
|
||||
五分五分の問いのずれは、主張にも否定形にも同じ向きに出るので、合わせると打ち消し合う。段階は低い順に並べるほうがよい。
|
||||
|
||||
| 主張 | 正しい確率 | 主張だけ | 否定形と合わせる |
|
||||
| --- | --- | --- | --- |
|
||||
| トランプでエースを引く | 0.08 | 0.06 | 0.13 |
|
||||
| サイコロで1が出る | 0.17 | 0.05 | 0.19 |
|
||||
| トランプでハートを引く | 0.25 | 0.32 | 0.29 |
|
||||
| コイン2回とも表 | 0.25 | 0.06 | 0.12 |
|
||||
| サイコロで1か2が出る | 0.33 | 0.53 | 0.44 |
|
||||
|
||||
### 3.6 「偽ですか？」は、否定形の代わりにならない
|
||||
|
||||
否定形を書く代わりに、主張はそのままにして、質問を反対向きにする方法（F1、F6）を試した。合わせた値は、「（Q1 の値 ＋ 1 − F1 の値）÷ 2」で計算した。
|
||||
|
||||
| グループ | 否定形の主張と合わせる | F1 と合わせる |
|
||||
| --- | --- | --- |
|
||||
| 既知の事実 | 0.99 | 0.99 |
|
||||
| ほぼ確実な未来 | 0.86 | 0.87 |
|
||||
| 傾向のある未来 | 0.49 | 0.36 |
|
||||
| 五分五分の問い | 0.51 | 0.42 |
|
||||
|
||||
| 指標 | 否定形の主張と合わせる | 「偽」の質問と合わせる |
|
||||
| --- | --- | --- |
|
||||
| 0.5 からのずれ（Yes / No） | 0.015 | 0.109 |
|
||||
| 0.5 からのずれ（6段階、低い順） | 0.027 | 0.072 |
|
||||
| 誤差の平均（6段階、低い順） | 0.069 | 0.146 |
|
||||
|
||||
確実な問いでは差がない。崩れるのは、不確かな問いである。Qwen3.5-9B は、不確かな主張に「真ですか？」と聞くとほぼ必ず No と答えるが、「偽ですか？」には Yes と答えることがあった（「コインは裏が出る」に、Q1 は 0.03、F1 は 0.60）。不確かな主張を「真だ」とは言わないが、「偽だ」とは言いやすい。
|
||||
|
||||
否定形の主張が効くのは、同じ質問文で聞くので、質問文による偏りが両方に同じようにかかり、打ち消し合うためと考えられる。
|
||||
|
||||
### 3.7 聞き方のまとめ（Qwen3.5-9B）
|
||||
|
||||
| 聞き方 | 五分五分を 0.5 に | 確率の程度 | 否定形を書く | 質問の数 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Yes / No、主張だけ | できない | 出ない | 不要 | 1 |
|
||||
| Yes / No、否定形と合わせる | できる | 出ない | 必要 | 2 |
|
||||
| 11段階の点、主張だけ | できる | ほぼ出ない | 不要 | 1 |
|
||||
| 6段階の幅、主張だけ | ずれる（約 0.60） | ある程度出る | 不要 | 1 |
|
||||
| 6段階の幅、否定形と合わせる | できる | ある程度出る | 必要 | 2 |
|
||||
| 「偽」の質問と合わせる | ずれる | 出ない | 不要 | 2 |
|
||||
| 数字で書かせる | できる | 出る | 不要 | 文章の生成が必要 |
|
||||
|
||||
## 4. 汎用モデルを使うときの指針
|
||||
|
||||
以上から、汎用のローカルモデルで `jev.*` を使うスクリプトについて、次のように言える。
|
||||
|
||||
1. **`jev.noul` や `jev.yes` の低い値を、「起こりにくい」と読まない。** 「真だとは言えない」を含む。判断材料が足りない入力は、ほぼ常に No 側に分岐する。
|
||||
2. **「真 / 偽 / わからない」がわかればよいなら、11段階の点で1回聞く。**
|
||||
3. **Yes / No で聞くなら、主張と否定形の両方について聞く。** 両方が No なら「わからない」、両方が Yes なら「信頼できない答え」として扱う。
|
||||
4. **確率の程度も知りたいなら、6段階の幅を低い順に並べ、主張と否定形の両方について聞く。**
|
||||
5. **否定形は、主張として書く。** 質問を「偽ですか？」に変える方法は、代わりにならない。そのため、否定形の質問をノードの側で自動的に作る方法は、いまのところない。
|
||||
6. **聞きたいことを、そのまま質問文にする。** 「真か」と「起こりやすいほうか」では、答えが大きく変わる。
|
||||
7. **モデルを替えたら、しきい値を見直す。** 同じスクリプトでも、Jev、Clef-Flash、汎用モデルで、不確かな入力の分岐が変わる。
|
||||
|
||||
主張と否定形の両方について聞く例。数値の出力（`ov1` など）は整数なので、否定形と合わせた値は 100 倍して 0〜100 で出力する。
|
||||
|
||||
```python
|
||||
# it1: 主張、it2: その否定形
|
||||
# ov1: 否定形と合わせた値（0〜100）、ot1: 分類
|
||||
a = jev.noul(it1, "Is this statement true?")
|
||||
b = jev.noul(it2, "Is this statement true?")
|
||||
ov1 = (a + 1 - b) / 2 * 100
|
||||
|
||||
if a < 0.5 and b < 0.5:
|
||||
ot1 = "unsure"
|
||||
elif a >= 0.5 and b >= 0.5:
|
||||
ot1 = "contradiction"
|
||||
elif a >= 0.5:
|
||||
ot1 = "true"
|
||||
else:
|
||||
ot1 = "false"
|
||||
```
|
||||
|
||||
11段階の点で聞く例。`jev.score` は選ばれた段階の番号の期待値（0〜10）を返すので、10 倍すると 0〜100 の値になる。
|
||||
|
||||
```python
|
||||
levels = ["0%", "10%", "20%", "30%", "40%", "50%", "60%", "70%", "80%", "90%", "100%"]
|
||||
ov1 = jev.score(it1, "How likely is this statement to be true?", levels) * 10
|
||||
```
|
||||
|
||||
6段階の幅では、段階の中央の値が等間隔でないので、`jev.score` の値を 20 倍しても、3.5 節の推定値（の 100 倍）とは同じにならない。近似として使う場合は、その点に注意する。
|
||||
|
||||
## 5. 限界
|
||||
|
||||
- 31組62文の小さな試験である。確率のわかっている問いは5組10文だけで、聞き方の比較はこの少ない問いにもとづく。
|
||||
- モデルの優劣を評価したものではない。未来の問いには正解がなく、予測の正しさは測っていない。
|
||||
- 否定形と合わせる方法は、このデータを見て思いついたもので、同じデータで効果を確かめている。別の設問で確かめてはいない。
|
||||
- 設問の対は、厳密な否定になっていないものがある（「高い」と「低い」など）。否定形の作り方で、結果は変わりうる。
|
||||
- 段階で聞く方法（S6、S11）、否定形と合わせる方法の段階での効果、「偽」の質問（F1、F6）は、Qwen3.5-9B でしか試していない。Jev と Clef-Flash には聞いていない。
|
||||
- 段階の作り方は、2通りしか試していない。
|
||||
- 汎用モデルは Qwen3.5-9B の Q8_0 だけである。別のモデル、別の量子化、別の判定方式では、違う結果になりうる。
|
||||
- 汎用モデルの値は、較正をしていない。SemIf (OpenJev) は、用途ごとに較正して使うよう勧めているが、今回は較正を試していない。3.2 節のとおり、Yes / No の値には確率の高さの情報がなく、較正に使える設問も用意できていない。
|
||||
- どの質問も1回ずつの測定である。ローカルのモデルの値は、サンプリングをしないので、同じ条件なら変わらない。Jev は、同じ質問でも値が少し変わる。五分五分の問い16文を Q1 で3回聞いたところ、値の幅は最大で 0.04 だった。
|
||||
- 主張と質問は英語である。ほかの言語では試していない。
|
||||
|
||||
## 6. データ
|
||||
|
||||
[`data/`](data/) に、設問と結果を置いている。主張の ID は、対の記号に `a`（1つめ）か `b`（2つめ）を付けたものである（例：`U1a`）。
|
||||
|
||||
| ファイル | 内容 |
|
||||
| --- | --- |
|
||||
| `items.json` | 設問（`pairs`）、質問文（`questions`）、グループの説明（`groups`）、確率のわかっている問いの正しい確率（`probabilities`） |
|
||||
| `results_jev.json` | Jev。Q1・Q2 の `p_yes`。五分五分の問いの Q1 は、`run` 1〜3 の3回分 |
|
||||
| `results_clef_flash.json` | Clef-Flash。Q1・Q2 の `p_yes` |
|
||||
| `results_qwen35.json` | Qwen3.5-9B。Q1・Q2 の `p_yes`（26組） |
|
||||
| `results_qwen35_mass.json` | Qwen3.5-9B。Q1・Q2（31組）の `p_yes`、A と B のロジット、`allowed_token_mass`、確率の高い上位10トークン |
|
||||
| `results_qwen35_score.json` | Qwen3.5-9B。S6 の段階ごとの確率（`score`）と、数字で書かせた対照の答え（`text`） |
|
||||
| `results_qwen35_score11.json` | Qwen3.5-9B。S11 の段階ごとの確率 |
|
||||
| `results_qwen35_false.json` | Qwen3.5-9B。F1 の `p_yes`（`noul`）と、F6 の段階ごとの確率（`score`） |
|
||||
|
||||
段階ごとの確率（`p`）は、選択肢をどちらの順で示した場合も、低い段階から順に並べて保存している。`order` は、示した順である。
|
||||
|
||||
## 7. 測定の条件
|
||||
|
||||
- 実施日：2026年10月5日（3つのモデルの比較）、10月6日〜8日（Qwen3.5-9B の追加の測定）
|
||||
- ノード：ComfyUI-ScriptFlow v1.3.3
|
||||
- Jev：TypeSafe API（typesafe-sdk 0.7.1）。API が返すモデル名は、2026年10月8日に確認した時点で `jev-1.13.0` だった。測定時（10月5日）のモデル名は記録していない。
|
||||
- ローカルのモデルの実行：llama-cpp-python 0.4.2（SYCL ビルド）、Windows 11
|
||||
@@ -0,0 +1,384 @@
|
||||
# How general local models answer uncertain questions: `jev.*` behavior and how to ask
|
||||
[en | [ja](README.ja.md)]
|
||||
|
||||
Technical report (October 2026)
|
||||
|
||||
## Summary
|
||||
|
||||
`MultiOutputScript (Jev)` can make decisions with a local GGUF model instead of TypeSafe's Jev. This report compares how Jev, Clef-Flash, and a general model (Qwen3.5-9B) answer questions whose answer cannot be known, and looks at how to ask when a general model is used.
|
||||
|
||||
- On questions with a certain answer, the three models gave almost the same answers.
|
||||
- On questions with an uncertain answer, they behaved differently. When 50/50 statements were asked "Is this statement true?" with `jev.noul`, the mean was 0.44 for Jev, 0.31 for Clef-Flash, and 0.06 for Qwen3.5-9B.
|
||||
- A low value from Qwen3.5-9B means "cannot say it is true", not "unlikely to happen". It answered 0.02 for a statement whose probability is 0.92.
|
||||
- Asking about both a statement and its negation, and combining the two answers, brought the 50/50 statements to 0.51 with Qwen3.5-9B as well.
|
||||
- Having the model pick a probability level (`jev.score`) recovered the size of the probability to some degree. The result depended on how the levels were defined and on their order.
|
||||
- Reversing the question to "Is this statement false?" did not work as a substitute for the negation.
|
||||
|
||||
This is a small test of 31 pairs (62 statements), and the level-based questions were tried with one quantization of Qwen3.5-9B only. It shows tendencies.
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
The output of a decision model is often cut at a threshold and used to branch (for example `jev.yes(state, question, 0.5)`). When some inputs do not give enough to decide, the branch taken depends on which value the model uses to express "I cannot tell".
|
||||
|
||||
This report checks two things:
|
||||
|
||||
1. How a general local model answers uncertain questions differently from Jev.
|
||||
2. How much of that difference can be closed by how the script asks.
|
||||
|
||||
Two properties expected of a model that answers with probabilities were used as criteria:
|
||||
|
||||
- **Criterion 1**: the probabilities of the two statements of a pair add up to 1.
|
||||
- **Criterion 2**: a 50/50 statement gets 0.5.
|
||||
|
||||
## 2. Method
|
||||
|
||||
### 2.1 Models
|
||||
|
||||
| Model | Kind | How the answer is produced |
|
||||
| --- | --- | --- |
|
||||
| Jev | TypeSafe's decision model (API) | The API returns probabilities |
|
||||
| Clef-Flash | Qwen3.5-9B further trained for decisions (`ggml-org/Clef-Flash-GGUF`, Q8_0) | The values from the decision head are turned into probabilities |
|
||||
| Qwen3.5-9B | General LLM (`unsloth/Qwen3.5-9B-GGUF`, Q8_0) | The probabilities of the option letters as the next token are read |
|
||||
|
||||
For a general model, the options are shown with letters (for `jev.noul`, A: Yes and B: No) and the model is told to answer with one letter only. No text is generated. The probability of each letter as the next token is read, and the values are rescaled so that the letters add up to 1 (any other output is left out). This is the method proposed by SemIf (OpenJev). The values in this report are uncalibrated. Calibration is discussed in sections 3.2 and 5. Because nothing is sampled, there is no random seed or temperature setting.
|
||||
|
||||
The instruction is as follows. The statement, the question, and the options are passed together as JSON.
|
||||
|
||||
```text
|
||||
Apply the supplied criterion to the supplied evidence. Choose exactly one listed option. Respond with only its uppercase letter, with no explanation or reasoning.
|
||||
```
|
||||
|
||||
All three models were run through this node's own decision code (v1.3.3). Questions were sent one at a time.
|
||||
|
||||
### 2.2 Statements
|
||||
|
||||
There are 31 pairs of statements, 62 statements in total. Statements and questions are in English. The full list is in [`data/items.json`](data/items.json).
|
||||
|
||||
| Group | Code | Pairs | Content | Example |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Known facts | K | 6 | The first is true, the second false | Paris is the capital of France. / ... of Germany. |
|
||||
| Near-certain future | N | 6 | The first is almost sure to happen | The sun will rise tomorrow. / ... will not rise tomorrow. |
|
||||
| Future with a tendency | T | 6 | The first is more likely, but not certain | The S&P 500 index will be higher one year from now ... / lower ... |
|
||||
| 50/50 | U | 8 | Both are about 50% | The next coin I flip will land heads. / ... tails. |
|
||||
| Known probability | P | 5 | The exact probability can be calculated | The next roll of a fair six-sided die will show a 1. (1/6) / ... a number other than 1. (5/6) |
|
||||
|
||||
The 26 pairs of K, N, T, and U were asked to all three models. The 5 pairs of P were added later and asked to Qwen3.5-9B only.
|
||||
|
||||
### 2.3 Questions
|
||||
|
||||
| Code | Function | Question | Options |
|
||||
| --- | --- | --- | --- |
|
||||
| Q1 | `jev.noul` | Is this statement true? | Yes / No |
|
||||
| Q2 | `jev.noul` | Is this statement more likely to be true than false? | Yes / No |
|
||||
| F1 | `jev.noul` | Is this statement false? | Yes / No |
|
||||
| S6 | `jev.score` | How likely is this statement to be true? | 0-10%, 10-30%, 30-50%, 50-70%, 70-90%, 90-100% |
|
||||
| S11 | `jev.score` | How likely is this statement to be true? | 0%, 10%, ..., 100% |
|
||||
| F6 | `jev.score` | How likely is this statement to be false? | The same six levels as S6 |
|
||||
|
||||
Q1 and Q2 were asked to all three models. F1, S6, S11, and F6 were asked to Qwen3.5-9B only. The level-based questions were asked twice, with the options in ascending and in descending order.
|
||||
|
||||
As a control, Qwen3.5-9B was also asked to write the probability as a number (20 statements). The instruction was "What is the probability that the following statement is true? Answer with a single number from 0 to 100 (percent) and nothing else.", and the text was generated by taking the most likely token at each step.
|
||||
|
||||
### 2.4 Measures
|
||||
|
||||
- **Pair sum**: the sum of the values for the two statements of a pair (criterion 1).
|
||||
- **Distance from 0.5**: the mean absolute difference between the value and 0.5 over the 16 50/50 statements (criterion 2).
|
||||
- **Mean error**: the mean absolute difference between the value and the true probability over the 10 known-probability statements.
|
||||
- **Rank correlation**: Spearman's rank correlation between the value and the true probability over the same 10 statements.
|
||||
- **Negation-combined value**: (value for the statement + 1 − value for the negation) / 2. The pair sum is 1 by construction.
|
||||
- **Estimate from levels**: the value of each level (for S6 and F6 the midpoints 0.05, 0.20, 0.40, 0.60, 0.80, 0.95; for S11 the points 0, 0.1, ..., 1.0) weighted by the probability the model gave that level.
|
||||
|
||||
Negation-combined values were not measured separately. They are calculated from the values already measured.
|
||||
|
||||
## 3. Results
|
||||
|
||||
### 3.1 The three models compared (Q1)
|
||||
|
||||
Values are mean yes-probabilities, with the first statement of the pair on the left and the second on the right. The results of Qwen3.5-9B for "Future with a tendency" and "50/50", shown in bold italics, are far from criteria 1 and 2 of section 1, and are examined further from section 3.2 on.
|
||||
|
||||
| Group | Jev | Clef-Flash | Qwen3.5-9B |
|
||||
| --- | --- | --- | --- |
|
||||
| Known facts | 0.92 / 0.03 | 0.95 / 0.02 | 0.99 / 0.01 |
|
||||
| Near-certain future | 0.88 / 0.07 | 0.86 / 0.11 | 0.85 / 0.13 |
|
||||
| Future with a tendency | 0.58 / 0.37 | 0.62 / 0.36 | ***0.07 / 0.08*** |
|
||||
| 50/50 | 0.44 / 0.44 | 0.36 / 0.26 | ***0.07 / 0.04*** |
|
||||
|
||||
| Measure | Jev | Clef-Flash | Qwen3.5-9B |
|
||||
| --- | --- | --- | --- |
|
||||
| Pair sum: known facts | 0.95 | 0.97 | 1.00 |
|
||||
| Pair sum: near-certain future | 0.95 | 0.96 | 0.98 |
|
||||
| Pair sum: future with a tendency | 0.95 | 0.98 | ***0.15*** |
|
||||
| Pair sum: 50/50 | 0.88 | 0.62 | ***0.11*** |
|
||||
| Distance from 0.5 (50/50) | 0.06 | 0.22 | ***0.44*** |
|
||||
|
||||
- For known facts, all three models gave a value on the correct side for all 12 statements.
|
||||
- Jev returned 0.34 to 0.51 for all 16 50/50 statements.
|
||||
- Qwen3.5-9B answered No to almost every uncertain statement (future with a tendency, 50/50). It does not answer No to everything about the future: it returned 0.99 for "The sun will rise tomorrow." (near-certain future).
|
||||
- Clef-Flash had a pair sum of 0.98 for the future with a tendency, very different from Qwen3.5-9B before tuning (0.15). For questions like coins and dice it stayed at 0.62 (the coin flip was 0.16 / 0.10).
|
||||
|
||||
Clef-Flash and Qwen3.5-9B also differ in how the answer is produced. This test cannot separate whether the difference between the two comes from the tuned weights or from the way the answer is produced.
|
||||
|
||||
Clef-Flash chose one side clearly for the future with a tendency (0.80 / 0.06 for the S&P 500, where Jev gave 0.53 / 0.31). However, it returned 0.09 / 0.81 for "It will not rain / will rain in Tokyo on this day next year", the opposite of the actual frequency, and 0.81 / 0.71 for "The next US president will be older / younger than 50 when taking office", answering Yes to both.
|
||||
|
||||
### 3.2 What the No of Qwen3.5-9B represents
|
||||
|
||||
From here on, the results of Qwen3.5-9B for "Future with a tendency" and "50/50" are examined.
|
||||
|
||||
The first check was that the No of Qwen3.5-9B is **not the result of probability going to tokens other than the letters**. The share of the whole vocabulary's probability taken by A and B (`allowed_token_mass` in SemIf) was recorded for 124 questions: 62 statements × Q1 and Q2.
|
||||
|
||||
| | Minimum | Median | Maximum |
|
||||
| --- | --- | --- | --- |
|
||||
| Share taken by A and B | 0.9971 | 0.9992 | 0.9999 |
|
||||
|
||||
The most likely token other than A and B was "C", at 0.15% at most. Tokens that would begin a refusal or an explanation did not appear among the top 10. The model chooses B (No) clearly.
|
||||
|
||||
**"Cannot tell" and "almost surely will not happen" get the same value.** The mean difference between the logits of A and B (positive means toward Yes) is as follows.
|
||||
|
||||
| Kind of statement | Logit difference | As a probability |
|
||||
| --- | --- | --- |
|
||||
| True fact | +5.6 | about 1.00 |
|
||||
| Near-certain to happen | +2.9 | about 0.95 |
|
||||
| Future with a tendency | −3.0 | about 0.05 |
|
||||
| 50/50 | −3.3 | about 0.04 |
|
||||
| Near-certain not to happen | −3.3 | about 0.04 |
|
||||
| False fact | −6.2 | about 0.002 |
|
||||
|
||||
**The values do not follow the order of the probabilities.** The Q1 answers for known-probability statements are below. The rank correlation over the 10 statements was −0.38.
|
||||
|
||||
| Statement | True probability | Q1 value |
|
||||
| --- | --- | --- |
|
||||
| Drawing an ace from a deck | 0.08 | 0.03 |
|
||||
| Rolling a 1 | 0.17 | 0.02 |
|
||||
| Drawing a heart | 0.25 | 0.03 |
|
||||
| Rolling a 3, 4, 5, or 6 | 0.67 | 0.35 |
|
||||
| Drawing a card that is not a heart | 0.75 | 0.02 |
|
||||
| Rolling a number other than 1 | 0.83 | 0.01 |
|
||||
| Drawing a card that is not an ace | 0.92 | 0.02 |
|
||||
|
||||
Qwen3.5-9B appears to answer "Is this statement true?" by the standard "can it be said to be true for certain?". The value that comes out is not the probability of the event but the confidence in the answer No. Because the value carries no information about the size of the probability, turning Q1 values into probabilities afterwards (calibration) is difficult.
|
||||
|
||||
### 3.3 Changing the question (Q2)
|
||||
|
||||
With the question changed to "Is this statement more likely to be true than false?":
|
||||
|
||||
- Jev and Clef-Flash showed almost the same tendency as with Q1.
|
||||
- Qwen3.5-9B no longer answered No to everything. For the known-probability statements it chose the correct side (above or below 0.5) for 8 of 10.
|
||||
- However, the values of Qwen3.5-9B were close to 0 or 1 and did not express the size of the probability. The 50/50 statements swung to Yes or No depending on the statement (0.02 to 0.99; distance from 0.5 was 0.36).
|
||||
|
||||
The answers of a general model depend strongly on the wording of the question.
|
||||
|
||||
### 3.4 Asking about both the statement and its negation
|
||||
|
||||
With Q1, Qwen3.5-9B answers Yes to "near-certain to happen" and No to "cannot tell". What it does not separate is "cannot tell" from "near-certain not to happen". The negation of the latter is "near-certain to happen", so asking about the negation as well separates them.
|
||||
|
||||
| Kind of statement | Answer for the statement | Answer for the negation |
|
||||
| --- | --- | --- |
|
||||
| Near-certain to happen | Yes | No |
|
||||
| Near-certain not to happen | No | Yes |
|
||||
| Cannot tell | No | No |
|
||||
|
||||
The statements were written as pairs, so the other statement of the pair was taken as the negation and the negation-combined value was calculated (Q1). The value in parentheses is from asking about the statement alone. Bold italics mark the items that were far from the criteria in section 3.1.
|
||||
|
||||
| Group | Jev | Clef-Flash | Qwen3.5-9B |
|
||||
| --- | --- | --- | --- |
|
||||
| Known facts | 0.95 (0.92) | 0.96 (0.95) | 0.99 (0.99) |
|
||||
| Near-certain future | 0.90 (0.88) | 0.88 (0.86) | 0.86 (0.85) |
|
||||
| Future with a tendency | 0.60 (0.58) | 0.63 (0.62) | ***0.49 (0.07)*** |
|
||||
| 50/50 | 0.50 (0.44) | 0.55 (0.36) | ***0.51 (0.07)*** |
|
||||
|
||||
| Distance from 0.5 (50/50) | Jev | Clef-Flash | Qwen3.5-9B |
|
||||
| --- | --- | --- | --- |
|
||||
| Statement alone | 0.06 | 0.22 | ***0.44*** |
|
||||
| Combined with the negation | 0.006 | 0.053 | ***0.015*** |
|
||||
|
||||
For Qwen3.5-9B, cutting the two answers at 0.5 and sorting each pair into "true / false / cannot tell" gave the expected result for 29 of 31 pairs.
|
||||
|
||||
- Known facts and near-certain future (12 pairs): 10 pairs were "true".
|
||||
- Future with a tendency, 50/50, and known probability (19 pairs): all 19 pairs were "cannot tell".
|
||||
|
||||
The two pairs that missed were "It will snow / will not snow in Sapporo next January" (No to both) and "A total solar eclipse will / will not occur within the next five years" (Yes to both). A Yes to both is a sign that the answer cannot be trusted.
|
||||
|
||||
This method does not recover the size of the probability. A statement with probability 0.08 and one with 0.92 both get a negation-combined value near 0.5 (for the known-probability statements the mean error was 0.244 and the rank correlation −0.12).
|
||||
|
||||
### 3.5 Having the model pick a probability level
|
||||
|
||||
The results below are for Qwen3.5-9B only.
|
||||
|
||||
**Control: written as a number, the answer is almost exact.** The mean error over the 10 known-probability statements was 0.003 (for example "9" for a true probability of 7.7%, "16.67" for 16.7%, and "92" for 92.3%). For the coin flip it answered "50". The model has the knowledge of the probabilities. That the size of the probability does not show with Q1 comes from how the question is asked.
|
||||
|
||||
| Statement | True probability | Number written |
|
||||
| --- | --- | --- |
|
||||
| Drawing an ace from a deck | 7.7% | 9 |
|
||||
| Rolling a 1 | 16.7% | 16.67 |
|
||||
| Drawing a heart | 25% | 25 |
|
||||
| Two heads in two coin flips | 25% | 25 |
|
||||
| Rolling a 1 or a 2 | 33.3% | 33 |
|
||||
| Rolling a 3, 4, 5, or 6 | 66.7% | 66 |
|
||||
| Drawing a card that is not a heart | 75% | 75 |
|
||||
| Not two heads in two coin flips | 75% | 75 |
|
||||
| Rolling a number other than 1 | 83.3% | 83 |
|
||||
| Drawing a card that is not an ace | 92.3% | 92 |
|
||||
| 50/50 statements (4: coin, die, time of birth, grains of sand) | 50% | 50 for all |
|
||||
| The sun will rise / will not rise tomorrow | — | 100 / 0 |
|
||||
| Paris is the capital of France / of Germany | — | 100 / 0 |
|
||||
| The S&P 500 will be higher / lower in one year | — | 65 / 50 |
|
||||
|
||||
**Asked by level, the size of the probability shows to some degree.** The results for the 10 known-probability statements are as follows.
|
||||
|
||||
| How it was asked | Rank correlation | Mean error |
|
||||
| --- | --- | --- |
|
||||
| Q1 (Yes / No) | −0.38 | — |
|
||||
| S6, ascending | +0.88 | 0.095 |
|
||||
| S6, descending | +0.76 | 0.208 |
|
||||
| S11, ascending | +0.77 | 0.202 |
|
||||
| S11, descending | +0.80 | 0.203 |
|
||||
| Written as a number | +1.00 | 0.003 |
|
||||
|
||||
The two level-based questions are shown again here. Both use the question "How likely is this statement to be true?", and only the options differ.
|
||||
|
||||
- **S6 (six ranges)**: 0-10%, 10-30%, 30-50%, 50-70%, 70-90%, 90-100%. The estimate uses the midpoint of each range (0.05, 0.20, 0.40, 0.60, 0.80, 0.95).
|
||||
- **S11 (eleven points)**: 0%, 10%, 20%, ..., 100%. The estimate uses the value of each point (0, 0.1, ..., 1.0).
|
||||
|
||||
**Six ranges (S6) and eleven points (S11) are good at different things.**
|
||||
|
||||
| Measure | S6 | S11 |
|
||||
| --- | --- | --- |
|
||||
| Distance from 0.5 (50/50; ascending / descending) | 0.118 / 0.140 | 0.027 / 0.029 |
|
||||
| Difference in the estimate between the two orders (mean of 62 statements) | 0.087 | 0.042 |
|
||||
| Pair sum: known facts (mean of the two orders) | 1.07 | 1.03 |
|
||||
| Pair sum: near-certain future (mean of the two orders) | 1.14 | 1.04 |
|
||||
|
||||
- With S6, the 50/50 statements came out at about 0.60. 50% lies on the border between "30-50%" and "50-70%", and the model chose "50-70%" for 14 of 16 statements.
|
||||
- S6 depended strongly on the order. "Drawing a card that is not a heart" (true probability 0.75) was 0.75 in ascending order and 0.24 in descending order.
|
||||
- With S11, 14 of the 16 50/50 statements came out at 0.50, and the pair sums are close to 1. Asking about the statement alone almost meets criteria 1 and 2.
|
||||
- With S11, the size of the probability hardly shows, because the model chooses "50%" for most statements that are not certain. It chose "50%" both for "Drawing a heart" (0.25) and for "Drawing a card that is not a heart" (0.75).
|
||||
|
||||
When there is a "50%" option, 50/50 statements are answered correctly, but uncertain statements gather there. S11 works much like sorting into "true / false / cannot tell".
|
||||
|
||||
**Six ranges asked about both the statement and its negation cover both.**
|
||||
|
||||
| How it was asked (S6) | Distance from 0.5 | Mean error | Rank correlation |
|
||||
| --- | --- | --- | --- |
|
||||
| Ascending, statement alone | 0.118 | 0.095 | +0.88 |
|
||||
| Ascending, combined with the negation | 0.027 | 0.069 | +0.90 |
|
||||
| Descending, combined with the negation | 0.034 | 0.148 | +0.92 |
|
||||
|
||||
The shift of the 50/50 statements goes in the same direction for the statement and for the negation, so it cancels when the two are combined. The levels are better listed in ascending order.
|
||||
|
||||
| Statement | True probability | Statement alone | Combined with the negation |
|
||||
| --- | --- | --- | --- |
|
||||
| Drawing an ace from a deck | 0.08 | 0.06 | 0.13 |
|
||||
| Rolling a 1 | 0.17 | 0.05 | 0.19 |
|
||||
| Drawing a heart | 0.25 | 0.32 | 0.29 |
|
||||
| Two heads in two coin flips | 0.25 | 0.06 | 0.12 |
|
||||
| Rolling a 1 or a 2 | 0.33 | 0.53 | 0.44 |
|
||||
|
||||
### 3.6 "Is this statement false?" is not a substitute for the negation
|
||||
|
||||
Instead of writing a negation, the statement was kept as it is and the question was reversed (F1, F6). The combined value was calculated as (Q1 value + 1 − F1 value) / 2.
|
||||
|
||||
| Group | Combined with the negated statement | Combined with F1 |
|
||||
| --- | --- | --- |
|
||||
| Known facts | 0.99 | 0.99 |
|
||||
| Near-certain future | 0.86 | 0.87 |
|
||||
| Future with a tendency | 0.49 | 0.36 |
|
||||
| 50/50 | 0.51 | 0.42 |
|
||||
|
||||
| Measure | Combined with the negated statement | Combined with the "false" question |
|
||||
| --- | --- | --- |
|
||||
| Distance from 0.5 (Yes / No) | 0.015 | 0.109 |
|
||||
| Distance from 0.5 (six ranges, ascending) | 0.027 | 0.072 |
|
||||
| Mean error (six ranges, ascending) | 0.069 | 0.146 |
|
||||
|
||||
There is no difference for certain statements. What breaks is the uncertain ones. Asked "Is this statement true?" about an uncertain statement, Qwen3.5-9B almost always answers No, but asked "Is this statement false?" it sometimes answered Yes (for "The next coin I flip will land tails.", Q1 was 0.03 and F1 was 0.60). It does not say an uncertain statement is "true", but it says "false" more easily.
|
||||
|
||||
The negated statement likely works because it is asked with the same question, so the bias of the question applies to both in the same way and cancels.
|
||||
|
||||
### 3.7 Summary of the ways of asking (Qwen3.5-9B)
|
||||
|
||||
| How it was asked | 50/50 at 0.5 | Size of the probability | Writing a negation | Questions |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Yes / No, statement alone | No | Does not show | Not needed | 1 |
|
||||
| Yes / No, combined with the negation | Yes | Does not show | Needed | 2 |
|
||||
| Eleven points, statement alone | Yes | Hardly shows | Not needed | 1 |
|
||||
| Six ranges, statement alone | Shifted (about 0.60) | Shows to some degree | Not needed | 1 |
|
||||
| Six ranges, combined with the negation | Yes | Shows to some degree | Needed | 2 |
|
||||
| Combined with the "false" question | Shifted | Does not show | Not needed | 2 |
|
||||
| Written as a number | Yes | Shows | Not needed | Needs text generation |
|
||||
|
||||
## 4. Guidelines for using a general model
|
||||
|
||||
From the above, the following can be said for scripts that use `jev.*` with a general local model.
|
||||
|
||||
1. **Do not read a low `jev.noul` or `jev.yes` value as "unlikely".** It includes "cannot say it is true". Inputs that do not give enough to decide almost always take the No branch.
|
||||
2. **If "true / false / cannot tell" is enough, ask once with eleven points.**
|
||||
3. **If you ask Yes / No, ask about both the statement and its negation.** Treat No to both as "cannot tell", and Yes to both as an answer that cannot be trusted.
|
||||
4. **If you also want the size of the probability, list six ranges in ascending order and ask about both the statement and its negation.**
|
||||
5. **Write the negation as a statement.** Changing the question to "Is this statement false?" is not a substitute. For this reason, there is currently no way for the node to create the negated question automatically.
|
||||
6. **Put what you want to know directly into the question.** "Is it true" and "is it the more likely side" give very different answers.
|
||||
7. **Review thresholds when you change the model.** With the same script, the branch taken for uncertain inputs differs between Jev, Clef-Flash, and a general model.
|
||||
|
||||
Asking about both the statement and its negation. Numeric outputs (`ov1` and so on) are integers, so the negation-combined value is multiplied by 100 and output as 0 to 100.
|
||||
|
||||
```python
|
||||
# it1: the statement, it2: its negation
|
||||
# ov1: negation-combined value (0-100), ot1: classification
|
||||
a = jev.noul(it1, "Is this statement true?")
|
||||
b = jev.noul(it2, "Is this statement true?")
|
||||
ov1 = (a + 1 - b) / 2 * 100
|
||||
|
||||
if a < 0.5 and b < 0.5:
|
||||
ot1 = "unsure"
|
||||
elif a >= 0.5 and b >= 0.5:
|
||||
ot1 = "contradiction"
|
||||
elif a >= 0.5:
|
||||
ot1 = "true"
|
||||
else:
|
||||
ot1 = "false"
|
||||
```
|
||||
|
||||
Asking with eleven points. `jev.score` returns the expected index of the chosen level (0 to 10), so multiplying by 10 gives a value from 0 to 100.
|
||||
|
||||
```python
|
||||
levels = ["0%", "10%", "20%", "30%", "40%", "50%", "60%", "70%", "80%", "90%", "100%"]
|
||||
ov1 = jev.score(it1, "How likely is this statement to be true?", levels) * 10
|
||||
```
|
||||
|
||||
With six ranges, the midpoints are not evenly spaced, so multiplying the `jev.score` value by 20 does not give the same value as 100 times the estimate in section 3.5. Keep this in mind when using it as an approximation.
|
||||
|
||||
## 5. Limitations
|
||||
|
||||
- This is a small test of 31 pairs (62 statements). There are only 5 known-probability pairs (10 statements), and the comparison of the ways of asking rests on these few statements.
|
||||
- It does not rank the models. Questions about the future have no correct answer, and the accuracy of the predictions was not measured.
|
||||
- Combining with the negation was thought of while looking at this data, and its effect was checked on the same data. It has not been checked on other statements.
|
||||
- Some pairs are not strict negations (such as "higher" and "lower"). The result can change with how the negation is written.
|
||||
- The level-based questions (S6, S11), the effect of combining with the negation for levels, and the "false" questions (F1, F6) were tried with Qwen3.5-9B only. They were not asked to Jev or Clef-Flash.
|
||||
- Only two ways of defining the levels were tried.
|
||||
- The only general model is Qwen3.5-9B at Q8_0. Another model, another quantization, or another decision method may give different results.
|
||||
- The values of the general model are uncalibrated. SemIf (OpenJev) recommends calibrating for each use, but calibration was not tried here. As section 3.2 shows, the Yes / No values carry no information about the size of the probability, and no statements suitable for calibration have been prepared.
|
||||
- Every question was measured once. The values of a local model do not change under the same conditions, because nothing is sampled. Jev's values vary a little for the same question: when the 16 50/50 statements were asked three times with Q1, the largest range was 0.04.
|
||||
- Statements and questions are in English. Other languages were not tried.
|
||||
|
||||
## 6. Data
|
||||
|
||||
The statements and the results are in [`data/`](data/). A statement ID is the pair code followed by `a` (first statement) or `b` (second statement), for example `U1a`.
|
||||
|
||||
| File | Content |
|
||||
| --- | --- |
|
||||
| `items.json` | Statements (`pairs`), question texts (`questions`), group descriptions (`groups`), and the true probabilities of the known-probability statements (`probabilities`) |
|
||||
| `results_jev.json` | Jev. `p_yes` for Q1 and Q2. Q1 of the 50/50 statements has three runs (`run` 1 to 3) |
|
||||
| `results_clef_flash.json` | Clef-Flash. `p_yes` for Q1 and Q2 |
|
||||
| `results_qwen35.json` | Qwen3.5-9B. `p_yes` for Q1 and Q2 (26 pairs) |
|
||||
| `results_qwen35_mass.json` | Qwen3.5-9B. For Q1 and Q2 (31 pairs): `p_yes`, the logits of A and B, `allowed_token_mass`, and the 10 most likely tokens |
|
||||
| `results_qwen35_score.json` | Qwen3.5-9B. The probability of each level for S6 (`score`) and the answers of the written-number control (`text`) |
|
||||
| `results_qwen35_score11.json` | Qwen3.5-9B. The probability of each level for S11 |
|
||||
| `results_qwen35_false.json` | Qwen3.5-9B. `p_yes` for F1 (`noul`) and the probability of each level for F6 (`score`) |
|
||||
|
||||
The level probabilities (`p`) are stored from the lowest level to the highest, whichever order the options were shown in. `order` is the order shown.
|
||||
|
||||
## 7. Measurement conditions
|
||||
|
||||
- Dates: October 5, 2026 (comparison of the three models), October 6 to 8, 2026 (additional measurements with Qwen3.5-9B)
|
||||
- Node: ComfyUI-ScriptFlow v1.3.3
|
||||
- Jev: TypeSafe API (typesafe-sdk 0.7.1). The model name returned by the API was `jev-1.13.0` when checked on October 8, 2026. The model name at the time of measurement (October 5) was not recorded.
|
||||
- Local models: llama-cpp-python 0.4.2 (SYCL build), Windows 11
|
||||
@@ -0,0 +1,182 @@
|
||||
{
|
||||
"questions": {
|
||||
"Q1": "Is this statement true?",
|
||||
"Q2": "Is this statement more likely to be true than false?"
|
||||
},
|
||||
"groups": {
|
||||
"U": "Unknowable, symmetric: each statement of a pair should be about 50%",
|
||||
"T": "Future with a tendency: one statement of a pair is more likely, but not certain",
|
||||
"N": "Future, near-certain: the first statement of a pair is almost sure",
|
||||
"K": "Known facts (control): the first statement of a pair is true, the second false",
|
||||
"P": "Known probability: the statement has an exact probability, listed under probabilities"
|
||||
},
|
||||
"pairs": [
|
||||
[
|
||||
"U1",
|
||||
"The next coin I flip will land heads.",
|
||||
"The next coin I flip will land tails."
|
||||
],
|
||||
[
|
||||
"U2",
|
||||
"The next roll of a fair six-sided die will show an even number.",
|
||||
"The next roll of a fair six-sided die will show an odd number."
|
||||
],
|
||||
[
|
||||
"U3",
|
||||
"The next card I draw from a well-shuffled standard deck will be red.",
|
||||
"The next card I draw from a well-shuffled standard deck will be black."
|
||||
],
|
||||
[
|
||||
"U4",
|
||||
"A fair spinner with two equal halves labelled A and B will land on A on its next spin.",
|
||||
"A fair spinner with two equal halves labelled A and B will land on B on its next spin."
|
||||
],
|
||||
[
|
||||
"U5",
|
||||
"The last digit of the next winning lottery number will be even.",
|
||||
"The last digit of the next winning lottery number will be odd."
|
||||
],
|
||||
[
|
||||
"U6",
|
||||
"A person chosen at random from the world's population was born in the first half of the year.",
|
||||
"A person chosen at random from the world's population was born in the second half of the year."
|
||||
],
|
||||
[
|
||||
"U7",
|
||||
"The number of grains of sand on the nearest beach is even.",
|
||||
"The number of grains of sand on the nearest beach is odd."
|
||||
],
|
||||
[
|
||||
"U8",
|
||||
"The next ball drawn from a bag holding five white and five black balls will be white.",
|
||||
"The next ball drawn from a bag holding five white and five black balls will be black."
|
||||
],
|
||||
[
|
||||
"T1",
|
||||
"The S&P 500 index will be higher one year from now than it is today.",
|
||||
"The S&P 500 index will be lower one year from now than it is today."
|
||||
],
|
||||
[
|
||||
"T2",
|
||||
"The price of Bitcoin will be higher one year from now than it is today.",
|
||||
"The price of Bitcoin will be lower one year from now than it is today."
|
||||
],
|
||||
[
|
||||
"T3",
|
||||
"It will not rain in Tokyo on this day next year.",
|
||||
"It will rain in Tokyo on this day next year."
|
||||
],
|
||||
[
|
||||
"T4",
|
||||
"The next FIFA World Cup will be won by a European team.",
|
||||
"The next FIFA World Cup will be won by a team from outside Europe."
|
||||
],
|
||||
[
|
||||
"T5",
|
||||
"The men's 100 metres world record will not be broken within the next year.",
|
||||
"The men's 100 metres world record will be broken within the next year."
|
||||
],
|
||||
[
|
||||
"T6",
|
||||
"The next person elected president of the United States will be older than 50 when taking office.",
|
||||
"The next person elected president of the United States will be younger than 50 when taking office."
|
||||
],
|
||||
[
|
||||
"N1",
|
||||
"The sun will rise tomorrow.",
|
||||
"The sun will not rise tomorrow."
|
||||
],
|
||||
[
|
||||
"N2",
|
||||
"The world's population will be larger one year from now than it is today.",
|
||||
"The world's population will be smaller one year from now than it is today."
|
||||
],
|
||||
[
|
||||
"N3",
|
||||
"An earthquake of magnitude 6 or greater will occur somewhere in the world during the next year.",
|
||||
"No earthquake of magnitude 6 or greater will occur anywhere in the world during the next year."
|
||||
],
|
||||
[
|
||||
"N4",
|
||||
"It will snow in Sapporo at some point next January.",
|
||||
"It will not snow in Sapporo at any point next January."
|
||||
],
|
||||
[
|
||||
"N5",
|
||||
"Next year will have 365 or 366 days.",
|
||||
"Next year will have 400 days."
|
||||
],
|
||||
[
|
||||
"N6",
|
||||
"The next total solar eclipse visible from Earth will occur within the next five years.",
|
||||
"No total solar eclipse will be visible from Earth during the next five years."
|
||||
],
|
||||
[
|
||||
"K1",
|
||||
"Paris is the capital of France.",
|
||||
"Paris is the capital of Germany."
|
||||
],
|
||||
[
|
||||
"K2",
|
||||
"Water boils at 100 degrees Celsius at sea level.",
|
||||
"Water boils at 50 degrees Celsius at sea level."
|
||||
],
|
||||
[
|
||||
"K3",
|
||||
"In April 1986, a reactor at the Chernobyl nuclear power plant exploded, and people were killed.",
|
||||
"No one was killed in the accident at the Chernobyl nuclear power plant in April 1986."
|
||||
],
|
||||
[
|
||||
"K4",
|
||||
"The potato blight of the 1840s led to a famine in Ireland in which about one million people died.",
|
||||
"No famine occurred in Ireland during the 1840s."
|
||||
],
|
||||
[
|
||||
"K5",
|
||||
"A 1964 report by the US Surgeon General found that smoking causes lung cancer.",
|
||||
"The 1964 report by the US Surgeon General on smoking found no evidence that smoking harms health."
|
||||
],
|
||||
[
|
||||
"K6",
|
||||
"The Earth orbits the Sun.",
|
||||
"The Sun orbits the Earth."
|
||||
],
|
||||
[
|
||||
"P1",
|
||||
"The next roll of a fair six-sided die will show a 1.",
|
||||
"The next roll of a fair six-sided die will show a number other than 1."
|
||||
],
|
||||
[
|
||||
"P2",
|
||||
"The next card I draw from a well-shuffled standard deck will be a heart.",
|
||||
"The next card I draw from a well-shuffled standard deck will not be a heart."
|
||||
],
|
||||
[
|
||||
"P3",
|
||||
"The next card I draw from a well-shuffled standard deck will be an ace.",
|
||||
"The next card I draw from a well-shuffled standard deck will not be an ace."
|
||||
],
|
||||
[
|
||||
"P4",
|
||||
"The next roll of a fair six-sided die will show a 1 or a 2.",
|
||||
"The next roll of a fair six-sided die will show a 3, 4, 5, or 6."
|
||||
],
|
||||
[
|
||||
"P5",
|
||||
"The next two flips of a fair coin will both land heads.",
|
||||
"The next two flips of a fair coin will not both land heads."
|
||||
]
|
||||
],
|
||||
"probabilities": {
|
||||
"P1a": 0.1667,
|
||||
"P1b": 0.8333,
|
||||
"P2a": 0.25,
|
||||
"P2b": 0.75,
|
||||
"P3a": 0.0769,
|
||||
"P3b": 0.9231,
|
||||
"P4a": 0.3333,
|
||||
"P4b": 0.6667,
|
||||
"P5a": 0.25,
|
||||
"P5b": 0.75
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,626 @@
|
||||
[
|
||||
{
|
||||
"id": "U1a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.1563546622523346
|
||||
},
|
||||
{
|
||||
"id": "U1b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.10298498607890458
|
||||
},
|
||||
{
|
||||
"id": "U2a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.27531802064465766
|
||||
},
|
||||
{
|
||||
"id": "U2b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.29130223232773783
|
||||
},
|
||||
{
|
||||
"id": "U3a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.596886783643548
|
||||
},
|
||||
{
|
||||
"id": "U3b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.3328182831349319
|
||||
},
|
||||
{
|
||||
"id": "U4a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.4135071008052303
|
||||
},
|
||||
{
|
||||
"id": "U4b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.25353891261846645
|
||||
},
|
||||
{
|
||||
"id": "U5a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.3597190663096565
|
||||
},
|
||||
{
|
||||
"id": "U5b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.3477136050354511
|
||||
},
|
||||
{
|
||||
"id": "U6a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.6314669640969492
|
||||
},
|
||||
{
|
||||
"id": "U6b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.3741538921045834
|
||||
},
|
||||
{
|
||||
"id": "U7a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.31978838456545733
|
||||
},
|
||||
{
|
||||
"id": "U7b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.2546097220836936
|
||||
},
|
||||
{
|
||||
"id": "U8a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.15263059327321835
|
||||
},
|
||||
{
|
||||
"id": "U8b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.13091564626756208
|
||||
},
|
||||
{
|
||||
"id": "T1a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.803736698937114
|
||||
},
|
||||
{
|
||||
"id": "T1b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.056274844577240885
|
||||
},
|
||||
{
|
||||
"id": "T2a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.3678579990777839
|
||||
},
|
||||
{
|
||||
"id": "T2b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.37329297956124446
|
||||
},
|
||||
{
|
||||
"id": "T3a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.08927474404407679
|
||||
},
|
||||
{
|
||||
"id": "T3b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.8137230093194368
|
||||
},
|
||||
{
|
||||
"id": "T4a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.790477390492118
|
||||
},
|
||||
{
|
||||
"id": "T4b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.08047192678047338
|
||||
},
|
||||
{
|
||||
"id": "T5a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.8520500870846421
|
||||
},
|
||||
{
|
||||
"id": "T5b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.13444005790013824
|
||||
},
|
||||
{
|
||||
"id": "T6a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.8111855757100763
|
||||
},
|
||||
{
|
||||
"id": "T6b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.7050209843722118
|
||||
},
|
||||
{
|
||||
"id": "N1a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.9356080934390207
|
||||
},
|
||||
{
|
||||
"id": "N1b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.032338866863635074
|
||||
},
|
||||
{
|
||||
"id": "N2a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.9559256289426079
|
||||
},
|
||||
{
|
||||
"id": "N2b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.016569419950235302
|
||||
},
|
||||
{
|
||||
"id": "N3a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.908919745901481
|
||||
},
|
||||
{
|
||||
"id": "N3b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.026298370171733743
|
||||
},
|
||||
{
|
||||
"id": "N4a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.9392960484183409
|
||||
},
|
||||
{
|
||||
"id": "N4b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.02327075306277671
|
||||
},
|
||||
{
|
||||
"id": "N5a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.9076664209994167
|
||||
},
|
||||
{
|
||||
"id": "N5b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.035031695022713166
|
||||
},
|
||||
{
|
||||
"id": "N6a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.49885577163170747
|
||||
},
|
||||
{
|
||||
"id": "N6b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.5087399222304279
|
||||
},
|
||||
{
|
||||
"id": "K1a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.9774759389069352
|
||||
},
|
||||
{
|
||||
"id": "K1b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.012810059886409647
|
||||
},
|
||||
{
|
||||
"id": "K2a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.9532845343591053
|
||||
},
|
||||
{
|
||||
"id": "K2b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.025426653952945435
|
||||
},
|
||||
{
|
||||
"id": "K3a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.9683758009733633
|
||||
},
|
||||
{
|
||||
"id": "K3b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.034370633898392955
|
||||
},
|
||||
{
|
||||
"id": "K4a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.8849612771088546
|
||||
},
|
||||
{
|
||||
"id": "K4b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.018067256873252214
|
||||
},
|
||||
{
|
||||
"id": "K5a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.9283748177780613
|
||||
},
|
||||
{
|
||||
"id": "K5b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.031002559742295537
|
||||
},
|
||||
{
|
||||
"id": "K6a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.9763295039553002
|
||||
},
|
||||
{
|
||||
"id": "K6b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.022602225223762408
|
||||
},
|
||||
{
|
||||
"id": "U1a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.24064914137892165
|
||||
},
|
||||
{
|
||||
"id": "U1b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.1549050607030875
|
||||
},
|
||||
{
|
||||
"id": "U2a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.7114077534383975
|
||||
},
|
||||
{
|
||||
"id": "U2b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.6654411859926993
|
||||
},
|
||||
{
|
||||
"id": "U3a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.6207051703754343
|
||||
},
|
||||
{
|
||||
"id": "U3b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.5670477720065128
|
||||
},
|
||||
{
|
||||
"id": "U4a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.6319509842446309
|
||||
},
|
||||
{
|
||||
"id": "U4b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.6359247882034883
|
||||
},
|
||||
{
|
||||
"id": "U5a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.5442487038788544
|
||||
},
|
||||
{
|
||||
"id": "U5b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.3274244257434919
|
||||
},
|
||||
{
|
||||
"id": "U6a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.728263732364755
|
||||
},
|
||||
{
|
||||
"id": "U6b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.4342110118229748
|
||||
},
|
||||
{
|
||||
"id": "U7a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.27081585373424377
|
||||
},
|
||||
{
|
||||
"id": "U7b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.18296151484483048
|
||||
},
|
||||
{
|
||||
"id": "U8a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.47336041183055105
|
||||
},
|
||||
{
|
||||
"id": "U8b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.30528242405046324
|
||||
},
|
||||
{
|
||||
"id": "T1a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.8882269104060471
|
||||
},
|
||||
{
|
||||
"id": "T1b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.04396967521678573
|
||||
},
|
||||
{
|
||||
"id": "T2a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.3013701517706645
|
||||
},
|
||||
{
|
||||
"id": "T2b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.32712378007033704
|
||||
},
|
||||
{
|
||||
"id": "T3a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.08464050383877865
|
||||
},
|
||||
{
|
||||
"id": "T3b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.8699422179938108
|
||||
},
|
||||
{
|
||||
"id": "T4a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.8827551196528663
|
||||
},
|
||||
{
|
||||
"id": "T4b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.05284938359932525
|
||||
},
|
||||
{
|
||||
"id": "T5a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.899715889250761
|
||||
},
|
||||
{
|
||||
"id": "T5b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.1015997172090742
|
||||
},
|
||||
{
|
||||
"id": "T6a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.8614926406864682
|
||||
},
|
||||
{
|
||||
"id": "T6b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.7546528239764947
|
||||
},
|
||||
{
|
||||
"id": "N1a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9389274955639704
|
||||
},
|
||||
{
|
||||
"id": "N1b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.02653183601637322
|
||||
},
|
||||
{
|
||||
"id": "N2a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9424760623004863
|
||||
},
|
||||
{
|
||||
"id": "N2b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.022103155252441102
|
||||
},
|
||||
{
|
||||
"id": "N3a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.921112184919618
|
||||
},
|
||||
{
|
||||
"id": "N3b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.033456409653188746
|
||||
},
|
||||
{
|
||||
"id": "N4a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9354006228802139
|
||||
},
|
||||
{
|
||||
"id": "N4b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.027246941111788263
|
||||
},
|
||||
{
|
||||
"id": "N5a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.8264708812070534
|
||||
},
|
||||
{
|
||||
"id": "N5b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.03676608855142873
|
||||
},
|
||||
{
|
||||
"id": "N6a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.48855409348700174
|
||||
},
|
||||
{
|
||||
"id": "N6b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.5130744566996128
|
||||
},
|
||||
{
|
||||
"id": "K1a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9696001534468888
|
||||
},
|
||||
{
|
||||
"id": "K1b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.017463957778993393
|
||||
},
|
||||
{
|
||||
"id": "K2a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9438410895630032
|
||||
},
|
||||
{
|
||||
"id": "K2b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.04037044384111732
|
||||
},
|
||||
{
|
||||
"id": "K3a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9591383366948559
|
||||
},
|
||||
{
|
||||
"id": "K3b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.0316748786508726
|
||||
},
|
||||
{
|
||||
"id": "K4a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.8854235311543147
|
||||
},
|
||||
{
|
||||
"id": "K4b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.02242497296914693
|
||||
},
|
||||
{
|
||||
"id": "K5a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.941702509009343
|
||||
},
|
||||
{
|
||||
"id": "K5b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.026032544145669488
|
||||
},
|
||||
{
|
||||
"id": "K6a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9674552732131715
|
||||
},
|
||||
{
|
||||
"id": "K6b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.022730758403467992
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,818 @@
|
||||
[
|
||||
{
|
||||
"id": "U1a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.46
|
||||
},
|
||||
{
|
||||
"id": "U1b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.44
|
||||
},
|
||||
{
|
||||
"id": "U2a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.44
|
||||
},
|
||||
{
|
||||
"id": "U2b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.45
|
||||
},
|
||||
{
|
||||
"id": "U3a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.46
|
||||
},
|
||||
{
|
||||
"id": "U3b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.45
|
||||
},
|
||||
{
|
||||
"id": "U4a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.34
|
||||
},
|
||||
{
|
||||
"id": "U4b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.37
|
||||
},
|
||||
{
|
||||
"id": "U5a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.5
|
||||
},
|
||||
{
|
||||
"id": "U5b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.51
|
||||
},
|
||||
{
|
||||
"id": "U6a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.49
|
||||
},
|
||||
{
|
||||
"id": "U6b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.49
|
||||
},
|
||||
{
|
||||
"id": "U7a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.45
|
||||
},
|
||||
{
|
||||
"id": "U7b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.46
|
||||
},
|
||||
{
|
||||
"id": "U8a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.38
|
||||
},
|
||||
{
|
||||
"id": "U8b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.37
|
||||
},
|
||||
{
|
||||
"id": "T1a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.53
|
||||
},
|
||||
{
|
||||
"id": "T1b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.31
|
||||
},
|
||||
{
|
||||
"id": "T2a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.49
|
||||
},
|
||||
{
|
||||
"id": "T2b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.39
|
||||
},
|
||||
{
|
||||
"id": "T3a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.62
|
||||
},
|
||||
{
|
||||
"id": "T3b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.53
|
||||
},
|
||||
{
|
||||
"id": "T4a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.49
|
||||
},
|
||||
{
|
||||
"id": "T4b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.5
|
||||
},
|
||||
{
|
||||
"id": "T5a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.57
|
||||
},
|
||||
{
|
||||
"id": "T5b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.31
|
||||
},
|
||||
{
|
||||
"id": "T6a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.76
|
||||
},
|
||||
{
|
||||
"id": "T6b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.19
|
||||
},
|
||||
{
|
||||
"id": "N1a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.91
|
||||
},
|
||||
{
|
||||
"id": "N1b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.04
|
||||
},
|
||||
{
|
||||
"id": "N2a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.86
|
||||
},
|
||||
{
|
||||
"id": "N2b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.09
|
||||
},
|
||||
{
|
||||
"id": "N3a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.95
|
||||
},
|
||||
{
|
||||
"id": "N3b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.04
|
||||
},
|
||||
{
|
||||
"id": "N4a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.89
|
||||
},
|
||||
{
|
||||
"id": "N4b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.08
|
||||
},
|
||||
{
|
||||
"id": "N5a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.97
|
||||
},
|
||||
{
|
||||
"id": "N5b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.06
|
||||
},
|
||||
{
|
||||
"id": "N6a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.69
|
||||
},
|
||||
{
|
||||
"id": "N6b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.13
|
||||
},
|
||||
{
|
||||
"id": "K1a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.98
|
||||
},
|
||||
{
|
||||
"id": "K1b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.01
|
||||
},
|
||||
{
|
||||
"id": "K2a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.96
|
||||
},
|
||||
{
|
||||
"id": "K2b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.01
|
||||
},
|
||||
{
|
||||
"id": "K3a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.89
|
||||
},
|
||||
{
|
||||
"id": "K3b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.09
|
||||
},
|
||||
{
|
||||
"id": "K4a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.87
|
||||
},
|
||||
{
|
||||
"id": "K4b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.01
|
||||
},
|
||||
{
|
||||
"id": "K5a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.84
|
||||
},
|
||||
{
|
||||
"id": "K5b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.03
|
||||
},
|
||||
{
|
||||
"id": "K6a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.98
|
||||
},
|
||||
{
|
||||
"id": "K6b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.01
|
||||
},
|
||||
{
|
||||
"id": "U1a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.32
|
||||
},
|
||||
{
|
||||
"id": "U1b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.28
|
||||
},
|
||||
{
|
||||
"id": "U2a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.21
|
||||
},
|
||||
{
|
||||
"id": "U2b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.26
|
||||
},
|
||||
{
|
||||
"id": "U3a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.29
|
||||
},
|
||||
{
|
||||
"id": "U3b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.29
|
||||
},
|
||||
{
|
||||
"id": "U4a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.31
|
||||
},
|
||||
{
|
||||
"id": "U4b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.28
|
||||
},
|
||||
{
|
||||
"id": "U5a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.48
|
||||
},
|
||||
{
|
||||
"id": "U5b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.49
|
||||
},
|
||||
{
|
||||
"id": "U6a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.46
|
||||
},
|
||||
{
|
||||
"id": "U6b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.41
|
||||
},
|
||||
{
|
||||
"id": "U7a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.51
|
||||
},
|
||||
{
|
||||
"id": "U7b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.54
|
||||
},
|
||||
{
|
||||
"id": "U8a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.36
|
||||
},
|
||||
{
|
||||
"id": "U8b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.36
|
||||
},
|
||||
{
|
||||
"id": "T1a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.57
|
||||
},
|
||||
{
|
||||
"id": "T1b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.33
|
||||
},
|
||||
{
|
||||
"id": "T2a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.5
|
||||
},
|
||||
{
|
||||
"id": "T2b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.4
|
||||
},
|
||||
{
|
||||
"id": "T3a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.7
|
||||
},
|
||||
{
|
||||
"id": "T3b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.47
|
||||
},
|
||||
{
|
||||
"id": "T4a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.52
|
||||
},
|
||||
{
|
||||
"id": "T4b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.48
|
||||
},
|
||||
{
|
||||
"id": "T5a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.61
|
||||
},
|
||||
{
|
||||
"id": "T5b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.32
|
||||
},
|
||||
{
|
||||
"id": "T6a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.82
|
||||
},
|
||||
{
|
||||
"id": "T6b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.18
|
||||
},
|
||||
{
|
||||
"id": "N1a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.97
|
||||
},
|
||||
{
|
||||
"id": "N1b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.03
|
||||
},
|
||||
{
|
||||
"id": "N2a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.89
|
||||
},
|
||||
{
|
||||
"id": "N2b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.09
|
||||
},
|
||||
{
|
||||
"id": "N3a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.97
|
||||
},
|
||||
{
|
||||
"id": "N3b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.04
|
||||
},
|
||||
{
|
||||
"id": "N4a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.96
|
||||
},
|
||||
{
|
||||
"id": "N4b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.06
|
||||
},
|
||||
{
|
||||
"id": "N5a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.98
|
||||
},
|
||||
{
|
||||
"id": "N5b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.07
|
||||
},
|
||||
{
|
||||
"id": "N6a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.74
|
||||
},
|
||||
{
|
||||
"id": "N6b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.17
|
||||
},
|
||||
{
|
||||
"id": "K1a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.99
|
||||
},
|
||||
{
|
||||
"id": "K1b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.01
|
||||
},
|
||||
{
|
||||
"id": "K2a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.98
|
||||
},
|
||||
{
|
||||
"id": "K2b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.02
|
||||
},
|
||||
{
|
||||
"id": "K3a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.95
|
||||
},
|
||||
{
|
||||
"id": "K3b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.2
|
||||
},
|
||||
{
|
||||
"id": "K4a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.88
|
||||
},
|
||||
{
|
||||
"id": "K4b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.02
|
||||
},
|
||||
{
|
||||
"id": "K5a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.94
|
||||
},
|
||||
{
|
||||
"id": "K5b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.03
|
||||
},
|
||||
{
|
||||
"id": "K6a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.99
|
||||
},
|
||||
{
|
||||
"id": "K6b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.02
|
||||
},
|
||||
{
|
||||
"id": "U1a",
|
||||
"q": "Q1",
|
||||
"run": 2,
|
||||
"p_yes": 0.46
|
||||
},
|
||||
{
|
||||
"id": "U1b",
|
||||
"q": "Q1",
|
||||
"run": 2,
|
||||
"p_yes": 0.43
|
||||
},
|
||||
{
|
||||
"id": "U2a",
|
||||
"q": "Q1",
|
||||
"run": 2,
|
||||
"p_yes": 0.44
|
||||
},
|
||||
{
|
||||
"id": "U2b",
|
||||
"q": "Q1",
|
||||
"run": 2,
|
||||
"p_yes": 0.46
|
||||
},
|
||||
{
|
||||
"id": "U3a",
|
||||
"q": "Q1",
|
||||
"run": 2,
|
||||
"p_yes": 0.46
|
||||
},
|
||||
{
|
||||
"id": "U3b",
|
||||
"q": "Q1",
|
||||
"run": 2,
|
||||
"p_yes": 0.45
|
||||
},
|
||||
{
|
||||
"id": "U4a",
|
||||
"q": "Q1",
|
||||
"run": 2,
|
||||
"p_yes": 0.33
|
||||
},
|
||||
{
|
||||
"id": "U4b",
|
||||
"q": "Q1",
|
||||
"run": 2,
|
||||
"p_yes": 0.36
|
||||
},
|
||||
{
|
||||
"id": "U5a",
|
||||
"q": "Q1",
|
||||
"run": 2,
|
||||
"p_yes": 0.51
|
||||
},
|
||||
{
|
||||
"id": "U5b",
|
||||
"q": "Q1",
|
||||
"run": 2,
|
||||
"p_yes": 0.5
|
||||
},
|
||||
{
|
||||
"id": "U6a",
|
||||
"q": "Q1",
|
||||
"run": 2,
|
||||
"p_yes": 0.47
|
||||
},
|
||||
{
|
||||
"id": "U6b",
|
||||
"q": "Q1",
|
||||
"run": 2,
|
||||
"p_yes": 0.49
|
||||
},
|
||||
{
|
||||
"id": "U7a",
|
||||
"q": "Q1",
|
||||
"run": 2,
|
||||
"p_yes": 0.41
|
||||
},
|
||||
{
|
||||
"id": "U7b",
|
||||
"q": "Q1",
|
||||
"run": 2,
|
||||
"p_yes": 0.46
|
||||
},
|
||||
{
|
||||
"id": "U8a",
|
||||
"q": "Q1",
|
||||
"run": 2,
|
||||
"p_yes": 0.38
|
||||
},
|
||||
{
|
||||
"id": "U8b",
|
||||
"q": "Q1",
|
||||
"run": 2,
|
||||
"p_yes": 0.35
|
||||
},
|
||||
{
|
||||
"id": "U1a",
|
||||
"q": "Q1",
|
||||
"run": 3,
|
||||
"p_yes": 0.47
|
||||
},
|
||||
{
|
||||
"id": "U1b",
|
||||
"q": "Q1",
|
||||
"run": 3,
|
||||
"p_yes": 0.44
|
||||
},
|
||||
{
|
||||
"id": "U2a",
|
||||
"q": "Q1",
|
||||
"run": 3,
|
||||
"p_yes": 0.44
|
||||
},
|
||||
{
|
||||
"id": "U2b",
|
||||
"q": "Q1",
|
||||
"run": 3,
|
||||
"p_yes": 0.44
|
||||
},
|
||||
{
|
||||
"id": "U3a",
|
||||
"q": "Q1",
|
||||
"run": 3,
|
||||
"p_yes": 0.45
|
||||
},
|
||||
{
|
||||
"id": "U3b",
|
||||
"q": "Q1",
|
||||
"run": 3,
|
||||
"p_yes": 0.45
|
||||
},
|
||||
{
|
||||
"id": "U4a",
|
||||
"q": "Q1",
|
||||
"run": 3,
|
||||
"p_yes": 0.34
|
||||
},
|
||||
{
|
||||
"id": "U4b",
|
||||
"q": "Q1",
|
||||
"run": 3,
|
||||
"p_yes": 0.36
|
||||
},
|
||||
{
|
||||
"id": "U5a",
|
||||
"q": "Q1",
|
||||
"run": 3,
|
||||
"p_yes": 0.51
|
||||
},
|
||||
{
|
||||
"id": "U5b",
|
||||
"q": "Q1",
|
||||
"run": 3,
|
||||
"p_yes": 0.5
|
||||
},
|
||||
{
|
||||
"id": "U6a",
|
||||
"q": "Q1",
|
||||
"run": 3,
|
||||
"p_yes": 0.49
|
||||
},
|
||||
{
|
||||
"id": "U6b",
|
||||
"q": "Q1",
|
||||
"run": 3,
|
||||
"p_yes": 0.5
|
||||
},
|
||||
{
|
||||
"id": "U7a",
|
||||
"q": "Q1",
|
||||
"run": 3,
|
||||
"p_yes": 0.45
|
||||
},
|
||||
{
|
||||
"id": "U7b",
|
||||
"q": "Q1",
|
||||
"run": 3,
|
||||
"p_yes": 0.47
|
||||
},
|
||||
{
|
||||
"id": "U8a",
|
||||
"q": "Q1",
|
||||
"run": 3,
|
||||
"p_yes": 0.37
|
||||
},
|
||||
{
|
||||
"id": "U8b",
|
||||
"q": "Q1",
|
||||
"run": 3,
|
||||
"p_yes": 0.36
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,626 @@
|
||||
[
|
||||
{
|
||||
"id": "U1a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.04293589372706846
|
||||
},
|
||||
{
|
||||
"id": "U1b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.029438944319826802
|
||||
},
|
||||
{
|
||||
"id": "U2a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.012875717935652332
|
||||
},
|
||||
{
|
||||
"id": "U2b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.015392259115030355
|
||||
},
|
||||
{
|
||||
"id": "U3a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.04062919162805074
|
||||
},
|
||||
{
|
||||
"id": "U3b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.02698182131824937
|
||||
},
|
||||
{
|
||||
"id": "U4a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.00848915157377681
|
||||
},
|
||||
{
|
||||
"id": "U4b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.017836672013940468
|
||||
},
|
||||
{
|
||||
"id": "U5a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.0243622668628212
|
||||
},
|
||||
{
|
||||
"id": "U5b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.024844143004769465
|
||||
},
|
||||
{
|
||||
"id": "U6a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.28326839905182954
|
||||
},
|
||||
{
|
||||
"id": "U6b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.10557380048752749
|
||||
},
|
||||
{
|
||||
"id": "U7a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.10408252745000653
|
||||
},
|
||||
{
|
||||
"id": "U7b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.11869719610087492
|
||||
},
|
||||
{
|
||||
"id": "U8a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.015351063883996655
|
||||
},
|
||||
{
|
||||
"id": "U8b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.016817785729793713
|
||||
},
|
||||
{
|
||||
"id": "T1a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.022057347437745733
|
||||
},
|
||||
{
|
||||
"id": "T1b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.021683283520289053
|
||||
},
|
||||
{
|
||||
"id": "T2a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.025373198109493362
|
||||
},
|
||||
{
|
||||
"id": "T2b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.027161821769643532
|
||||
},
|
||||
{
|
||||
"id": "T3a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.03047279286820563
|
||||
},
|
||||
{
|
||||
"id": "T3b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.035284298468013904
|
||||
},
|
||||
{
|
||||
"id": "T4a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.08476684431458896
|
||||
},
|
||||
{
|
||||
"id": "T4b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.343042922540492
|
||||
},
|
||||
{
|
||||
"id": "T5a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.12621128760152997
|
||||
},
|
||||
{
|
||||
"id": "T5b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.021762038393210035
|
||||
},
|
||||
{
|
||||
"id": "T6a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.10113807328739997
|
||||
},
|
||||
{
|
||||
"id": "T6b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.049815864450550015
|
||||
},
|
||||
{
|
||||
"id": "N1a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.9923621500632038
|
||||
},
|
||||
{
|
||||
"id": "N1b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.015732907919588063
|
||||
},
|
||||
{
|
||||
"id": "N2a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.9824827957993699
|
||||
},
|
||||
{
|
||||
"id": "N2b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.0075136385868251965
|
||||
},
|
||||
{
|
||||
"id": "N3a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.868971045455636
|
||||
},
|
||||
{
|
||||
"id": "N3b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.0381296967843491
|
||||
},
|
||||
{
|
||||
"id": "N4a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.30421726759545153
|
||||
},
|
||||
{
|
||||
"id": "N4b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.04969070102240233
|
||||
},
|
||||
{
|
||||
"id": "N5a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.9886396837495554
|
||||
},
|
||||
{
|
||||
"id": "N5b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.004316507429744988
|
||||
},
|
||||
{
|
||||
"id": "N6a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.9430432488085654
|
||||
},
|
||||
{
|
||||
"id": "N6b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.6918752290584557
|
||||
},
|
||||
{
|
||||
"id": "K1a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.9983674985865608
|
||||
},
|
||||
{
|
||||
"id": "K1b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.001096284565803176
|
||||
},
|
||||
{
|
||||
"id": "K2a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.9968625419382113
|
||||
},
|
||||
{
|
||||
"id": "K2b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.0007492928963817307
|
||||
},
|
||||
{
|
||||
"id": "K3a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.9973426193681626
|
||||
},
|
||||
{
|
||||
"id": "K3b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.00564343885975838
|
||||
},
|
||||
{
|
||||
"id": "K4a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.9799021310424756
|
||||
},
|
||||
{
|
||||
"id": "K4b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.0005369441215446696
|
||||
},
|
||||
{
|
||||
"id": "K5a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.9905241203741747
|
||||
},
|
||||
{
|
||||
"id": "K5b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.021604404187287096
|
||||
},
|
||||
{
|
||||
"id": "K6a",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.9987152563331709
|
||||
},
|
||||
{
|
||||
"id": "K6b",
|
||||
"q": "Q1",
|
||||
"run": 1,
|
||||
"p_yes": 0.0010153053389281193
|
||||
},
|
||||
{
|
||||
"id": "U1a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.036065371680301876
|
||||
},
|
||||
{
|
||||
"id": "U1b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.016015495356012023
|
||||
},
|
||||
{
|
||||
"id": "U2a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9928237567830527
|
||||
},
|
||||
{
|
||||
"id": "U2b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9819637611648699
|
||||
},
|
||||
{
|
||||
"id": "U3a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9911519256833594
|
||||
},
|
||||
{
|
||||
"id": "U3b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9896628364737204
|
||||
},
|
||||
{
|
||||
"id": "U4a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.07382253819152373
|
||||
},
|
||||
{
|
||||
"id": "U4b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.1958787322666148
|
||||
},
|
||||
{
|
||||
"id": "U5a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.8139829254666326
|
||||
},
|
||||
{
|
||||
"id": "U5b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.31925609067679434
|
||||
},
|
||||
{
|
||||
"id": "U6a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.5005578992435681
|
||||
},
|
||||
{
|
||||
"id": "U6b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.17424720135095165
|
||||
},
|
||||
{
|
||||
"id": "U7a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.2632437468400511
|
||||
},
|
||||
{
|
||||
"id": "U7b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.2088569763616402
|
||||
},
|
||||
{
|
||||
"id": "U8a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9238114271132076
|
||||
},
|
||||
{
|
||||
"id": "U8b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.8458922188739306
|
||||
},
|
||||
{
|
||||
"id": "T1a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.06844525246176938
|
||||
},
|
||||
{
|
||||
"id": "T1b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.03048891340927842
|
||||
},
|
||||
{
|
||||
"id": "T2a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.06271505803458369
|
||||
},
|
||||
{
|
||||
"id": "T2b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.10705744486969046
|
||||
},
|
||||
{
|
||||
"id": "T3a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.10345688735641251
|
||||
},
|
||||
{
|
||||
"id": "T3b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.09088666758694891
|
||||
},
|
||||
{
|
||||
"id": "T4a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.5041784267436615
|
||||
},
|
||||
{
|
||||
"id": "T4b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.5448751840860677
|
||||
},
|
||||
{
|
||||
"id": "T5a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.8486552233735357
|
||||
},
|
||||
{
|
||||
"id": "T5b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.03331734516287631
|
||||
},
|
||||
{
|
||||
"id": "T6a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.5903050903416717
|
||||
},
|
||||
{
|
||||
"id": "T6b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.06163418042691574
|
||||
},
|
||||
{
|
||||
"id": "N1a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9952257610652505
|
||||
},
|
||||
{
|
||||
"id": "N1b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.003253875244862294
|
||||
},
|
||||
{
|
||||
"id": "N2a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9945098325502285
|
||||
},
|
||||
{
|
||||
"id": "N2b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.01492617814078072
|
||||
},
|
||||
{
|
||||
"id": "N3a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9879770039907161
|
||||
},
|
||||
{
|
||||
"id": "N3b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.03849597359112111
|
||||
},
|
||||
{
|
||||
"id": "N4a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9700146589847463
|
||||
},
|
||||
{
|
||||
"id": "N4b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.2949851143799287
|
||||
},
|
||||
{
|
||||
"id": "N5a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9854057133971014
|
||||
},
|
||||
{
|
||||
"id": "N5b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.004827952746193422
|
||||
},
|
||||
{
|
||||
"id": "N6a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9450633776994529
|
||||
},
|
||||
{
|
||||
"id": "N6b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.8507882764263268
|
||||
},
|
||||
{
|
||||
"id": "K1a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9987219591117057
|
||||
},
|
||||
{
|
||||
"id": "K1b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.0018607377876475697
|
||||
},
|
||||
{
|
||||
"id": "K2a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9986368018314127
|
||||
},
|
||||
{
|
||||
"id": "K2b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.0021420849777845407
|
||||
},
|
||||
{
|
||||
"id": "K3a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9983703033039442
|
||||
},
|
||||
{
|
||||
"id": "K3b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.026889236926969314
|
||||
},
|
||||
{
|
||||
"id": "K4a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9958700895758017
|
||||
},
|
||||
{
|
||||
"id": "K4b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.0007353601143941215
|
||||
},
|
||||
{
|
||||
"id": "K5a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9974361652933591
|
||||
},
|
||||
{
|
||||
"id": "K5b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.046099363839834075
|
||||
},
|
||||
{
|
||||
"id": "K6a",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.9990418145507582
|
||||
},
|
||||
{
|
||||
"id": "K6b",
|
||||
"q": "Q2",
|
||||
"run": 1,
|
||||
"p_yes": 0.0009381008392334215
|
||||
}
|
||||
]
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
+68
-8
@@ -79,6 +79,11 @@ _CLEF_SET_ORDER_SYMBOLS = (
|
||||
_LLAMA_PROCESS_TYPE_DECODE = 1
|
||||
_GGUF_TYPE_STRING = 8
|
||||
|
||||
# d1 (Liquid AI's decision model) prompt and answer words, as in prompt.py of LiquidAI/d1-3B
|
||||
# and the "systemone" template of its GGUF.
|
||||
_D1_CODES = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
|
||||
_D1_NOUL_FORMS = [("yes", "Yes", "YES"), ("no", "No", "NO")]
|
||||
|
||||
_loaded_model: tuple[str, Any, Any] | None = None
|
||||
|
||||
|
||||
@@ -184,12 +189,18 @@ class JevBackend:
|
||||
|
||||
|
||||
class LocalJev:
|
||||
"""Answers with a local GGUF model, one forward pass per question. Noul is asked as Yes/No options."""
|
||||
"""Answers with a local GGUF model, one forward pass per question.
|
||||
|
||||
A d1 model is asked in the prompt format it was tuned on. Other models get lettered options, noul as Yes/No.
|
||||
"""
|
||||
|
||||
def __init__(self, model_path: str):
|
||||
self.model_path = model_path
|
||||
|
||||
def ask(self, state: Any, questions: list[tuple]) -> list[list[float]]:
|
||||
llm, _ = load_local_model(self.model_path)
|
||||
if llm.metadata.get("lfm2.decision.type") == "lfm2-d1":
|
||||
return [_d1_probabilities(llm, state, *question) for question in questions]
|
||||
answers = []
|
||||
for kind, question, criteria in questions:
|
||||
if kind == "noul":
|
||||
@@ -351,15 +362,67 @@ def _option_probabilities(model_path: str, state: Any, question: Any, options: l
|
||||
{"role": "user", "content": json.dumps(payload, ensure_ascii=False)},
|
||||
]
|
||||
prompt = formatter(messages=messages, enable_thinking=False).prompt
|
||||
tokens = llm.tokenize(prompt.encode("utf-8"), add_bos=False, special=True)
|
||||
if len(tokens) >= llm.n_ctx():
|
||||
raise ValueError(f"Jev prompt is {len(tokens)} tokens, over the {llm.n_ctx()} token context")
|
||||
slots = []
|
||||
for letter in _LETTERS[: len(options)]:
|
||||
encoded = llm.tokenize(letter.encode("utf-8"), add_bos=False, special=False)
|
||||
if len(encoded) != 1:
|
||||
raise ValueError(f"Model tokenizer does not encode {letter!r} as one token")
|
||||
slots.append(encoded[0])
|
||||
logits = _next_token_logits(llm, prompt)[slots].astype(np.float64)
|
||||
weights = np.exp(logits - logits.max())
|
||||
return (weights / weights.sum()).tolist()
|
||||
|
||||
|
||||
def _d1_text(value: Any) -> str:
|
||||
return value if isinstance(value, str) else json.dumps(value, ensure_ascii=False)
|
||||
|
||||
|
||||
def _d1_probabilities(llm: Any, state: Any, kind: str, question: Any, criteria: Any) -> list[float]:
|
||||
"""Reads one decision from the next-token logits of the answer words d1 was tuned to reply with."""
|
||||
|
||||
def token_ids(forms: Any) -> list[int]:
|
||||
encoded = [llm.tokenize(form.encode("utf-8"), add_bos=False, special=False) for form in forms]
|
||||
return [tokens[0] for tokens in encoded if len(tokens) == 1]
|
||||
|
||||
if kind == "noul":
|
||||
body = "\n\nReply with yes or no only."
|
||||
groups = [token_ids(forms) for forms in _D1_NOUL_FORMS]
|
||||
elif kind == "choice":
|
||||
if not 2 <= len(criteria) <= len(_D1_CODES):
|
||||
raise ValueError(f"d1 choice questions need 2-{len(_D1_CODES)} options, got {len(criteria)}")
|
||||
labels = [str(label).strip() for label in criteria]
|
||||
# Labels that are single letters are their own codes.
|
||||
codes = labels if all(len(label) == 1 and label.isalpha() for label in labels) else _D1_CODES[: len(labels)]
|
||||
lines = [
|
||||
f"{code} {_d1_text(description) if description else label.replace('_', ' ')}"
|
||||
for code, label, description in zip(codes, labels, criteria.values())
|
||||
]
|
||||
body = "\n\nOptions:\n" + "\n".join(lines) + "\n\nReply with the option code only."
|
||||
groups = [token_ids((code, f" {code}")) for code in codes]
|
||||
else:
|
||||
if not 2 <= len(criteria) <= 10:
|
||||
raise ValueError(f"d1 score questions need 2-10 levels, got {len(criteria)}")
|
||||
legend = "".join(f"{level} {_d1_text(description)}\n" for level, description in enumerate(criteria))
|
||||
body = f"\n\n{legend}\nReply with a single digit 0-{len(criteria) - 1} only."
|
||||
groups = [token_ids([str(level)]) for level in range(len(criteria))]
|
||||
if not all(groups) or len({group[0] for group in groups}) != len(groups):
|
||||
raise ValueError("Model tokenizer does not encode each d1 answer as one distinct token")
|
||||
state_text = state if isinstance(state, str) else json.dumps(state, ensure_ascii=False, indent=2)
|
||||
prompt = (
|
||||
f"<|startoftext|><|im_start|>user\n{state_text}\n\n\nQUESTION:\n"
|
||||
f"{_d1_text(question)}{body}<|im_end|>\n<|im_start|>assistant\n"
|
||||
)
|
||||
vocabulary = _next_token_logits(llm, prompt)
|
||||
# Each option scores its best form.
|
||||
scores = np.array([vocabulary[group].max() for group in groups], dtype=np.float64)
|
||||
weights = np.exp(scores - scores.max())
|
||||
return (weights / weights.sum()).tolist()
|
||||
|
||||
|
||||
def _next_token_logits(llm: Any, prompt: str) -> Any:
|
||||
tokens = llm.tokenize(prompt.encode("utf-8"), add_bos=False, special=True)
|
||||
if len(tokens) >= llm.n_ctx():
|
||||
raise ValueError(f"Jev prompt is {len(tokens)} tokens, over the {llm.n_ctx()} token context")
|
||||
# Each question starts from empty memory, including the recurrent state of hybrid models such as Qwen3.5.
|
||||
# Decode with the llama.cpp API directly instead of the Llama evaluation helper,
|
||||
# whose method name trips the ComfyUI Registry dynamic-execution scanner.
|
||||
@@ -377,10 +440,7 @@ def _option_probabilities(model_path: str, state: Any, question: Any, options: l
|
||||
raise RuntimeError("llama_decode failed for the Jev prompt")
|
||||
finally:
|
||||
llama_batch_free(batch)
|
||||
vocabulary = np.ctypeslib.as_array(llama_get_logits_ith(llm.ctx, -1), shape=(llm.n_vocab(),))
|
||||
logits = vocabulary[slots].astype(np.float64)
|
||||
weights = np.exp(logits - logits.max())
|
||||
return (weights / weights.sum()).tolist()
|
||||
return np.ctypeslib.as_array(llama_get_logits_ith(llm.ctx, -1), shape=(llm.n_vocab(),))
|
||||
|
||||
|
||||
def _model_dirs() -> list[str]:
|
||||
|
||||
+1
-1
@@ -1,7 +1,7 @@
|
||||
[project]
|
||||
name = "ComfyUI-ScriptFlow"
|
||||
description = "Safe Python-like script node with a System One decision model: ask yes/no, choice, and score questions via TypeSafe AI's Jev API, with local GGUF LLMs as a fallback. Multiple inputs and outputs let one node handle math, logic, text editing, conditional routing, resolution selection, filename generation, and more."
|
||||
version = "1.3.3"
|
||||
version = "1.3.4"
|
||||
license = {file = "LICENSE"}
|
||||
# classifiers = [
|
||||
# # For OS-independent nodes (works on all operating systems)
|
||||
|
||||
Reference in New Issue
Block a user