Add the English report on uncertain questions and revise the Japanese one

docs/local-model-uncertainty/README.ja.md

@@ -1,4 +1,5 @@
 # 汎用のローカルモデルは、不確かな問いにどう答えるか：`jev.*` の挙動と聞き方の指針
+[[en](README.md) | ja]
 
 技術報告（2026年10月）
 
@@ -9,7 +10,7 @@
 - 答えが確実な問いでは、3つのモデルの答えはほぼ一致した。
 - 答えが不確実な問いでは、振る舞いが分かれた。五分五分の問いに `jev.noul` で「真ですか？」と聞くと、平均は Jev が 0.44、Clef-Flash が 0.31、Qwen3.5-9B が 0.06 だった。
 - Qwen3.5-9B の低い値は、「起こる確率が低い」ではなく「真だとは言えない」を表している。確率が 0.92 の主張にも 0.02 と答えた。
-- 主張とその否定形の両方に聞き、2つの答えをまとめることで、Qwen3.5-9B でも五分五分の問いを 0.51 にすることができた。
+- 主張とその否定形の両方について聞き、2つの答えを合わせることで、Qwen3.5-9B でも五分五分の問いを 0.51 にすることができた。
 - 確率の段階を選ばせる（`jev.score`）と、確率の高さがある程度取り出せた。結果は、段階の作り方と並び順に左右された。
 - 「偽ですか？」と質問を反対向きにする方法は、否定形の代わりにならなかった。
 
@@ -39,7 +40,7 @@
 | Clef-Flash | Qwen3.5-9B を判断用に追加学習したモデル（`ggml-org/Clef-Flash-GGUF`、Q8_0） | 判断専用の層が出す値を確率に直す |
 | Qwen3.5-9B | 汎用の LLM（`unsloth/Qwen3.5-9B-GGUF`、Q8_0） | 選択肢の記号が次に出る確率を読む |
 
-汎用モデルでは、選択肢に記号を付けて示し（`jev.noul` なら A：Yes、B：No）、記号1文字だけで答えるよう指示する。文章は生成させず、次のトークンとして各記号が出る確率を読み、（記号以外の出力は除き）記号どうしの合計が 1 になるように直す。この方式は SemIf (OpenJev) で提案されているものである。実験結果は較正なしの数値を用いており、較正の可能性について考察している。なお、サンプリングをしないので、乱数のシードや温度の設定はない。
+汎用モデルでは、選択肢に記号を付けて示し（`jev.noul` なら A：Yes、B：No）、記号1文字だけで答えるよう指示する。文章は生成させず、次のトークンとして各記号が出る確率を読み、（記号以外の出力は除き）記号どうしの合計が 1 になるように直す。この方式は SemIf (OpenJev) で提案されているものである。この報告の値は、較正をしていない値である。較正については、3.2 節と 5 節で触れる。なお、サンプリングをしないので、乱数のシードや温度の設定はない。
 
 指示文は次のとおりである。主張、質問、選択肢は、JSON にまとめて渡す。
 
@@ -84,16 +85,16 @@ Q1 と Q2 は3つのモデルに聞いた。F1、S6、S11、F6 は Qwen3.5-9B
 - **0.5 からのずれ**：五分五分の問い16文での、値と 0.5 の差の絶対値の平均（基準2）。
 - **誤差の平均**：確率のわかっている問い10文での、値と正しい確率の差の絶対値の平均。
 - **順位相関**：同じ10文での、値と正しい確率のスピアマンの順位相関。
-- **肯定否定バイアスを除去した値**：（主張への値 ＋ 1 − 否定形への値）÷ 2。対の合計が当然 1 になる。
-- **段階からの推定値**：各段階の値（S6 と F6 は段階の中央の値である 0.05、0.20、0.40、0.60、0.80、0.95）を、モデルが各段階に付けた確率で重み付けした値。
+- **否定形と合わせた値**：（主張への値 ＋ 1 − 否定形への値）÷ 2。対の合計が当然 1 になる。
+- **段階からの推定値**：各段階の値（S6 と F6 は段階の中央の値である 0.05、0.20、0.40、0.60、0.80、0.95、S11 はその点の値である 0、0.1、…、1.0）を、モデルが各段階に付けた確率で重み付けした値。
 
-まとめた値は、新しく測ったものではなく、測定済みの値から計算したものである。
+否定形と合わせた値は、新しく測ったものではなく、測定済みの値から計算したものである。
 
 ## 3. 結果
 
 ### 3.1 3つのモデルの比較（Q1）
 
-値は Yes の確率の平均で、左が対の1つめ、右が2つめの主張である。特に太文字斜体で記載したQwen3.5-9Bの「傾向のある未来」、「五分五分の問い」の結果について、先に挙げた **モデルに期待する2つの基準** と乖離しており、後段でさらに検証を行う。
+値は Yes の確率の平均で、左が対の1つめ、右が2つめの主張である。特に、太字の斜体で示した Qwen3.5-9B の「傾向のある未来」と「五分五分の問い」の結果は、1 節の基準1・基準2 と乖離しており、3.2 節以降でさらに検証を行う。
 
 | グループ | Jev | Clef-Flash | Qwen3.5-9B |
 | --- | --- | --- | --- |
@@ -112,7 +113,7 @@ Q1 と Q2 は3つのモデルに聞いた。F1、S6、S11、F6 は Qwen3.5-9B
 
 - 既知の事実は、3つとも12文すべてに正しい側の値を返した。
 - Jev は、五分五分の問い16文すべてに 0.34〜0.51 を返した。
-- Qwen3.5-9B は、不確かな主張(傾向のある未来、五分五分の問い)のほぼすべてに No と答えた。未来のことをすべて No とするわけではなく、「明日、太陽は昇る（ほぼ確実な未来）」には 0.99 を返している。
+- Qwen3.5-9B は、不確かな主張（傾向のある未来、五分五分の問い）のほぼすべてに No と答えた。未来のことをすべて No とするわけではなく、「明日、太陽は昇る」（ほぼ確実な未来）には 0.99 を返している。
 - Clef-Flash は、傾向のある未来では対の合計が 0.98 で、調整前の Qwen3.5-9B（0.15）と大きく違う。コインやサイコロのような問いでは 0.62 にとどまった（コイン投げは 0.16 / 0.10）。
 
 Clef-Flash と Qwen3.5-9B では、答えの出し方も違う。2つの差が、重みの調整によるものか、答えの出し方によるものかは、この試験では分けられない。
@@ -121,7 +122,7 @@ Clef-Flash は、傾向のある未来で片方をはっきり選んだ（S&P 50
 
 ### 3.2 Qwen3.5-9B の No は、何を表しているか
 
-ここから、Qwen3.5-9Bの「傾向のある未来」、「五分五分の問い」の結果について検証する。
+ここから、Qwen3.5-9B の「傾向のある未来」と「五分五分の問い」の結果について検証する。
 
 まず確認したのは、Qwen3.5-9B の No が **記号以外のトークンに確率が流れた結果ではない** ということである。語彙全体の確率のうち、A と B が占める割合（SemIf の `allowed_token_mass`）を、62文 × Q1・Q2 の124問で記録した。
 
@@ -166,9 +167,9 @@ Qwen3.5-9B は、「真ですか？」に対して、「確実に真だと言え
 
 汎用モデルの答えは、質問の言い回しに強く左右される。
 
-### 3.4 主張と否定形の両方を聞く
+### 3.4 主張と否定形の両方について聞く
 
-Qwen3.5-9B は、Q1 で「ほぼ確実に起こる」には Yes、「わからない」には No と答える。区別できないのは、「わからない」と「ほぼ確実に起こらない」である。後者の否定形は「ほぼ確実に起こる」なので、否定形も聞けば区別できる。
+Qwen3.5-9B は、Q1 で「ほぼ確実に起こる」には Yes、「わからない」には No と答える。区別できないのは、「わからない」と「ほぼ確実に起こらない」である。後者の否定形は「ほぼ確実に起こる」なので、否定形についても聞けば区別できる。
 
 | 主張の種類 | 主張への答え | 否定形への答え |
 | --- | --- | --- |
@@ -176,7 +177,7 @@ Qwen3.5-9B は、Q1 で「ほぼ確実に起こる」には Yes、「わから
 | ほぼ確実に起こらない | No | Yes |
 | わからない | No | No |
 
-設問は対で作ってあるので、対の相手を否定形とみなして、まとめた値を計算した（Q1）。かっこ内は、主張だけを聞いたときの値である。
+設問は対で作ってあるので、対の相手を否定形とみなして、否定形と合わせた値を計算した（Q1）。かっこ内は、主張だけを聞いたときの値である。太字の斜体は、3.1 節で基準から外れていた項目である。
 
 | グループ | Jev | Clef-Flash | Qwen3.5-9B |
 | --- | --- | --- | --- |
@@ -197,7 +198,7 @@ Qwen3.5-9B について、2つの答えを 0.5 で区切って「真 / 偽 / わ
 
 外れた2組は、「来年1月に札幌で雪が降る / 降らない」（両方に No）と、「皆既日食が5年以内に起こる / 起こらない」（両方に Yes）である。両方に Yes という答えは、その答えが信頼できないことの目印になる。
 
-この方法では、確率の程度は取り出せない。確率が 0.08 の主張も 0.92 の主張も、まとめた値は 0.5 付近になる（確率のわかっている問いでの誤差の平均は 0.244、順位相関は −0.12）。
+この方法では、確率の程度は取り出せない。確率が 0.08 の主張も 0.92 の主張も、否定形と合わせた値は 0.5 付近になる（確率のわかっている問いでの誤差の平均は 0.244、順位相関は −0.12）。
 
 ### 3.5 確率の段階を選ばせる
 
@@ -205,7 +206,22 @@ Qwen3.5-9B について、2つの答えを 0.5 で区切って「真 / 偽 / わ
 
 **対照：数字で書かせると、ほぼ正確である。** 確率のわかっている10文での誤差の平均は 0.003 だった（たとえば、正しい確率 7.7% に「9」、16.7% に「16.67」、92.3% に「92」）。コイン投げには「50」と答えた。モデルは確率の知識を持っている。Q1 で確率の高さが出ないのは、聞き方による。
 
-[ここに結果を示す表を挿入して下さい。]
+| 主張 | 正しい確率 | 書かれた数字 |
+| --- | --- | --- |
+| トランプでエースを引く | 7.7% | 9 |
+| サイコロで1が出る | 16.7% | 16.67 |
+| トランプでハートを引く | 25% | 25 |
+| コイン2回とも表 | 25% | 25 |
+| サイコロで1か2が出る | 33.3% | 33 |
+| サイコロで3〜6が出る | 66.7% | 66 |
+| トランプでハート以外を引く | 75% | 75 |
+| コイン2回とも表、にはならない | 75% | 75 |
+| サイコロで1以外が出る | 83.3% | 83 |
+| トランプでエース以外を引く | 92.3% | 92 |
+| 五分五分の問い（コイン、サイコロ、生まれた時期、砂粒の4文） | 50% | すべて 50 |
+| 明日、太陽は昇る / 昇らない | — | 100 / 0 |
+| パリはフランスの首都である / ドイツの首都である | — | 100 / 0 |
+| S&P 500 は1年後に高い / 低い | — | 65 / 50 |
 
 **段階で聞くと、確率の高さがある程度出る。** 確率のわかっている10文での結果は、次のとおりである。
 
@@ -218,7 +234,10 @@ Qwen3.5-9B について、2つの答えを 0.5 で区切って「真 / 偽 / わ
 | S11、高い順 | ＋0.80 | 0.203 |
 | 数字で書かせる | ＋1.00 | 0.003 |
 
-[ここにS6、S11の定義を再度記載してください。]
+段階で聞く2通りの質問を、もう一度示す。どちらも質問文は「How likely is this statement to be true?」で、選択肢だけが違う。
+
+- **S6（6段階の幅）**：0-10%、10-30%、30-50%、50-70%、70-90%、90-100%。推定値には、各段階の中央の値（0.05、0.20、0.40、0.60、0.80、0.95）を使う。
+- **S11（11段階の点）**：0%、10%、20%、…、100%。推定値には、その点の値（0、0.1、…、1.0）を使う。
 
 **6段階の幅（S6）と11段階の点（S11）は、得意な点が違う。**
 
@@ -236,7 +255,7 @@ Qwen3.5-9B について、2つの答えを 0.5 で区切って「真 / 偽 / わ
 
 「50%」という選択肢があると、五分五分の問いには正しく答えられるが、確実でない問いがそこに集まる。S11 は、「真 / 偽 / わからない」の3つに分ける方法に近い働きをする。
 
-**6段階の幅を、主張と否定形の両方に聞くと、両方に対応できる。**
+**6段階の幅で、主張と否定形の両方について聞くと、両方に対応できる。**
 
 | 聞き方（S6） | 0.5 からのずれ | 誤差の平均 | 順位相関 |
 | --- | --- | --- | --- |
@@ -244,7 +263,7 @@ Qwen3.5-9B について、2つの答えを 0.5 で区切って「真 / 偽 / わ
 | 低い順、否定形と合わせる | 0.027 | 0.069 | ＋0.90 |
 | 高い順、否定形と合わせる | 0.034 | 0.148 | ＋0.92 |
 
-五分五分の問いのずれは、主張にも否定形にも同じ向きに出るので、まとめると打ち消し合う。段階は低い順に並べるほうがよい。
+五分五分の問いのずれは、主張にも否定形にも同じ向きに出るので、合わせると打ち消し合う。段階は低い順に並べるほうがよい。
 
 | 主張 | 正しい確率 | 主張だけ | 否定形と合わせる |
 | --- | --- | --- | --- |
@@ -256,7 +275,7 @@ Qwen3.5-9B について、2つの答えを 0.5 で区切って「真 / 偽 / わ
 
 ### 3.6 「偽ですか？」は、否定形の代わりにならない
 
-否定形を書く代わりに、主張はそのままにして、質問を反対向きにする方法（F1、F6）を試した。まとめた値は、「（Q1 の値 ＋ 1 − F1 の値）÷ 2」で計算した。
+否定形を書く代わりに、主張はそのままにして、質問を反対向きにする方法（F1、F6）を試した。合わせた値は、「（Q1 の値 ＋ 1 − F1 の値）÷ 2」で計算した。
 
 | グループ | 否定形の主張と合わせる | F1 と合わせる |
 | --- | --- | --- |
@@ -293,19 +312,20 @@ Qwen3.5-9B について、2つの答えを 0.5 で区切って「真 / 偽 / わ
 
 1. **`jev.noul` や `jev.yes` の低い値を、「起こりにくい」と読まない。** 「真だとは言えない」を含む。判断材料が足りない入力は、ほぼ常に No 側に分岐する。
 2. **「真 / 偽 / わからない」がわかればよいなら、11段階の点で1回聞く。**
-3. **Yes / No で聞くなら、主張と否定形の両方に聞く。** 両方が No なら「わからない」、両方が Yes なら「信頼できない答え」として扱う。
-4. **確率の程度も知りたいなら、6段階の幅を低い順に並べ、主張と否定形の両方に聞く。**
-5. **否定形は、主張として書く。** 質問を「偽ですか？」に変える方法は、代わりにならない。この結果から機械的な対応が難しいと判断される。
+3. **Yes / No で聞くなら、主張と否定形の両方について聞く。** 両方が No なら「わからない」、両方が Yes なら「信頼できない答え」として扱う。
+4. **確率の程度も知りたいなら、6段階の幅を低い順に並べ、主張と否定形の両方について聞く。**
+5. **否定形は、主張として書く。** 質問を「偽ですか？」に変える方法は、代わりにならない。そのため、否定形の質問をノードの側で自動的に作る方法は、いまのところない。
 6. **聞きたいことを、そのまま質問文にする。** 「真か」と「起こりやすいほうか」では、答えが大きく変わる。
 7. **モデルを替えたら、しきい値を見直す。** 同じスクリプトでも、Jev、Clef-Flash、汎用モデルで、不確かな入力の分岐が変わる。
 
-主張と否定形の両方に聞く例：
+主張と否定形の両方について聞く例。数値の出力（`ov1` など）は整数なので、否定形と合わせた値は 100 倍して 0〜100 で出力する。
 
 ```python
 # it1: 主張、it2: その否定形
+# ov1: 否定形と合わせた値（0〜100）、ot1: 分類
 a = jev.noul(it1, "Is this statement true?")
 b = jev.noul(it2, "Is this statement true?")
-ov1 = (a + 1 - b) / 2
+ov1 = (a + 1 - b) / 2 * 100
 
 if a < 0.5 and b < 0.5:
     ot1 = "unsure"
@@ -317,14 +337,14 @@ else:
     ot1 = "false"
 ```
 
-11段階の点で聞く例。`jev.score` は選ばれた段階の番号の期待値（0〜10）を返すので、10 で割ると 0〜1 の値になる。
+11段階の点で聞く例。`jev.score` は選ばれた段階の番号の期待値（0〜10）を返すので、10 倍すると 0〜100 の値になる。
 
 ```python
 levels = ["0%", "10%", "20%", "30%", "40%", "50%", "60%", "70%", "80%", "90%", "100%"]
-ov1 = jev.score(it1, "How likely is this statement to be true?", levels) / 10
+ov1 = jev.score(it1, "How likely is this statement to be true?", levels) * 10
 ```
 
-6段階の幅では、段階の中央の値が等間隔でないので、`jev.score` の値を 5 で割っても、3.5 節の推定値とは同じにならない。近似として使う場合は、その点に注意する。
+6段階の幅では、段階の中央の値が等間隔でないので、`jev.score` の値を 20 倍しても、3.5 節の推定値（の 100 倍）とは同じにならない。近似として使う場合は、その点に注意する。
 
 ## 5. 限界
 
@@ -335,7 +355,7 @@ ov1 = jev.score(it1, "How likely is this statement to be true?", levels) / 10
 - 段階で聞く方法（S6、S11）、否定形と合わせる方法の段階での効果、「偽」の質問（F1、F6）は、Qwen3.5-9B でしか試していない。Jev と Clef-Flash には聞いていない。
 - 段階の作り方は、2通りしか試していない。
 - 汎用モデルは Qwen3.5-9B の Q8_0 だけである。別のモデル、別の量子化、別の判定方式では、違う結果になりうる。
-- 汎用モデルの値は、較正をしていない。SemIf (OpenJev) は、用途ごとに較正して使うよう勧めているが、これまでの検証で、較正のための適切な質問群を設計できていない。
+- 汎用モデルの値は、較正をしていない。SemIf (OpenJev) は、用途ごとに較正して使うよう勧めているが、今回は較正を試していない。3.2 節のとおり、Yes / No の値には確率の高さの情報がなく、較正に使える設問も用意できていない。
 - どの質問も1回ずつの測定である。ローカルのモデルの値は、サンプリングをしないので、同じ条件なら変わらない。Jev は、同じ質問でも値が少し変わる。五分五分の問い16文を Q1 で3回聞いたところ、値の幅は最大で 0.04 だった。
 - 主張と質問は英語である。ほかの言語では試していない。
 

docs/local-model-uncertainty/README.md

@@ -0,0 +1,384 @@
+# How general local models answer uncertain questions: `jev.*` behavior and how to ask
+[en | [ja](README.ja.md)]
+
+Technical report (October 2026)
+
+## Summary
+
+`MultiOutputScript (Jev)` can make decisions with a local GGUF model instead of TypeSafe's Jev. This report compares how Jev, Clef-Flash, and a general model (Qwen3.5-9B) answer questions whose answer cannot be known, and looks at how to ask when a general model is used.
+
+- On questions with a certain answer, the three models gave almost the same answers.
+- On questions with an uncertain answer, they behaved differently. When 50/50 statements were asked "Is this statement true?" with `jev.noul`, the mean was 0.44 for Jev, 0.31 for Clef-Flash, and 0.06 for Qwen3.5-9B.
+- A low value from Qwen3.5-9B means "cannot say it is true", not "unlikely to happen". It answered 0.02 for a statement whose probability is 0.92.
+- Asking about both a statement and its negation, and combining the two answers, brought the 50/50 statements to 0.51 with Qwen3.5-9B as well.
+- Having the model pick a probability level (`jev.score`) recovered the size of the probability to some degree. The result depended on how the levels were defined and on their order.
+- Reversing the question to "Is this statement false?" did not work as a substitute for the negation.
+
+This is a small test of 31 pairs (62 statements), and the level-based questions were tried with one quantization of Qwen3.5-9B only. It shows tendencies.
+
+## 1. Purpose
+
+The output of a decision model is often cut at a threshold and used to branch (for example `jev.yes(state, question, 0.5)`). When some inputs do not give enough to decide, the branch taken depends on which value the model uses to express "I cannot tell".
+
+This report checks two things:
+
+1. How a general local model answers uncertain questions differently from Jev.
+2. How much of that difference can be closed by how the script asks.
+
+Two properties expected of a model that answers with probabilities were used as criteria:
+
+- **Criterion 1**: the probabilities of the two statements of a pair add up to 1.
+- **Criterion 2**: a 50/50 statement gets 0.5.
+
+## 2. Method
+
+### 2.1 Models
+
+| Model | Kind | How the answer is produced |
+| --- | --- | --- |
+| Jev | TypeSafe's decision model (API) | The API returns probabilities |
+| Clef-Flash | Qwen3.5-9B further trained for decisions (`ggml-org/Clef-Flash-GGUF`, Q8_0) | The values from the decision head are turned into probabilities |
+| Qwen3.5-9B | General LLM (`unsloth/Qwen3.5-9B-GGUF`, Q8_0) | The probabilities of the option letters as the next token are read |
+
+For a general model, the options are shown with letters (for `jev.noul`, A: Yes and B: No) and the model is told to answer with one letter only. No text is generated. The probability of each letter as the next token is read, and the values are rescaled so that the letters add up to 1 (any other output is left out). This is the method proposed by SemIf (OpenJev). The values in this report are uncalibrated. Calibration is discussed in sections 3.2 and 5. Because nothing is sampled, there is no random seed or temperature setting.
+
+The instruction is as follows. The statement, the question, and the options are passed together as JSON.
+
+```text
+Apply the supplied criterion to the supplied evidence. Choose exactly one listed option. Respond with only its uppercase letter, with no explanation or reasoning.
+```
+
+All three models were run through this node's own decision code (v1.3.3). Questions were sent one at a time.
+
+### 2.2 Statements
+
+There are 31 pairs of statements, 62 statements in total. Statements and questions are in English. The full list is in [`data/items.json`](data/items.json).
+
+| Group | Code | Pairs | Content | Example |
+| --- | --- | --- | --- | --- |
+| Known facts | K | 6 | The first is true, the second false | Paris is the capital of France. / ... of Germany. |
+| Near-certain future | N | 6 | The first is almost sure to happen | The sun will rise tomorrow. / ... will not rise tomorrow. |
+| Future with a tendency | T | 6 | The first is more likely, but not certain | The S&P 500 index will be higher one year from now ... / lower ... |
+| 50/50 | U | 8 | Both are about 50% | The next coin I flip will land heads. / ... tails. |
+| Known probability | P | 5 | The exact probability can be calculated | The next roll of a fair six-sided die will show a 1. (1/6) / ... a number other than 1. (5/6) |
+
+The 26 pairs of K, N, T, and U were asked to all three models. The 5 pairs of P were added later and asked to Qwen3.5-9B only.
+
+### 2.3 Questions
+
+| Code | Function | Question | Options |
+| --- | --- | --- | --- |
+| Q1 | `jev.noul` | Is this statement true? | Yes / No |
+| Q2 | `jev.noul` | Is this statement more likely to be true than false? | Yes / No |
+| F1 | `jev.noul` | Is this statement false? | Yes / No |
+| S6 | `jev.score` | How likely is this statement to be true? | 0-10%, 10-30%, 30-50%, 50-70%, 70-90%, 90-100% |
+| S11 | `jev.score` | How likely is this statement to be true? | 0%, 10%, ..., 100% |
+| F6 | `jev.score` | How likely is this statement to be false? | The same six levels as S6 |
+
+Q1 and Q2 were asked to all three models. F1, S6, S11, and F6 were asked to Qwen3.5-9B only. The level-based questions were asked twice, with the options in ascending and in descending order.
+
+As a control, Qwen3.5-9B was also asked to write the probability as a number (20 statements). The instruction was "What is the probability that the following statement is true? Answer with a single number from 0 to 100 (percent) and nothing else.", and the text was generated by taking the most likely token at each step.
+
+### 2.4 Measures
+
+- **Pair sum**: the sum of the values for the two statements of a pair (criterion 1).
+- **Distance from 0.5**: the mean absolute difference between the value and 0.5 over the 16 50/50 statements (criterion 2).
+- **Mean error**: the mean absolute difference between the value and the true probability over the 10 known-probability statements.
+- **Rank correlation**: Spearman's rank correlation between the value and the true probability over the same 10 statements.
+- **Negation-combined value**: (value for the statement + 1 − value for the negation) / 2. The pair sum is 1 by construction.
+- **Estimate from levels**: the value of each level (for S6 and F6 the midpoints 0.05, 0.20, 0.40, 0.60, 0.80, 0.95; for S11 the points 0, 0.1, ..., 1.0) weighted by the probability the model gave that level.
+
+Negation-combined values were not measured separately. They are calculated from the values already measured.
+
+## 3. Results
+
+### 3.1 The three models compared (Q1)
+
+Values are mean yes-probabilities, with the first statement of the pair on the left and the second on the right. The results of Qwen3.5-9B for "Future with a tendency" and "50/50", shown in bold italics, are far from criteria 1 and 2 of section 1, and are examined further from section 3.2 on.
+
+| Group | Jev | Clef-Flash | Qwen3.5-9B |
+| --- | --- | --- | --- |
+| Known facts | 0.92 / 0.03 | 0.95 / 0.02 | 0.99 / 0.01 |
+| Near-certain future | 0.88 / 0.07 | 0.86 / 0.11 | 0.85 / 0.13 |
+| Future with a tendency | 0.58 / 0.37 | 0.62 / 0.36 | ***0.07 / 0.08*** |
+| 50/50 | 0.44 / 0.44 | 0.36 / 0.26 | ***0.07 / 0.04*** |
+
+| Measure | Jev | Clef-Flash | Qwen3.5-9B |
+| --- | --- | --- | --- |
+| Pair sum: known facts | 0.95 | 0.97 | 1.00 |
+| Pair sum: near-certain future | 0.95 | 0.96 | 0.98 |
+| Pair sum: future with a tendency | 0.95 | 0.98 | ***0.15*** |
+| Pair sum: 50/50 | 0.88 | 0.62 | ***0.11*** |
+| Distance from 0.5 (50/50) | 0.06 | 0.22 | ***0.44*** |
+
+- For known facts, all three models gave a value on the correct side for all 12 statements.
+- Jev returned 0.34 to 0.51 for all 16 50/50 statements.
+- Qwen3.5-9B answered No to almost every uncertain statement (future with a tendency, 50/50). It does not answer No to everything about the future: it returned 0.99 for "The sun will rise tomorrow." (near-certain future).
+- Clef-Flash had a pair sum of 0.98 for the future with a tendency, very different from Qwen3.5-9B before tuning (0.15). For questions like coins and dice it stayed at 0.62 (the coin flip was 0.16 / 0.10).
+
+Clef-Flash and Qwen3.5-9B also differ in how the answer is produced. This test cannot separate whether the difference between the two comes from the tuned weights or from the way the answer is produced.
+
+Clef-Flash chose one side clearly for the future with a tendency (0.80 / 0.06 for the S&P 500, where Jev gave 0.53 / 0.31). However, it returned 0.09 / 0.81 for "It will not rain / will rain in Tokyo on this day next year", the opposite of the actual frequency, and 0.81 / 0.71 for "The next US president will be older / younger than 50 when taking office", answering Yes to both.
+
+### 3.2 What the No of Qwen3.5-9B represents
+
+From here on, the results of Qwen3.5-9B for "Future with a tendency" and "50/50" are examined.
+
+The first check was that the No of Qwen3.5-9B is **not the result of probability going to tokens other than the letters**. The share of the whole vocabulary's probability taken by A and B (`allowed_token_mass` in SemIf) was recorded for 124 questions: 62 statements × Q1 and Q2.
+
+| | Minimum | Median | Maximum |
+| --- | --- | --- | --- |
+| Share taken by A and B | 0.9971 | 0.9992 | 0.9999 |
+
+The most likely token other than A and B was "C", at 0.15% at most. Tokens that would begin a refusal or an explanation did not appear among the top 10. The model chooses B (No) clearly.
+
+**"Cannot tell" and "almost surely will not happen" get the same value.** The mean difference between the logits of A and B (positive means toward Yes) is as follows.
+
+| Kind of statement | Logit difference | As a probability |
+| --- | --- | --- |
+| True fact | +5.6 | about 1.00 |
+| Near-certain to happen | +2.9 | about 0.95 |
+| Future with a tendency | −3.0 | about 0.05 |
+| 50/50 | −3.3 | about 0.04 |
+| Near-certain not to happen | −3.3 | about 0.04 |
+| False fact | −6.2 | about 0.002 |
+
+**The values do not follow the order of the probabilities.** The Q1 answers for known-probability statements are below. The rank correlation over the 10 statements was −0.38.
+
+| Statement | True probability | Q1 value |
+| --- | --- | --- |
+| Drawing an ace from a deck | 0.08 | 0.03 |
+| Rolling a 1 | 0.17 | 0.02 |
+| Drawing a heart | 0.25 | 0.03 |
+| Rolling a 3, 4, 5, or 6 | 0.67 | 0.35 |
+| Drawing a card that is not a heart | 0.75 | 0.02 |
+| Rolling a number other than 1 | 0.83 | 0.01 |
+| Drawing a card that is not an ace | 0.92 | 0.02 |
+
+Qwen3.5-9B appears to answer "Is this statement true?" by the standard "can it be said to be true for certain?". The value that comes out is not the probability of the event but the confidence in the answer No. Because the value carries no information about the size of the probability, turning Q1 values into probabilities afterwards (calibration) is difficult.
+
+### 3.3 Changing the question (Q2)
+
+With the question changed to "Is this statement more likely to be true than false?":
+
+- Jev and Clef-Flash showed almost the same tendency as with Q1.
+- Qwen3.5-9B no longer answered No to everything. For the known-probability statements it chose the correct side (above or below 0.5) for 8 of 10.
+- However, the values of Qwen3.5-9B were close to 0 or 1 and did not express the size of the probability. The 50/50 statements swung to Yes or No depending on the statement (0.02 to 0.99; distance from 0.5 was 0.36).
+
+The answers of a general model depend strongly on the wording of the question.
+
+### 3.4 Asking about both the statement and its negation
+
+With Q1, Qwen3.5-9B answers Yes to "near-certain to happen" and No to "cannot tell". What it does not separate is "cannot tell" from "near-certain not to happen". The negation of the latter is "near-certain to happen", so asking about the negation as well separates them.
+
+| Kind of statement | Answer for the statement | Answer for the negation |
+| --- | --- | --- |
+| Near-certain to happen | Yes | No |
+| Near-certain not to happen | No | Yes |
+| Cannot tell | No | No |
+
+The statements were written as pairs, so the other statement of the pair was taken as the negation and the negation-combined value was calculated (Q1). The value in parentheses is from asking about the statement alone. Bold italics mark the items that were far from the criteria in section 3.1.
+
+| Group | Jev | Clef-Flash | Qwen3.5-9B |
+| --- | --- | --- | --- |
+| Known facts | 0.95 (0.92) | 0.96 (0.95) | 0.99 (0.99) |
+| Near-certain future | 0.90 (0.88) | 0.88 (0.86) | 0.86 (0.85) |
+| Future with a tendency | 0.60 (0.58) | 0.63 (0.62) | ***0.49 (0.07)*** |
+| 50/50 | 0.50 (0.44) | 0.55 (0.36) | ***0.51 (0.07)*** |
+
+| Distance from 0.5 (50/50) | Jev | Clef-Flash | Qwen3.5-9B |
+| --- | --- | --- | --- |
+| Statement alone | 0.06 | 0.22 | ***0.44*** |
+| Combined with the negation | 0.006 | 0.053 | ***0.015*** |
+
+For Qwen3.5-9B, cutting the two answers at 0.5 and sorting each pair into "true / false / cannot tell" gave the expected result for 29 of 31 pairs.
+
+- Known facts and near-certain future (12 pairs): 10 pairs were "true".
+- Future with a tendency, 50/50, and known probability (19 pairs): all 19 pairs were "cannot tell".
+
+The two pairs that missed were "It will snow / will not snow in Sapporo next January" (No to both) and "A total solar eclipse will / will not occur within the next five years" (Yes to both). A Yes to both is a sign that the answer cannot be trusted.
+
+This method does not recover the size of the probability. A statement with probability 0.08 and one with 0.92 both get a negation-combined value near 0.5 (for the known-probability statements the mean error was 0.244 and the rank correlation −0.12).
+
+### 3.5 Having the model pick a probability level
+
+The results below are for Qwen3.5-9B only.
+
+**Control: written as a number, the answer is almost exact.** The mean error over the 10 known-probability statements was 0.003 (for example "9" for a true probability of 7.7%, "16.67" for 16.7%, and "92" for 92.3%). For the coin flip it answered "50". The model has the knowledge of the probabilities. That the size of the probability does not show with Q1 comes from how the question is asked.
+
+| Statement | True probability | Number written |
+| --- | --- | --- |
+| Drawing an ace from a deck | 7.7% | 9 |
+| Rolling a 1 | 16.7% | 16.67 |
+| Drawing a heart | 25% | 25 |
+| Two heads in two coin flips | 25% | 25 |
+| Rolling a 1 or a 2 | 33.3% | 33 |
+| Rolling a 3, 4, 5, or 6 | 66.7% | 66 |
+| Drawing a card that is not a heart | 75% | 75 |
+| Not two heads in two coin flips | 75% | 75 |
+| Rolling a number other than 1 | 83.3% | 83 |
+| Drawing a card that is not an ace | 92.3% | 92 |
+| 50/50 statements (4: coin, die, time of birth, grains of sand) | 50% | 50 for all |
+| The sun will rise / will not rise tomorrow | — | 100 / 0 |
+| Paris is the capital of France / of Germany | — | 100 / 0 |
+| The S&P 500 will be higher / lower in one year | — | 65 / 50 |
+
+**Asked by level, the size of the probability shows to some degree.** The results for the 10 known-probability statements are as follows.
+
+| How it was asked | Rank correlation | Mean error |
+| --- | --- | --- |
+| Q1 (Yes / No) | −0.38 | — |
+| S6, ascending | +0.88 | 0.095 |
+| S6, descending | +0.76 | 0.208 |
+| S11, ascending | +0.77 | 0.202 |
+| S11, descending | +0.80 | 0.203 |
+| Written as a number | +1.00 | 0.003 |
+
+The two level-based questions are shown again here. Both use the question "How likely is this statement to be true?", and only the options differ.
+
+- **S6 (six ranges)**: 0-10%, 10-30%, 30-50%, 50-70%, 70-90%, 90-100%. The estimate uses the midpoint of each range (0.05, 0.20, 0.40, 0.60, 0.80, 0.95).
+- **S11 (eleven points)**: 0%, 10%, 20%, ..., 100%. The estimate uses the value of each point (0, 0.1, ..., 1.0).
+
+**Six ranges (S6) and eleven points (S11) are good at different things.**
+
+| Measure | S6 | S11 |
+| --- | --- | --- |
+| Distance from 0.5 (50/50; ascending / descending) | 0.118 / 0.140 | 0.027 / 0.029 |
+| Difference in the estimate between the two orders (mean of 62 statements) | 0.087 | 0.042 |
+| Pair sum: known facts (mean of the two orders) | 1.07 | 1.03 |
+| Pair sum: near-certain future (mean of the two orders) | 1.14 | 1.04 |
+
+- With S6, the 50/50 statements came out at about 0.60. 50% lies on the border between "30-50%" and "50-70%", and the model chose "50-70%" for 14 of 16 statements.
+- S6 depended strongly on the order. "Drawing a card that is not a heart" (true probability 0.75) was 0.75 in ascending order and 0.24 in descending order.
+- With S11, 14 of the 16 50/50 statements came out at 0.50, and the pair sums are close to 1. Asking about the statement alone almost meets criteria 1 and 2.
+- With S11, the size of the probability hardly shows, because the model chooses "50%" for most statements that are not certain. It chose "50%" both for "Drawing a heart" (0.25) and for "Drawing a card that is not a heart" (0.75).
+
+When there is a "50%" option, 50/50 statements are answered correctly, but uncertain statements gather there. S11 works much like sorting into "true / false / cannot tell".
+
+**Six ranges asked about both the statement and its negation cover both.**
+
+| How it was asked (S6) | Distance from 0.5 | Mean error | Rank correlation |
+| --- | --- | --- | --- |
+| Ascending, statement alone | 0.118 | 0.095 | +0.88 |
+| Ascending, combined with the negation | 0.027 | 0.069 | +0.90 |
+| Descending, combined with the negation | 0.034 | 0.148 | +0.92 |
+
+The shift of the 50/50 statements goes in the same direction for the statement and for the negation, so it cancels when the two are combined. The levels are better listed in ascending order.
+
+| Statement | True probability | Statement alone | Combined with the negation |
+| --- | --- | --- | --- |
+| Drawing an ace from a deck | 0.08 | 0.06 | 0.13 |
+| Rolling a 1 | 0.17 | 0.05 | 0.19 |
+| Drawing a heart | 0.25 | 0.32 | 0.29 |
+| Two heads in two coin flips | 0.25 | 0.06 | 0.12 |
+| Rolling a 1 or a 2 | 0.33 | 0.53 | 0.44 |
+
+### 3.6 "Is this statement false?" is not a substitute for the negation
+
+Instead of writing a negation, the statement was kept as it is and the question was reversed (F1, F6). The combined value was calculated as (Q1 value + 1 − F1 value) / 2.
+
+| Group | Combined with the negated statement | Combined with F1 |
+| --- | --- | --- |
+| Known facts | 0.99 | 0.99 |
+| Near-certain future | 0.86 | 0.87 |
+| Future with a tendency | 0.49 | 0.36 |
+| 50/50 | 0.51 | 0.42 |
+
+| Measure | Combined with the negated statement | Combined with the "false" question |
+| --- | --- | --- |
+| Distance from 0.5 (Yes / No) | 0.015 | 0.109 |
+| Distance from 0.5 (six ranges, ascending) | 0.027 | 0.072 |
+| Mean error (six ranges, ascending) | 0.069 | 0.146 |
+
+There is no difference for certain statements. What breaks is the uncertain ones. Asked "Is this statement true?" about an uncertain statement, Qwen3.5-9B almost always answers No, but asked "Is this statement false?" it sometimes answered Yes (for "The next coin I flip will land tails.", Q1 was 0.03 and F1 was 0.60). It does not say an uncertain statement is "true", but it says "false" more easily.
+
+The negated statement likely works because it is asked with the same question, so the bias of the question applies to both in the same way and cancels.
+
+### 3.7 Summary of the ways of asking (Qwen3.5-9B)
+
+| How it was asked | 50/50 at 0.5 | Size of the probability | Writing a negation | Questions |
+| --- | --- | --- | --- | --- |
+| Yes / No, statement alone | No | Does not show | Not needed | 1 |
+| Yes / No, combined with the negation | Yes | Does not show | Needed | 2 |
+| Eleven points, statement alone | Yes | Hardly shows | Not needed | 1 |
+| Six ranges, statement alone | Shifted (about 0.60) | Shows to some degree | Not needed | 1 |
+| Six ranges, combined with the negation | Yes | Shows to some degree | Needed | 2 |
+| Combined with the "false" question | Shifted | Does not show | Not needed | 2 |
+| Written as a number | Yes | Shows | Not needed | Needs text generation |
+
+## 4. Guidelines for using a general model
+
+From the above, the following can be said for scripts that use `jev.*` with a general local model.
+
+1. **Do not read a low `jev.noul` or `jev.yes` value as "unlikely".** It includes "cannot say it is true". Inputs that do not give enough to decide almost always take the No branch.
+2. **If "true / false / cannot tell" is enough, ask once with eleven points.**
+3. **If you ask Yes / No, ask about both the statement and its negation.** Treat No to both as "cannot tell", and Yes to both as an answer that cannot be trusted.
+4. **If you also want the size of the probability, list six ranges in ascending order and ask about both the statement and its negation.**
+5. **Write the negation as a statement.** Changing the question to "Is this statement false?" is not a substitute. For this reason, there is currently no way for the node to create the negated question automatically.
+6. **Put what you want to know directly into the question.** "Is it true" and "is it the more likely side" give very different answers.
+7. **Review thresholds when you change the model.** With the same script, the branch taken for uncertain inputs differs between Jev, Clef-Flash, and a general model.
+
+Asking about both the statement and its negation. Numeric outputs (`ov1` and so on) are integers, so the negation-combined value is multiplied by 100 and output as 0 to 100.
+
+```python
+# it1: the statement, it2: its negation
+# ov1: negation-combined value (0-100), ot1: classification
+a = jev.noul(it1, "Is this statement true?")
+b = jev.noul(it2, "Is this statement true?")
+ov1 = (a + 1 - b) / 2 * 100
+
+if a < 0.5 and b < 0.5:
+    ot1 = "unsure"
+elif a >= 0.5 and b >= 0.5:
+    ot1 = "contradiction"
+elif a >= 0.5:
+    ot1 = "true"
+else:
+    ot1 = "false"
+```
+
+Asking with eleven points. `jev.score` returns the expected index of the chosen level (0 to 10), so multiplying by 10 gives a value from 0 to 100.
+
+```python
+levels = ["0%", "10%", "20%", "30%", "40%", "50%", "60%", "70%", "80%", "90%", "100%"]
+ov1 = jev.score(it1, "How likely is this statement to be true?", levels) * 10
+```
+
+With six ranges, the midpoints are not evenly spaced, so multiplying the `jev.score` value by 20 does not give the same value as 100 times the estimate in section 3.5. Keep this in mind when using it as an approximation.
+
+## 5. Limitations
+
+- This is a small test of 31 pairs (62 statements). There are only 5 known-probability pairs (10 statements), and the comparison of the ways of asking rests on these few statements.
+- It does not rank the models. Questions about the future have no correct answer, and the accuracy of the predictions was not measured.
+- Combining with the negation was thought of while looking at this data, and its effect was checked on the same data. It has not been checked on other statements.
+- Some pairs are not strict negations (such as "higher" and "lower"). The result can change with how the negation is written.
+- The level-based questions (S6, S11), the effect of combining with the negation for levels, and the "false" questions (F1, F6) were tried with Qwen3.5-9B only. They were not asked to Jev or Clef-Flash.
+- Only two ways of defining the levels were tried.
+- The only general model is Qwen3.5-9B at Q8_0. Another model, another quantization, or another decision method may give different results.
+- The values of the general model are uncalibrated. SemIf (OpenJev) recommends calibrating for each use, but calibration was not tried here. As section 3.2 shows, the Yes / No values carry no information about the size of the probability, and no statements suitable for calibration have been prepared.
+- Every question was measured once. The values of a local model do not change under the same conditions, because nothing is sampled. Jev's values vary a little for the same question: when the 16 50/50 statements were asked three times with Q1, the largest range was 0.04.
+- Statements and questions are in English. Other languages were not tried.
+
+## 6. Data
+
+The statements and the results are in [`data/`](data/). A statement ID is the pair code followed by `a` (first statement) or `b` (second statement), for example `U1a`.
+
+| File | Content |
+| --- | --- |
+| `items.json` | Statements (`pairs`), question texts (`questions`), group descriptions (`groups`), and the true probabilities of the known-probability statements (`probabilities`) |
+| `results_jev.json` | Jev. `p_yes` for Q1 and Q2. Q1 of the 50/50 statements has three runs (`run` 1 to 3) |
+| `results_clef_flash.json` | Clef-Flash. `p_yes` for Q1 and Q2 |
+| `results_qwen35.json` | Qwen3.5-9B. `p_yes` for Q1 and Q2 (26 pairs) |
+| `results_qwen35_mass.json` | Qwen3.5-9B. For Q1 and Q2 (31 pairs): `p_yes`, the logits of A and B, `allowed_token_mass`, and the 10 most likely tokens |
+| `results_qwen35_score.json` | Qwen3.5-9B. The probability of each level for S6 (`score`) and the answers of the written-number control (`text`) |
+| `results_qwen35_score11.json` | Qwen3.5-9B. The probability of each level for S11 |
+| `results_qwen35_false.json` | Qwen3.5-9B. `p_yes` for F1 (`noul`) and the probability of each level for F6 (`score`) |
+
+The level probabilities (`p`) are stored from the lowest level to the highest, whichever order the options were shown in. `order` is the order shown.
+
+## 7. Measurement conditions
+
+- Dates: October 5, 2026 (comparison of the three models), October 6 to 8, 2026 (additional measurements with Qwen3.5-9B)
+- Node: ComfyUI-ScriptFlow v1.3.3
+- Jev: TypeSafe API (typesafe-sdk 0.7.1). The model name returned by the API was `jev-1.13.0` when checked on October 8, 2026. The model name at the time of measurement (October 5) was not recorded.
+- Local models: llama-cpp-python 0.4.2 (SYCL build), Windows 11