SolitaryThinker
6220d02746
The last audit stress test. A cheap draft model proposes K tokens, the target verifies them in one batched step, and SpeculativeARLoop accepts the matching prefix + one target correction — a variable accepted-length per round (a ragged AR loop the model owns). - Exactness: the emitted sequence equals the target's OWN greedy decode for any draft quality (every accepted token is one the target would produce; the correction is the target's token) — the speedup is free. - Speedup scales with accept rate: draft-agree 0.3→1x, 0.7→3x, 1.0→4x=K tokens/round (fewer verify_rounds, the expensive model's latency steps, for the same output). - Two components (draft + target) co-scheduled on one resident instance; each round an AR_TOKEN WorkUnit. models/speculative/ (loop+card+program); backend ToyTargetModel/ToyDraftModel with a shared length-dependent target formula (no degenerate fixed point). 5 tests; full suite 153 passed. designv4 §9.16 + counts (153 tests, 26 files).
20 lines
894 B
Python
20 lines
894 B
Python
"""Speculative-decoding program: tokenize → spec_decode (draft+verify) → emit_text (design_v3 §9.16)."""
|
|
from __future__ import annotations
|
|
|
|
from ...program import ComponentNode, ModelLoopNode, Program, ProgramKind
|
|
from ..omni import emit_text_node, tokenize_node
|
|
|
|
|
|
def build_speculative_program() -> Program:
|
|
return Program(
|
|
program_id="speculative.decode", kind=ProgramKind.INLINE,
|
|
nodes=[
|
|
ComponentNode("tokenize", fn=tokenize_node, writes=("prompt_tokens",)),
|
|
ModelLoopNode("spec", loop_id="spec_decode", output_slot="spec_out",
|
|
reads=("prompt_tokens",), writes=("spec_out",)),
|
|
ComponentNode("emit_text", fn=emit_text_node("spec_out", "text"),
|
|
reads=("spec_out",), writes=("text",)),
|
|
],
|
|
output_artifacts={"text": "text", "tokens": "spec_out"},
|
|
).validate()
|