small, fast, honest decision models
Replace LLM calls for routing, guardrails and triage with typed answers in milliseconds. Every answer carries a calibrated probability and an abstain flag with a coverage guarantee. Open weights, open data, runs on a CPU or in this browser tab.
from verdict import Verdict
v = Verdict() # 118M params, 100+ languages
v.choose("Billed twice, refund or we cancel", ["billing", "technical", "sales"]) # billing p=0.71
v.check("Can I talk to a person?", claim="the user asks for a human") # True p=0.84
d = v.compile(question).fit(examples).calibrate(held_out, coverage=0.9) # seconds, on a CPU
d("the courier lost my parcel").confidence.abstain # False: it commitsverdict-small, the trained 118M multilingual encoder (the same weights the Python package loads), running in your browser on WebGPU or WASM. Type in any language. Zero-shot probabilities here are uncalibrated; the package's calibrate() adds temperature scaling and the conformal abstain set.
Every Verdict figure comes from python -m bench.run with a no-leakage protocol: heads on train minus a fixed calibration slice, calibration on that slice only, scores on the untouched test split. Banking77, SST-5 and ToxicChat are never in the training mix. Jev and Laya figures are from their own docs and third-party evals.
| Jev (closed) | Laya (open) | kev (open) | Verdict | |
|---|---|---|---|---|
| Banking77, zero-shot | 0.80-0.87 | 0.425 | 0.425 | 0.594 base · 0.556 trained · 0.650 with reranker |
| Banking77, 16 labels per class | not possible | hours on 2 GPUs | GPU delta fine-tune | 0.860 in 0.9 s on a CPU |
| Banking77, full train | – | – | – | 0.920 |
| CLINC150, 16 labels per class | 0.87 (zero-shot) | – | – | 0.928 |
| MASSIVE, Hindi input, English labels, full | English first | 0.451 non-English mean | English only | 0.863 |
| SST-5 zero-shot (score) | – | 0.372 | – | 0.403 trained |
| ToxicChat zero-shot (check) | – | 0.755 acc | – | AUROC 0.892 trained |
| typed-decisions (2,000 decisions) | 0.727 | 0.766 fine-tuned | not published | 0.706 fine-tuned (bi-encoder by design; cross-encoder variant is v2) |
| Calibration (ECE) | 0.144-0.246 | 0.466 as shipped | "still a bit overconfident" | 0.01-0.03 |
| "I don't know" | opaque confidence | act head always 1.0 | confidence threshold | conformal set, coverage guarantee |
| Option order changes the answer | yes | yes | yes (13.75 pp) | never, by construction |
| Max options | 255 | ~20 before degrading | collapses past ~20 | unbounded |
| Latency | 70-500 ms network | 33 ms GPU · 329 ms+ CPU | 12-26 ms H100 · 780 ms M5 | 0.5 ms batched (M-series) · 4 ms CPU int8 |
| Wire format | /v1/systemone | /v1/systemone | /v1/systemone | /v1/systemone + native + MCP |
| Where it is weak | >255 options, non-English, no fine-tune | zero-shot near chance, 512 ctx | overconfident, English, no world knowledge | typed-decisions fine-tuned (0.71 vs 0.77); Banking77 zero-shot vs Jev |
Full table with fit time and per-suite ECE: docs/BENCHMARKS.md. Sources: docs/RESEARCH.md.
Verdict answers what it is sure about and abstains on the rest. Pick a coverage target; see how much of your traffic never reaches an LLM. Curves are the measured Banking77 rows (16 labels per class).
Input and options go through one small bi-encoder. 10 or 10,000 options cost the same per input. An optional cross-encoder reranks the top-k.
A prototype head from one example per class, a logistic head initialised at the zero-shot solution when you have more. No fine-tuning, no GPU, deterministic.
Temperature scaling for honest probabilities, then split conformal prediction: on data like your held-out set, the true label is in the set at least coverage of the time.