Benchmarks¶

Every model answered the same questions and was scored by the same code. TypeSafe Jev was measured directly through TypeSafe's own API, not taken from published figures. Full tables, per-task results and methodology: COMPARISON.md.

Reproduce every number: python benchmarks/report.py rebuilds all the tables from the committed results, and python benchmarks/run.py --model <name> re-scores any model (benchmarks/).

The test sets¶

• typed-decisions (dataset): 2,000 decisions across four business workflows, scored with the Jev-vs-Laya harness published by Kameshwara Pavan kumar Mantha and the Antz AI team. nano, small-td, medium-td, large-td and Laya's typed-decisions checkpoint were fine-tuned on its train split; the test split was never used for training or model selection. Zero-shot models are a reference there, not a head-to-head.
• 200 general decisions none of the models trained on: BANKING77 intent (78 options), BoolQ yes/no, Yelp 1–5 star rating and ChaosNLI (100 human labels per item).
• Laya's application battery: Laya's own 10-task benchmark, at a pinned commit.

OpenDecider vs TypeSafe Jev¶

Where Jev leads: Laya's application battery (0.774, and 0.803 on the five tasks Laya was not trained on), phishing (0.897 vs our 0.63–0.70), jailbreak detection (0.940), spam (0.985), model routing (0.975), BoolQ-style yes/no reading (0.94) and typed-decisions without fine-tuning (0.754 vs small's 0.672).

OpenDecider vs Laya¶

Like for like on typed-decisions (both fine-tuned on the train split), nano leads Laya's typed-decisions checkpoint by +0.030 (95% CI +0.014 to +0.044). Where Laya leads: the five datasets it was trained on (AG News 0.95, Enron spam 0.99, phishing 0.98), multilingual use (a 100+ language checkpoint; OpenDecider is evaluated in English only), and single-question speed on short inputs, where it is in the same range as nano.

OpenDecider vs Microsoft-Decision-1¶

Microsoft-Decision-1 is a 9B decision model (built on Qwen3.5-9B) served through Microsoft Foundry. It answers the same typed questions as Jev, so it ran through the same harness (--model foundry, our own Foundry deployment, 2026-10-10). Microsoft publishes neither its training data nor per-dataset results.

Paired 95% CIs (COMPARISON.md, section 4): on typed-decisions Decision-1 is not distinguishable from nano, small-td and medium-td (OpenDecider's were fine-tuned on the train split), and ahead of Jev (+0.029). On general decisions it is not distinguishable from medium-td and large-td and ahead of Jev (+0.065). On Laya's battery it is ahead of every OpenDecider model and behind Jev (−0.023).

Where Decision-1 leads: general decisions (0.795) and calibration (ECE 0.072, second only to Claude Fable 5.1), BANKING77 in Laya's battery (0.890), and phishing (0.777 vs our 0.63–0.70). Where it trails: toxicity (0.530 vs 0.77–0.81 for OpenDecider's 4B-and-larger models), the spread of human votes (JSD 0.079 vs 0.030–0.045), and it is API-only.

Frontier LLMs (same 200 general decisions)¶

Only Claude Fable 5.1, Microsoft-Decision-1 and GPT-6 Astra beat opendecider-medium-td here, all through an API with a per-call bill. The 200-item set is about ±3 points: Decision-1's lead over medium-td is within noise (+0.030, 95% CI −0.025 to +0.090), and medium-td, DeepSeek V4.1 Flash and MiniMax M3 are close. Decision-1's latency is the round trip from India to Foundry's East US region. Foundry lists it at $0.042 per million input tokens (output free); the runs did not record token counts, so there is no per-1,000 figure.

Prompt guard¶

opendecider.guard against Laya's guard on the test splits of three public datasets (2,438 prompts): deepset/prompt-injections (116, English and German), jackhhao/jailbreak-classification (262) and xTRam1/safe-guard-prompt-injection (2,060). Both ask the same two questions, the attack questions of Laya's guard preset. Each model's threshold was set on the train splits by a rule fixed before any test result was seen (the threshold that maximises balanced accuracy, averaged over the three datasets) and applied once to test. No test prompt is in OpenDecider's training data (0 text overlaps).

Accuracy per dataset. Paired with Laya's English checkpoint, opendecider-small-td is ahead by 0.038 overall (95% CI +0.023 to +0.052) and by 0.050 on safeguard, within noise on deepset (+0.060, CI −0.017 to +0.138), and behind on jailbreak-classification (−0.073, CI −0.111 to −0.034), where it flags 21% of the benign prompts against Laya's 6.5%. safeguard is 85% of the prompts, so it weighs most in the totals; averaged per dataset the two are close (balanced accuracy 0.889 vs 0.884). Tuning the threshold on the train splits did not help on test: at 0.5, small-td scores 0.945 and Laya 0.909, slightly above their tuned results. The guard keeps the train-split thresholds, as the rule set out.

Quantised builds¶

The published GGUF builds (run through Ollama) and MLX builds, on the same test splits. Each build's threshold was set on the train splits by the same rule as its model's. A 4-bit build scores higher than its model, so at 0.5 it flags far more benign prompts; at its own threshold it screens as well as the full model.

The guard uses each build's threshold when the model name says which build it is: an Ollama name with its tag (hf.co/manjunathshiva/opendecider-small-td-GGUF:Q4_K_M), an LM Studio name with its variant (opendecider-small-td@q4_k_m), or the MLX build's name. A name that does not say (an Ollama name without its tag, an LM Studio model loaded without its variant in the name, or a GGUF given a plain name) gets its model's threshold or 0.5, both at or below every build's, so it flags more rather than less. Averaged per dataset (balanced accuracy), the builds' thresholds and 0.5 are within 0.005 of each other except small's Q4_K_M (0.885 vs 0.853); the totals move more because two thirds of the test prompts are benign (1,589 of 2,438, 1,410 of them in safeguard), and a 4-bit build's extra flags at 0.5 fall on them.

python benchmarks/guard.py report prints these tables from the committed results, and two more setups: one yes/no question (Laya's published deepset question) and Laya's whole guard preset scored with LayaGuardrail's own rule. See Agent guardrails for what the numbers mean in practice.

In the browser¶

@opendecider/web runs opendecider-nano as ONNX: q8 (8-bit weights, float32 elsewhere; WebAssembly's default), q8f16 (8-bit weights, float16 elsewhere; WebGPU's default) and fp16 (float16 throughout; on WebGPU about 7 times faster than q8f16 for many questions at once, which is what OpenDecider Focus uses). Each was scored with native ONNX Runtime on the CPU against the PyTorch model's committed answers. The browser's WebAssembly gives the same logits as native ONNX Runtime to 1e-6 for q8 and q8f16, and within 0.01 for fp16. WebGPU computes in float16: there (Chrome, Apple M4 Max), fp16 gave PyTorch's answer on all 400 general and 99.45% of the 2,000 typed-decisions questions, and q8f16 on 99.5% and 99.3%, with the same typed-decisions accuracy. A build ships only if it gives PyTorch's answer on at least 99% of the questions and loses at most 0.5 points on each benchmark.

Not shipped: 4-bit builds (286–410 MiB) changed 2.5–4.3% of the answers (typed-decisions 0.790), whatever the quantisation method (symmetric, asymmetric, HQQ), and dynamic int8 gave different logits in the browser's WebAssembly than in native ONNX Runtime, so its benchmark would not describe what runs in the browser.

As a guard (the attack questions, each build at its own train-split threshold):

YouTube feed¶

The questions of the Chrome extension on 800 videos from a public set of US trending videos (2019–2020, MIT), 100 from each of 8 categories, with the creator's category as the label: News, Howto, Education and Science & Technology are kept, Music, Gaming, Comedy and Entertainment hidden. Below, the test half (400 videos); the dev half gives the same ranking. Balanced accuracy at 0.5 (the mean of the share of wanted videos kept and of unwanted ones hidden), so a rule that matches 100 of 400 videos cannot score by answering no.

"Quietly's request" is the one Quietly sends Jev for each video: a yes/no question with the title in it and Quietly's keep and hide lists as its true and false descriptions, and the same state for every video. opendecider-nano and Laya judge the state, so they answer it the same way for every video; the extension puts the video in the state. The creator's category is a noisy label (a cooking show is often filed under Entertainment), so read the kinds column as a comparison, not as the share of a feed sorted right. python benchmarks/run.py --suites youtube --model <model> reruns a row.

Speed¶

Throughput under concurrent load is in Serve it.