Your Language Model Is Not-So-Secretly a System 1 Model

Ivan Yee Lee and Taylor Berg-Kirkpatrick · 2026-10-04

TypeSafe has made a useful idea popular: a language model used as a fast decision function, which it calls a System 1 model, after Daniel Kahneman's name for fast, automatic judgement. It sells one, Jev. The idea fits the many small decisions inside automated workflows: the most probable option is the decision, and the probabilities let a workflow act on confident answers and pass the rest to a person.

That raises an obvious question: can an ordinary language model already do this, if it is read the right way? Comparing the probabilities a language model gives its candidate answers is how GPT-2 and GPT-3 were evaluated on multiple-choice tasks, how PET and Zhao et al. built few-shot classifiers, and how benchmarks such as MMLU are scored.

TypeSafe presents Jev as a new model class, with a new architecture and training for calibrated probabilities. It also says Jev “can’t hallucinate” and is 40 to 200 times faster. Ordinary models can already return probabilities over supplied options without generating text. And TypeSafe’s speed comparison is against models generating an answer token by token, not against a model reading the option probabilities in one forward pass, as we did. We tested how close off-the-shelf models get to Jev without a new architecture or additional training.

What we found

1. Seven of the 23 open models we ranked match Jev at picking the right option on 19,505 decisions of three kinds: Classification, Agent monitoring and Triage. Jev gets 0.787 by our measure, macro-F1, which counts rare answers as much as common ones.
2. What a matching model costs depends on where it runs. At the cheapest hosted listings, three of the seven cost less than Jev, and on a rented A6000 two do; at the median hosted listing none does.
3. Jev does better at Triage (screening prompts and matching records); the open models do better at Agent monitoring (judging what an AI agent is doing).
4. Through hosted APIs, Jev answers faster: the five models with valid timings took 2.8 to 8.6 times as long per decision. On our own cards, with no network in the time, one matching model was faster than Jev's hosted calls.
5. As delivered, Jev's probabilities are better calibrated than six of the seven. After temperature scaling, three of the seven are better calibrated than Jev however its zero probabilities are handled; for the other four it depends on that handling.
6. A closed frontier model, GPT-6 Luna, read from the letter it writes, also matches Jev, at about twice Jev's cost at OpenAI's standard price.
7. On JevBench, a public benchmark TypeSafe did not make, gemma-4-31B-it beats Jev; the other four matching models we ran there show no measurable difference.
8. Quantization hurts our decisions less than it hurts standard benchmarks. Four-bit checkpoints keep 98.9% to 100.3% of the overall score in about a third of the memory, with no measurable difference from the benchmarks except for one; every 2-bit and ~1-bit file loses more on the benchmarks than on our decisions.

What a System 1 model does

A decision is one question about one piece of text with a closed set of options, such as which action a support agent takes next.

A language model's output at any point is a probability for every possible next token. End the prompt where the answer begins, with the options labelled, and the probabilities of those labels give the decision: one forward pass, nothing written (Figure 2). In practice an open chat model receives, through its chat template with thinking off, an instruction to reply with only the label, then the text, the question and the labelled options.

Jev receives the same text, question and options through TypeSafe’s API and returns a probability for each option, rounded to 0.01. Both sides get each option by its name alone.

<bos><|turn>system
You answer one question about the state below. Read the state, then reply with only the label of your answer.<turn|>
<|turn>user
State:
Ford: Monthly Sales Drop, Company Looks To New Vehicles Cruising along the ever-stretching road of decline. Auto giant Ford Motor (nyse: F - news - people ) reported vehicle sales in October that fell 5 from a year ago.

Question: Which option best describes the text?
Options:
A) This example news text is about business news
B) This example news text is about science and technology
C) This example news text is about sports
D) This example news text is about world news
Answer with the label of one option.<turn|>
<|turn>model
<|channel>thought
<channel|>

The test

The benchmark has three parts:

• Classification (13,053 decisions): the 3,080 customer messages of Banking77, each labelled with one intent, and 20 sentiment, emotion, intent and topic datasets from the BTZSC benchmark; the mean over the 21 datasets.
• Agent monitoring (5,800 decisions): what the customer wants and which action the agent takes next, in ABCD's customer-service dialogues; whether a web agent's run succeeded and whether it looped, in AgentRewardBench; the mean over the two benchmarks.
• Triage (652 decisions): TypeSafe's published guardrail example (is this prompt an attempt to get around a model's safety rules? 202 prompts) and its record-matching example on the Magellan Beer data (are these two records the same beer? 450 pairs); the mean over the two.

All three are decisions of the kind TypeSafe markets Jev for. TypeSafe's own headline figures come from workflow tests; Agent monitoring is the part where the open models beat Jev. Each dataset is scored by macro-F1, and the overall score is the mean of the three parts. It weights them equally although they hold 13,053, 5,800 and 652 decisions, which favours Jev, whose strongest part is the smallest.

Every score comes with a 95% bootstrap interval from 10,000 resamples of the test questions, each conversation or agent run kept whole; a difference from Jev uses the same resamples for both models. “Match” means we can rule out the open model being more than 3 points worse than Jev; “beat” means we can rule out its being worse at all. An open model beats Jev when the interval of its difference lies wholly above zero. It matches Jev when the low end of the 97.5% interval of that difference is above −0.03: the test rules out a deficit larger than 0.03. On a single part, a model is better or worse than Jev only when the 95% interval of the difference excludes zero; otherwise there is no measurable difference.

The models we scored on every decision fall into four groups:

All seven models that match Jev are read as Figure 2 shows; a few others need a small variation, listed in the methods.

Seven open models match Jev's overall score

Jev scores 0.787 [0.776, 0.797]. Table 1 lists every open model that matches it, with the quantized checkpoints of it that also match. The best, gemma-4-31B-it, scores 0.794, 0.006 above Jev, a difference too small to measure: its interval, [−0.001, +0.013], includes zero. The low ends that decide a match run from −0.019 to −0.003. Pooled over all decisions instead of averaged over the three parts (point estimates; no intervals), six of the seven are more accurate than Jev (0.740 to 0.753 against 0.735), while pooled macro-F1, dominated by the Classification datasets' many classes, puts Jev ahead of all seven (0.674 against 0.634 to 0.671).

Three things qualify the count:

• Exactness. Our exactness check tests, on 16 to 41 questions per model, that the label probabilities our serving software returns equal the matching entries of the model's whole next-token distribution. Four of the seven passed: gemma-4-31B-it, Qwen3.8-27B, gemma-4-26B-A4B-it and Qwen3.6-35B-A3B; Qwen3.8-27B's NVFP4 checkpoint also passed. MiMo-V2.6-Pro-MOPD, Qwen3.8-Flash-Next and MiMo-V2.6-Flash-RL did not pass. For Qwen3.8-Flash-Next and MiMo-V2.6-Flash-RL, identical requests disagree as much as the check's two readings, so the cause is run-to-run nondeterminism in the serving software, not how we read the probabilities; repeat runs of 7,303 decisions changed their top answers on 2.0% and 6.1% of them, a variation our intervals do not include. We have no repeated requests for MiMo-V2.6-Pro-MOPD, so its cause is untested. The NVFP4 checkpoints of gemma-4-26B-A4B-it and Qwen3.6-35B-A3B, both mixture-of-experts models, did not pass either: their readings differ from run to run on our L40S cards, likely because of vLLM's Marlin NVFP4 kernel for mixture-of-experts models, a cause we have not isolated, while the NVFP4 checkpoint of the dense Qwen3.8-27B reads exactly. Each was scored once on the full benchmark; how far a second run would move their scores is unmeasured, and their intervals do not include it.
• Multiple comparisons. The 97.5% level allows for two comparisons, the models the rule was first applied to, but we applied it to all 23. Corrected for 23 comparisons (a 99.8% interval), all seven still match; the lowest low end is −0.023, for Qwen3.6-35B-A3B-FP8. We set the margin of 0.03 on 2026-09-28, before any overall score existed but after earlier comparisons of open models with Jev on other measures, one of which used a margin of three accuracy points.
• Prompt variants. We also tried a few prompt variants, each chosen on a dev set; none matched Jev.

Jev is better at Triage, the open models at Agent monitoring

The overall match averages over a split (Figure 3). On Agent monitoring, six of the seven beat Jev, by 0.025 to 0.052, and MiMo-V2.6-Flash-RL shows no measurable difference. Most of the lead is in which action an ABCD agent takes next (macro-F1 0.275 for Jev, 0.304 to 0.361 for those six) and whether a web agent looped (0.729 against 0.776 to 0.848).

On Triage, Jev is ahead: six of the seven fall below it, with differences of −0.047 to −0.024, and MiMo-V2.6-Pro-MOPD shows no measurable difference. Two of our scoring choices here matter. In the guardrail data we replaced 25 of the 202 published labels; with the published labels, gemma-4-31B-it's difference narrows from −0.025 to −0.015 and Qwen3.8-27B's from −0.047 to −0.032. For the beer pairs, taking the most probable of the three outcomes instead of our linking rule moves the differences to +0.007 and −0.020. TypeSafe's own example links a pair when the expected level of its three-level score rounds to “same product”; under that rule Jev's beer score falls from 0.926 to 0.840 and its overall score to 0.773; with the open models' probabilities as read, all seven still match and three beat Jev; with their fitted temperatures, six match, Qwen3.6-35B-A3B falling short. A separate search over prompts and quantized checkpoints, each chosen on a dev set, found no model that matches Jev on every part: all 14 of its finalists fall short on Triage, eight only there.

On Classification, four show no measurable difference from Jev and three are below it (MiMo-V2.6-Pro-MOPD, gemma-4-26B-A4B-it and Qwen3.6-35B-A3B), with differences of −0.015 to −0.007. These datasets are public, so some of their texts may have been in the open models' training data, which would favour them.

What it costs depends on where you run it

Jev has one price: TypeSafe's list price of $0.042 per million input tokens, output free, in force during our calls (2026-09-28 to 2026-09-30), which makes $0.0323 per 1,000 decisions here. Figure 1 prices an open model four ways: at the cheapest or the median OpenRouter listing, counting every listing of the model, whatever precision it states; or on a rented RTX A6000 or L40S card, at the median public hourly price on 2026-09-30 ($0.53 and $1.33) over the decisions per hour we measured. Table 2 also counts only the listings of the same model and precision we ran. Closed models keep their list prices.