When a Decision Model Says 90%, How Often Is It Right?

On five-level movie-review sentiment, two of the three decision models I tested were right 67.4% and 60.7% of the time on the answers they gave a probability of 90% or more. The third never stated 90% on that task. On banking intents and yes/no questions about a passage, the same 90% meant right 88.5% to 96.5% of the time.

The three models are Google’s Gemini Decision 1, OpenAI’s GPT-6 Luna through its Decisions API, and TypeSafe’s Jev 1.14. Each received the same 1,200 public, labelled items on 14 November 2026. This post describes what they returned, how well their probabilities matched their accuracy, how fast and expensive they were, and what I would check before putting one into a product. It is what each model did on three public sets, not a ranking.

What a decision model returns

A decision request contains an input and one or more typed questions: yes or no, pick one option from a list, or rate on an ordered scale. The response is a typed answer with probabilities, not prose. Each provider describes this in its own words:

• Google describes Gemini Decision 1 as evaluating typed questions “in a single prefill pass” and returning “calibrated outcome probabilities with zero output tokens” (model page (opens in a new tab)). Yes/no answers return a probability from 0.01 to 0.99.
• OpenAI says “The Decisions API is now generally available”, with gpt-6-luna as “the only currently supported model”, and “Output tokens are not charged” (guide (opens in a new tab)).
• TypeSafe says Jev returns “typed values and probability distributions that your code can branch on, sort by, and route with”, with “No text generation, no parsing” (introduction (opens in a new tab)).

I sent all three the same probe: “The blue mug I ordered on Monday arrived cracked on Wednesday.” One question asked which support queue should handle it: billing, replacements, or general. All three picked replacements, at 1.0 from Jev and OpenAI and 0.984 from Gemini. Your code branches on that label and its probability instead of parsing a sentence.

The request shapes are close. This is the Gemini version of the queue question; Jev’s is the same with state in place of input, and OpenAI uses an array of named questions:

{
  "model": "gemini-decision-1",
  "input": "The blue mug I ordered on Monday arrived cracked on Wednesday.",
  "questions": {
    "queue": {
      "type": "choice",
      "instructions": "Which support queue should handle this message?",
      "criteria": {
        "billing": "Payment, invoice, or refund questions.",
        "replacements": "Damaged, missing, or wrong items.",
        "general": "Product questions and account updates."
      }
    }
  }
}

How I measured it

A model is calibrated if, of all the answers it gives at about 90%, about nine in ten are right. Calibration can change with the task, so I used three public sets, 400 items each, drawn with a fixed random seed:

Every model got the same items with the same question wording, one request per item. For the scale questions, the most probable level counts as the answer.

For each model and set, I split the answers into ten equal-width bands by the top answer’s stated probability and compared each band’s average stated probability with its share correct. Expected calibration error (ECE) is the average gap, weighted by how many answers fall in each band. Zero means stated and actual match in every band. For yes/no questions, the top answer’s probability is the larger of p and 1 − p.

Results

“At 90% or more” means the top answer’s probability, not the field called confidence; the next section explains why that matters.

GPT-6 Luna refused 2 banking items and 4 BoolQ items, all ordinary public text, so its figures there are out of 398 and 396 answers. The documentation allows refusals but does not say when they happen.

Banking intents. Answers at 90% or more averaged 97% to 98% stated across the three models, and were right 96.4% (Gemini), 94.3% (Jev), and 88.5% (OpenAI) of the time.

Yes/no passages. Jev’s and Gemini’s confident answers were right 96.5% and 96.0% of the time, close to what they stated. OpenAI’s averaged 99.5% stated and were right 92.9% of the time. Its yes/no probabilities came back with two decimals, and on 287 of its 396 answers the probability was exactly 0 or 1. Seventeen of those 287 were wrong.

Sentiment. This is where the models differed most. Jev and OpenAI stated 90% or more on 95 and 112 sentences and were right on 67.4% and 60.7% of them. Gemini stated between 40% and 60% on 298 of its 400 answers and never reached 90%. Its calibration error on this set, 0.037, was about a fifth of Jev’s and a sixth of OpenAI’s. All three were within one level of the human label on 95.8% to 98.5% of sentences, but exactly right on only 53.8% to 56.5%. Five-level sentiment is hard to get exactly right, and two of the models did not show that in their probabilities.

Which number to put a threshold on

All three APIs return a field called confidence next to the probabilities. They do not define it the same way:

• Google documents it as the top answer’s probability.
• TypeSafe publishes formulas (opens in a new tab): a rescaled top probability, (p_top − 1/n) / (1 − 1/n), for a choice, and a distance-weighted formula for a scale. Its yes/no answers have no confidence.
• OpenAI’s reference (opens in a new tab) lists confidence on choice and scale answers without a formula, and none on yes/no answers.

With 77 options, the rescaling barely moves the number: it differed from the top probability by at most 0.013. On the five-level sentiment set it matters. OpenAI’s confidence matched TypeSafe’s published formula to within 0.005 and differed from its own top probability by up to 0.29. Jev’s differed from its top probability by up to 0.36. A threshold of 0.9 on confidence and a threshold of 0.9 on the top probability select different answers, so pick the field whose definition you know.

Is the decision endpoint really 10 times faster?

OpenAI says the Decisions API “returns typed answers about 10x faster than the Responses API”. I tested that on the same 400 banking messages, sending each to both endpoints and alternating which went first. On the Responses API, with reasoning effort set to none and a strict JSON schema, GPT-6 Luna wrote a label and a probability as JSON, using 22.1 output tokens per answer on average.

That is 10.0 times as long at the median, measured from my machine with the network included. OpenAI’s own openai-processing-ms header gives 13.6 times. The Responses call was slower on every one of the 400 items. Observed accuracy was 81.5% through Responses versus 77.9% through Decisions. This comparison does not establish equivalent accuracy.

Across providers, the median banking request took 118 ms on OpenAI, 262 ms on Jev, and 637 ms on Gemini, with 95th percentiles of 280, 304, and 857 ms. Treat those gaps as rough. Every request came from one machine over the public internet, and the Gemini run started about three hours after the other two, on the Gemini API free tier, where 6 of its 400 banking requests had to be retried.

What it costs

I applied each provider’s published paid price to the usage each response reported. These are list prices times measured usage, not invoices.

Jev billed more input tokens per request (1,686 on average, against 985 for OpenAI and 1,099 for Gemini), but its lower price more than made up for it. Each provider counts tokens with its own tokenizer and question formatting. Gemini’s row is what its usage would cost at the paid price (opens in a new tab); the run itself used the free tier.

Many questions about one input

All three APIs accept many questions about one shared input in a single request; OpenAI’s guide allows between 1 and 400. I asked 200 easy yes/no questions about one 472-word passage (“Does the passage contain the word X?”), five times each, and compared the billed input with 200 separate one-question requests.

In every case the shared passage was billed once, not once per question. The word questions are easy on purpose; this measures billing and latency, not reasoning.

Where they agreed, and one confident disagreement

Any two of the models gave the same answer on 88.4% to 93.0% of banking and BoolQ items, and on 74.8% to 76.8% of sentiment items. Only one item in the whole test had two models both at least 90% sure and still giving different answers.

The BoolQ passage says converted shoot-out penalties “are not considered as goals scored by a player for the purposes of their individual records, or for “golden boot” competitions.” The question was “do penalty shoot out goals count in golden boot”. Gemini answered no at 98.9%, and Jev answered no at 94%. GPT-6 Luna answered yes at 94%. The labelled answer is no.

Before you wire one in

Three things I would do with any of these models:

1. Check the stated probability on your own data. Label a few hundred of your real inputs and build the same band-by-band table before you choose a threshold. These results changed sharply from one task to the next.
2. Threshold a field whose definition you know. The top probability means the same thing on all three APIs. confidence does not.
3. Split multi-factor decisions. TypeSafe’s advice is to ask “each factor as a separate question, then combine the results with logic in your code” (introduction (opens in a new tab)). The many-questions test above shows that is cheap when the questions share an input.

My earlier post on Jev covers how a decision model fits into application code in more detail.

Limits

• BANKING77, BoolQ, and SST-5 have been public for years and may be in these models’ training data. Accuracy on your traffic may be lower.
• I used 400 items per set, one wording per question type, no prompt tuning, and one run per item. Accuracy and calibration depend on the instructions and option descriptions.
• Latency is from one machine and one network path, and includes the network. The Jev and OpenAI runs overlapped in time; the Gemini run came about three hours later, on the free tier.
• OpenAI’s probabilities came back rounded to two decimals, including exact 0 and 1, which limits how finely its calibration can be measured.
• Prices are the published list prices on 14 November 2026.

Sources

• Google: Gemini Decision 1 model page (opens in a new tab), Decisions API guide (opens in a new tab), and Gemini API pricing (opens in a new tab)
• OpenAI: Decisions API guide (opens in a new tab) and Decisions API reference (opens in a new tab)
• TypeSafe: documentation introduction (opens in a new tab), Confidence (opens in a new tab), and Models and pricing (opens in a new tab)
• Datasets: BANKING77 (opens in a new tab) (PolyAI), BoolQ (opens in a new tab) (Google), and SST-5 (opens in a new tab) (SetFit)

Statements attributed to the three providers are from their documentation as I read it on 14 November 2026. The accuracy, calibration, latency, token, and cost figures are my measurement from the same day.