The Benchmark That Rewards Admitting Uncertainty

Two models read the same 1,500 documents. One got all fifteen answers right on 36% of them. The other managed 24%. On the metric Artificial Analysis now leads with, they tie, because the less accurate one knew which of its answers to doubt.

What was published

On 8 November 2026 Artificial Analysis released version 2 of its Decision Index, the benchmark set it uses for models that pick from a fixed list of options and return a probability for each. The change that matters is the headline metric, which it calls the Decision Score.

Each question scores from 0 to 100. A right answer scores above 50, a wrong answer below 50, and the model confidence in its chosen answer moves the score within whichever half it landed in. Confidence amplifies in both directions, so the worst possible outcome is not being wrong, it is being wrong and certain.

The reasoning behind it is a description of how these systems are actually deployed. An answer gets acted on when confidence clears a threshold and is passed to a person or a stronger model when it does not. The Decision Score averages the payoff across every possible threshold, so it does not depend on guessing where you would set yours.

Why a weaker model can tie

The example in the announcement is the clearest statement of the idea. On AA-Paperclip, a new private benchmark of 1,500 documents including resumes, receipts, invoices and research papers, one model passes 36% of documents and another passes 24% on raw accuracy.

On the Decision Score they come out level, because the second is far better calibrated, with a Brier score of 0.15 against 0.27. A lower Brier score means the probabilities a model attaches to its answers track how often those answers are actually right.

Translate that into a workflow and the logic is obvious. If a model tells you honestly that it is unsure about a document, that document goes to a person and gets handled correctly. If a model is confidently wrong, nobody checks, and the error goes straight into whatever the output feeds.

Raw accuracy collapses those four outcomes into two. That is the gap this metric is trying to close, and a business feels it directly: an error only reaches a customer when the model was wrong and sure enough that nobody looked.

The reworded-question test

A second new component deserves attention on its own. AA-Symmetry asks whether a model gives the same answer when the same question is worded differently, running 120 questions through variations including reordered options, swapped labels and negated yes-or-no phrasing.

The reported consistency spreads widely across models. At the top end a model holds its answer across 99% of rephrasings. Further down, consistency falls into the sixties, which means roughly a third of the time a different wording produces a different answer to the same question.

For anyone running a classification workflow that is the more alarming number of the two. It means the output depends partly on how your prompt happens to be phrased, and that a prompt tweak made for readability can silently change results, which is the discipline behind keeping your own fixed test set.

The all-or-nothing scoring

AA-Paperclip scores each document as a single decision. Five decisions and ten facts to extract per document, fifteen answers in total, and the document counts as correct only if every one of them is right.

That is a harsh rule and it matches how extraction actually gets used. A record with fourteen correct fields and one wrong one is not 93% useful, it is a record somebody has to check, and if nobody checks it then it is a wrong record in your system.

Partial credit is a benchmark convenience. It is worth remembering when a vendor quotes you a field-level accuracy figure, because the number you care about is how often the whole record is right.

What to do with this

Treat it as a reason to ask a different question when evaluating a tool. Not how accurate is it, but can it tell me which answers to trust, and what happens to the ones it is unsure about.

A workflow with no route for uncertain cases will produce the worst outcome in the table, because every answer gets treated as equally good. Designing the uncertain path first is the same principle as the cause-unknown category in a collections process, and it is what makes automation safe rather than fast.

One caveat on the index itself. The methodology is published in detail, but AA-Paperclip is private and several other components are private adaptations, so the scores cannot be reproduced from outside. Keeping a benchmark private is what stops models being trained against it, and the cost of that choice is that you rely on the evaluator rather than check the work. That trade is reasonable and it is still a trade, which is the general caution in how benchmarks mislead.

Frequently Asked Questions