The Benchmark Went Private and the Scores Moved

Artificial Analysis published version 5 of its Intelligence Index on October 27, and the structural change is the headline. For the first time, the majority of the index is built on private data: 60% of it now rests on evaluations with private test sets or held-out answers. Coverage of the new leaderboard reports that the rankings shifted once that happened.

What version 5 is made of

The general category scores non-hallucination separately rather than folding it into accuracy. Measuring whether a model declines to answer when it should is a different question from whether it gets things right, and it is the one that matters most for business use.

The grading standard worth stealing

One of the two new benchmarks is FrontierCode 2.0, Cognition's private set of 132 tasks in real open-source repositories. The grading criterion is what makes it interesting: whether a repository maintainer would merge the agent's change. An attempt that fails any blocking check scores zero, and each task is run three times.

That is a materially harder standard than passing tests, and it is much closer to how a business actually judges work. The question is not whether the output looks plausible or satisfies a checker. It is whether someone responsible for the result would accept it without rework.

Any internal evaluation you run is improved by adopting it. Instead of asking whether the draft looks right, ask whether the person who owns that document would send it as-is. The scoring gets harsher and the number starts meaning something.

Why private tests became necessary

Public benchmarks circulate. They end up in scraped corpora, in discussion threads, in tutorials, and eventually inside training data. A model that encountered the questions during training scores higher without being better at the underlying task, and no deliberate wrongdoing is required at any point.

The effect accumulates quietly, which is why an index drifting toward private sets is a sign of maturity rather than an accusation. Artificial Analysis also released a recalibrated public subset of one benchmark so that scores stay comparable between the public and private versions, which is a reasonable way to keep some transparency while closing the loophole.

What happened to open-weight models

Coverage of the updated leaderboard reports that proprietary models now dominate and that the gap for open-weight models widened once the private sets came in.

Contamination is the obvious explanation and it is not the only one. Results also depend on which models were run, the agentic harness used, and the particular mix of tasks in the new version. A single methodology change producing a ranking shift is a reason to look again rather than a proof of anything.

It does suggest treating public-benchmark parity claims with more caution than they have been getting, which applies to proprietary and open models alike. The broader case for open weights rests on provider choice and portability rather than on leaderboard position, as set out in what open weights actually buy you.

The number buyers should actually read

Artificial Analysis reports that among the ten highest-scoring models, cost per task varies by more than thirteen times: $0.96 at the cheapest and $13.04 at the most expensive, per index task.

Hold that against the fact that these are all top-ten models, so the capability gap between them is far narrower than the cost gap. For most business workloads that ratio is the purchasing decision, and rank position is a distraction from it.

Translating that into your own numbers means knowing your cost per completed unit of work rather than per token, which is the exercise in cash flow forecasting when AI costs are variable. Where that cost sits above the gross margin line, the effect is direct, as in where AI actually moves your gross margin.

Use it as a shortlist

A benchmark with private tests is more trustworthy than one without, and it still cannot tell you which model handles your documents, your vocabulary and your awkward cases. It tells you which three are worth testing.

Building your own small evaluation set from real work with known good answers is the step that produces an answer, and the method is in why AI benchmarks mislead and how to evaluate AI yourself. Running that comparison as a real test with a baseline, rather than an impression, is the discipline in designing a proof of concept that decides.

Re-run it when you change models, because the thing you validated is the version you tested rather than the name on it. Keeping a second provider viable is what makes acting on the result cheap, which is the argument in model hedging to avoid vendor lock-in.

Frequently Asked Questions