They Felt 12x Faster. Measurement Said About 2x

An independent evaluator spent eleven weeks inside Anthropic with access almost nobody outside the company gets. Staff there, voting on how much the latest model sped them up, averaged about twelvefold. The evaluator estimate of how fast the company models are improving on well-specified tasks, against its own earlier rate, came out near twofold.

What the study was

METR, a non-profit that evaluates AI systems, ran a high-access assessment at Anthropic from 12 August to 29 October 2026 and published on 5 November. It had access to pretraining and reinforcement learning metrics, training compute, internal benchmark scores, the company own speedup analyses, model forecasts and staff interviews.

The question was narrow and worth stating precisely, because it is not the question most headlines will imply. METR was assessing whether AI is accelerating AI research and development, as an input to judging the risk of a sudden jump in capability. It was not measuring how much faster a developer writes code.

Its conclusion: as of October 2026, Anthropic is likely improving the capabilities of its models on well-specified tasks at around twice the rate it did between March 2024 and October 2025. METR is confident that is not explained by scaling up training compute, and attributes a majority of it to Anthropic using AI to do its own research and development, guessing at perhaps 80% in its own hedged wording.

The error bars are the honest part

METR does not present 2x as a finding. It presents it as a central estimate inside a wide range, and says it would not be surprised if the truth were closer to 1.3x or 3x.

There is a further limit on all of it. METR states that Anthropic redacted a majority of the evidence METR considered most informative, which leaves the public relying on METR judgment rather than able to check it. Anthropic co-founder and chief architect Sam McCandlish acknowledged redacting a lot, said it makes the work harder for others to check, and argued the detail was commercially sensitive and would hand rivals a roadmap.

That is a real constraint on how much weight any of these numbers should carry, and it is to METR credit that the redaction is disclosed rather than hidden.

The two numbers answer different questions

One reading is that people overstate their own performance. There is support for that: METR cites earlier work of its own in which developers overestimated the effect of AI on their time by about 40 percentage points. But overstatement alone does not explain the gap, because the two figures were never measuring the same thing.

The honest position is narrower than either reading. Sincerity is not accuracy: staff voting in a Slack poll are not lying, and they are also not measuring. METR gives its own reason the two figures cannot be stacked, noting that individual speedup estimates do not correspond to acceleration rates, and that sustaining a growth rate in outputs generally requires sustained growth in inputs to overcome diminishing returns. That is an argument about inputs and returns rather than about queues.

A separate piece of METR research separates the two questions directly. Its survey of 349 technical workers, run from February to April 2026, found a median self-reported speed improvement of 3x alongside three separate measures of self-reported value improvement with medians between 1.4x and 2x. Ask about speed and about value and the same people give you different multiples.

Both are perceptions rather than verified gains, and the caveats matter. It is a convenience sample, response rates were around 2% among those METR emailed, roughly 70% of participants were paid, and METR notes an earlier study of its own in which developers overestimated the effect of AI on their time by about 40 percentage points. METR own staff reported the lowest value gains of any subgroup.

METR gives a concrete illustration of why speed and value separate. Researchers can use AI to quickly build an interactive dashboard for their data, something that would have taken far longer without it, while that dashboard matters little to the project. The task got dramatically cheaper. The work did not get dramatically better.

One boundary needs stating plainly. METR did not find that organisational friction diluted a task-level gain. It reports not seeing evidence of significant bottlenecks in algorithmic progress, training compute, experiment compute or data taking effect in the next twelve months, while adding that its understanding changed several times and it is not confident in that assessment. The review queues and approval delays below are my own application to enterprise rollouts, not a finding from this study. Those are two different quantities, and treating one as evidence for the other is the mistake almost every AI business case makes.

What does not transfer to your business

Anthropic is close to the best possible case. Its core product is AI, its staff are unusually capable engineers, its work is text and code rather than physical or regulated, and the models they used include internal ones the public cannot buy.

The 2x is also a comparison against Anthropic own prior rate of a very specific kind of progress. It is not a multiplier anyone else can apply to anything. If a vendor quotes this study at you as evidence your business will double its output, they have misread it, and the underlying pattern is the one in the gap between Canadian businesses reporting AI use and those seeing a return.

What does transfer

The measurement discipline, and it is available to any business this quarter.

Stop treating how much faster the team feels as the business case. An evaluator with unusually deep access treated that question as only very loose evidence, and METR own earlier work found developers overestimating AI effect on their time by roughly 40 percentage points. Your version of the same survey will be no better.

Measure what the team finishes instead, counted per team rather than per person, against a baseline taken before the rollout. Features shipped, files closed, quotes issued, tickets resolved. If you have no baseline, the window to take one closes the day the tool arrives, which is the argument for doing the gap analysis first.

Track waiting time alongside it. If the work now gets done in a quarter of the time and then sits for four days awaiting a decision, the throughput gain is small and the fix is not another tool, which is the pattern in finding the real bottleneck.

And separate speed from value explicitly, the way the Anthropic survey did. Producing twice as many proposals is not worth much if the win rate falls, and a proof of concept that only tracks speed will tell you the programme succeeded either way.

Frequently Asked Questions