AI Proof of Concept: Designing One That Decides

Barclays said this month that it expects Claude Code adoption to reach half its developers by the end of 2026, and that its internal knowledge assistant already has more than sixteen thousand users. Those are rollout numbers. They exist because somebody, at some point, ran a pilot that ended in a decision rather than in a demo and a meeting that got rescheduled.

Write the decision down first

Before anyone configures a tool, three sentences need to exist on paper. The decision this pilot informs. The measurement that would settle it. The result that would make you stop.

A usable version reads like this: we are deciding whether to move invoice coding from manual entry to an assisted workflow; we will settle it on time per invoice and error rate measured against our current numbers; if the error rate exceeds two percent after correction we stop and keep doing it manually.

The third sentence is the one that gets left out, and leaving it out is what turns a pilot into an open-ended experiment. A stopping condition agreed in advance also protects whoever ran the pilot, because a negative result becomes a finding rather than a personal failure.

Get the baseline before the pilot

Three numbers, measured on the real work before anything changes. How long the task takes today. How often it comes out wrong today. What fixing those errors costs.

Almost nobody has these, which is why so many pilots conclude with it feels faster. Collecting them takes a week of someone noting times and counting mistakes, and that week is the highest-return part of the whole exercise, because without it every result afterwards is an impression.

Measuring the error rate honestly is harder than measuring time, and it is the number that decides whether the project is viable. Count what was wrong, not what felt wrong, and count it on a sample selected at random rather than on whatever was convenient, which is the discipline set out in quality assurance when AI writes the first draft.

Scope to a slice that proves something

Scoping choice	Tempting version	Version that transfers
Which work	The clean, simple cases	A random month of real cases
Which data	A tidied export	Whatever the system actually emits
Who runs it	The enthusiast or the vendor	Someone who does the job daily
How long	Until it looks good	One full cycle of the work
What counts as success	Decided afterwards	Written down beforehand

Hand-prepared data is where most pilots quietly cheat. A tidied export makes the tool look capable and tells you nothing about whether the workflow survives contact with your actual systems, which is usually where the project dies. Running on the messy real feed is the only version of the test that transfers, and if that feed is spread across five systems, that is itself the finding, as in breaking down data silos.

One full cycle, not one quarter

A pilot should cover one complete cycle of the work it touches. Invoice processing needs a month-end. Customer support needs a busy week and a slow one. Hiring needs one real vacancy from posting to offer.

Beyond one cycle, additional time rarely produces new information and reliably produces a pilot nobody will end. An open-ended pilot becomes a permanent arrangement with none of the commitments of a production system: no owner, no budget line, no review, and no decision on the record.

Put it in the hands of someone who does the work

A pilot run entirely by a vendor or by a technical team produces a result the business cannot defend or reproduce. The person who codes the invoices knows which ones are awkward and why, and that knowledge is what separates a real test from a demonstration.

The structure large organisations have landed on makes the same point at scale. Anthropic's Frontier Academy, announced in early October, pairs training with a twelve-week residency in which an engineer leads a real project inside their own employer. That embedded-practitioner shape transfers to a five-person pilot; the budget does not have to.

Give that person explicit permission to report a negative result. Pilots that can only succeed are not tests, and the enthusiasm of whoever championed the tool is the most common reason an unfavourable outcome goes unreported.

Cost the production version, not the pilot

Pilot economics mislead in both directions. Free trial credits understate what running this at volume costs, and the setup effort you absorbed once overstates the ongoing burden.

Work out the cost per unit of work from the pilot, then multiply by real volume at the published rate rather than the promotional one. Usage-based pricing makes that a live variable rather than a line item, which is the modelling problem covered in cash flow forecasting when AI costs are variable, and the framing for the return is in how to measure AI ROI.

End it on a date, in writing

Book the closing meeting when the pilot starts, and bring four things to it. The baseline numbers. The pilot numbers on the same work. What broke, in a list. And the recommendation, which is one of three: production, with a named owner and a budget; another pilot with one specific question changed; or stop, with the reason recorded.

The third outcome is a success for the process even though it feels like a failure. A documented stop prevents the same idea arriving again next year with the same enthusiasm and no memory, and the quality of your records is what determines whether that happens, which is the same point as in root cause analysis.

Businesses with no readiness baseline at all usually get more from a structured look at where the opportunities actually are before piloting anything, which is what the AI readiness assessment and data readiness are for.

Frequently Asked Questions

What makes an AI proof of concept successful?

It ends in a decision that someone acts on. That sounds obvious and it is the thing most pilots skip, because a proof of concept with no written question cannot fail and therefore cannot conclude. Write down the decision the pilot exists to inform, the measurement that would settle it, and the result that would make you stop, before anyone configures anything.

How long should an AI pilot run?

Long enough to cover one full cycle of the work, which is usually two to six weeks rather than a quarter. A pilot on invoice processing needs to see a month-end. A pilot on customer support needs to see a busy week and a quiet one. Running longer than one full cycle rarely produces new information and does reliably produce a pilot nobody is willing to end.

Why do most AI proofs of concept never go into production?

Three reasons recur. The pilot was scoped to the easy slice of the work, so its results do not transfer. Nobody agreed in advance what result would justify spending money, so a good outcome arrives with no mandate attached. Or the pilot ran on hand-prepared data that production will not supply, which means the thing being tested was never the thing you would run.

Who should run an AI pilot?

Someone who does the work, with support from whoever will own the system afterwards. A pilot run entirely by an outside party or by a technical team produces a result that nobody in the business can defend or reproduce. The pattern now being used in large enterprise programmes, where an engineer leads a real project inside their own organisation over several weeks, works for the same reason at a smaller scale.

What should we measure in an AI proof of concept?

The baseline first, which almost nobody has. How long the task takes today, how often it is done wrong today, and what the rework costs. Without those three numbers, any improvement the pilot shows is an impression. Then measure the same three on the pilot, on the same work, with the error rate counted rather than estimated.