A Benchmark Asked AI Agents to Run a Shop

Qwen published a benchmark this week testing whether AI agents can handle the work of operating an online business end to end. They did badly. That is a useful counterweight to a year of agent marketing, and it needs three qualifiers before anyone quotes it: the shop was simulated, a benchmark measures tasks somebody chose, and Qwen also sells models, so the evaluator has an interest in where the bar sits.

Read the source before the number

A benchmark published by a company that sells models is not disqualified, but it is not neutral either. Evaluators choose the tasks, and task choice determines the result more than anything else does.

So treat the headline as a direction rather than a measurement. That is the same caution we set out in why AI benchmarks mislead, and it applies whether a result flatters the technology or does not. A finding that confirms your scepticism deserves the same scrutiny as one that does not.

Why it is still worth reading

Because the failure shape matches what businesses actually report, and that agreement is more informative than either source alone.

Agents do well on a bounded task and degrade across long chains. An error at step three does not announce itself; it propagates through steps four to nine, each of which behaves reasonably given the wrong input it received. By the end the output is confidently wrong and the cause is buried several decisions back.

Anyone who has run an agent on real work recognises that description. A benchmark reaching the same conclusion is worth something precisely because it arrived independently.

This is not an argument against agents

Worth being clear, because the easy read is that agents do not work.

Bounded agent work is genuinely valuable. Retrieve information, fill a form, move data between two systems, draft a reply and queue it for review. Those are short jobs where a mistake is visible immediately and costs a minute to fix.

What the benchmark tested is something else entirely: running a business unsupervised across many steps and many decisions. Failing at the second says nothing bad about the first, and conflating them is how businesses talk themselves out of automation that would have worked.

Find the compounding step

Here is the practical response, and it is one question rather than a project.

If a workflow has nine steps, do not ask whether an agent can do all nine. Ask which step is the one where being wrong quietly ruins the following eight. There is almost always exactly one, and everybody who works on the process can name it immediately.

Put a human there. It usually costs a minute and it converts an unbounded failure into a caught one, which is most of the difference between agents being useful and agents becoming a story about a project that did not work. Keeping a log of what the agent actually did, rather than what it reported, is the other half, and that is covered in watching what AI does rather than what it says.

The question for vendors

How many steps does this run before a human sees anything, and what happens when step three is wrong?

A vendor who has thought seriously about reliability answers with specifics: where the checkpoints are, how retries work, how an error surfaces rather than propagating. One who has not will steer back to the demo, which invariably shows a clean run on a well-behaved example.

The demo is not the deployment, and the entire gap between them is what happens when something goes wrong midway. That gap is also where most of the organisational reasons behind AI projects failing tend to live.

Frequently Asked Questions