Judge AI on What It Finds, Not How It Talks

Perplexity reported that its Search as Code offering beat both OpenAI and Anthropic on four of five search benchmarks. Set aside the scoreboard for a second, because the interesting part is what it implies. The companies with the most impressive conversational models are not automatically the best at finding things. Those are separate skills. And for most of what businesses actually ask AI to do, the second one matters far more than the first.

Most business questions are lookups

Notice what your team actually asks AI. Very little of it is "write me something beautiful." Most of it is: what are the current rules for this, what did that supplier announce, what are competitors charging, what changed in this regulation, what is the standard practice here. Every one of those is a lookup. The model’s prose style contributes nothing; its ability to find current, correct, relevant sources contributes everything. If you are buying AI mainly for questions like these, you are buying a research tool that happens to be conversational, and you should evaluate it that way.

Why polish makes bad sourcing worse

Here is the uncomfortable interaction between the two skills. A beautifully written answer built on the wrong three web pages is more dangerous than a clumsy answer built on the right ones, because the fluency is what convinces you not to check. Strong writing on weak retrieval is the worst combination available, and it is unfortunately a common one. This is a cousin of the problem we covered in AI inventing its sources, except here the sources are real, they are simply the wrong ones, which makes it much harder to spot.

The warning signs

Weak retrieval has a recognizable signature once you know to look for it.

The one-hour test

Do not evaluate this from a leaderboard. Take five real questions from your own work where you already know the correct answer and where the good sources are, and deliberately include one where the obvious top web result is out of date or misleading, because that is the question that separates tools. Run all five through each candidate and check three things: did it find the right sources, was the information current, and does the answer actually follow from what it cited? An hour of that tells you more than any benchmark, for the same reason we argue benchmarks mislead: they measure someone else’s questions.

The Canadian trap

This one catches businesses here constantly, so make it one of your five test questions. Tools trained and tuned largely on American sources will confidently return US tax rules, US regulations, US pricing, and US employment law when you ask a general question, with no signal that the answer does not apply to you. Ask about something Canadian-specific where you know the right answer, and see whether the tool gets there or quietly hands you the American version. And regardless of which tool wins, keep the standing rule: anything consequential gets its sources checked by a person before it becomes a decision or reaches a client.

Split your evaluation

The practical upshot is to stop looking for one best AI and start matching tools to jobs. Judge the tools your team uses for drafting and thinking on reasoning and writing. Judge anything doing lookups on retrieval, using your own questions. They may well be different products, and that is fine, a slightly awkward tool that reliably finds the truth beats an elegant one that reliably sounds right. It is the same principle as making sure your data is ready for AI: the quality of what goes in decides the value of what comes out.

Frequently Asked Questions