Extract Data From PDFs With AI: What Works

Pulling numbers off invoices, statements and forms is one of the few AI use cases where the business case writes itself. It is also one where the demo and the deployment diverge sharply, and a benchmark called ExtractBench shows exactly where. Testing six frontier models on real business documents, the best result was 85.4 percent on one document type and zero on another. Same models, same day.

What the benchmark found

The ExtractBench paper evaluated six frontier models, Gemini 3 Pro and Flash, GPT-5 and GPT-5.2, and Claude Opus 4.5 and Sonnet 4.5, on 35 real business documents spanning credit agreements, research papers, resumes, sports records and SEC filings. Each document was sent through the model's native multimodal API with a zero-shot extraction prompt, and every field in the result was scored individually across 18,516 field evaluations.

Only 51 percent of the 210 extraction attempts produced valid JSON at all. The aggregate field-level pass rate was 4.6 percent, and the best model, Gemini 3 Flash, reached 6.9 percent overall. Those headline numbers are dominated by one domain, so they need unpacking before anyone quotes them.

The SEC filings account for 84 percent of all field evaluations in the study, which is why the aggregate looks catastrophic. Strip that domain out and the pass rate across everything else is 28 percent. Still not good enough to run unattended, and a very different picture from 4.6.

Breadth is the biggest factor, and not the only one

The authors name schema breadth as the dominant predictor of failure, and the 369-field collapse is the clearest evidence: no model produced valid output for any of the seven filings. Long field lists give the model more chances to break the output format, and one malformed field can invalidate an entire response.

The table above also shows breadth is not the whole story. Credit agreements at 13 fields scored 85.4 percent while sports records at 12 fields scored 18.3 percent. Similar breadth, very different results, which points at the documents themselves: layout, table structure, and how directly a value is stated all matter. Test on your documents, because a benchmark score on somebody else's tells you the shape of the problem rather than your number.

Structured output mode made it worse

This is the finding most likely to change what somebody does on Monday. Forcing the model into structured output mode, where the API constrains the response to match a schema, reduced both validity and accuracy compared with plain prompt-based extraction. Valid output fell from 51 percent to 37 percent. The best model's pass rate fell from 6.9 percent to 5.5. GPT-5 on credit agreements dropped from 86.9 percent to 70.0.

The authors suggest that holding a valid grammar state across thousands of tokens competes with the model's capacity to attend to the document content. Whatever the mechanism, the guaranteed-format setting cost real accuracy in their tests. If your pipeline turns it on by default because it feels safer, run both modes over fifty of your own documents and compare before deciding.

How to set the job up

1. Split wide schemas into passes. Ten to twenty fields per request, several requests over the same document, results merged afterwards. It costs more tokens and it is the single change most likely to move your accuracy.

2. Group fields that live together. Header details in one pass, line items in another, totals in a third. Fields that appear in the same region of a document are easier to get right together than a list assembled from across twelve pages.

3. Measure per field, not per document. A document that is 95 percent right is 100 percent wrong if the missing 5 percent is the invoice total. Score each field separately and know which ones fail.

4. Test on your awkward examples. The handwritten note in the margin, the scan someone photographed at an angle, the supplier whose format changed last year. Clean examples tell you nothing you did not already assume.

5. Keep a person on the numbers that matter. Extraction plus review still removes most of the typing, and it keeps someone accountable for figures that end up in your books. That design has held up well in the finance teams we work with, as covered in AI data entry for finance teams.

What this means for the business case

A vendor demonstrating extraction on a clean two-page invoice is showing you the credit agreement row of that table. Your business case should be built on the row your documents actually sit in, and you find that out by running fifty real ones through before signing anything.

The saving is still real. Reading and confirming a figure takes a fraction of the time that finding and typing it does, so even a pipeline that needs every field checked removes most of the work. Price the review step into the business case rather than assuming it away, and the numbers usually still hold up. Our ROI calculator handles that arithmetic, and data quality as the AI bottleneck covers what happens downstream when extracted values go into a system nobody checks.

ExtractBench itself is open, with the dataset, harness and methodology public, and LlamaIndex has since expanded it to 370 documents across 8 business domains and 67 document types with 14 systems evaluated. If document extraction is central to what you are building, it is worth running your own document types through the same harness rather than trusting a leaderboard. The invoice-specific case is covered in GST and HST invoice automation.

Frequently Asked Questions