How accurate are AI contract analysis tools, really?

It depends almost entirely on your documents, not on the tool. On clean, native, standard-form agreements, well-built extraction tools can hit high recall on defined terms. On scanned files, heavily redlined drafts, or non-standard drafting, accuracy drops sharply. That is why any accuracy claim has to be tested on your own paper before you commit.

What is a good sample size for testing?

Twenty documents is enough to see the shape of the problem, provided they are representative: include your most common template, two or three heavily negotiated deals, and at least two scanned files. Twenty documents drawn only from your clean template will tell you the tool works on your clean template, which you already knew.

Should I measure recall or precision?

Recall first. A missed liability cap or renewal date is a silent failure that costs you later, while a false positive just costs a reviewer a few seconds. Set your bar on missed critical fields, then look at precision as a nuisance metric.

How to Test AI Contract Analysis Accuracy Before You Buy

By Marcus Chow · Published: 2026-11-05 · Updated: 2026-11-05 · 8 min read

Every vendor demo looks the same. A clean, ten-page SaaS agreement goes in; a tidy table of parties, dates, and liability caps comes out, usually in under thirty seconds. The audience nods. Somebody asks about security. Nobody asks the only question that matters: what happens when the input is a scanned PDF of a contract that was negotiated over nine rounds of redlines?

This piece is about answering that question with evidence instead of a demo. It assumes you have already shortlisted a tool or two — if you have not, our comparison of AI contract analysis tools covers the landscape and the honest cost breakdown. What follows is how to test whatever you shortlisted, on your own paper, inside a trial window.

Why vendor accuracy numbers do not transfer

Two things drive extraction accuracy, and vendors only control one of them.

The first is the model and the training data behind it. A tool trained on commercial paper from large enterprises will do well on commercial paper from large enterprises, and less well on, say, construction subcontracts or clinical trial agreements. That is not a flaw; it is a coverage boundary that nobody advertises.

The second is your documents. A native, digitally generated agreement with consistent clause numbering is close to a solved problem. The same agreement photocopied, signed, and scanned at 150 DPI is a different problem. So is one where a junior associate left tracked changes in, or where two columns of state-specific riders sit side by side.

Vendors test on the first kind. Your backlog is full of the second. This gap is the entire reason a trial period exists, and it is routinely wasted on clicking around the interface.

Test one: recall on your own twenty

Pull twenty documents from your actual repository. Not the showcase contract your legal team is proud of — twenty representative ones. A workable spread:

Ten from your standard template, in whatever form they usually arrive
Five that were heavily negotiated, with non-standard clauses
Three scanned or photographed, as they come back from counterparties
Two with something structurally awkward: exhibits, schedules, dual-language versions

Now pick five fields per document that actually matter to your workflow. Typically: parties, effective date, renewal or termination date, liability cap, and governing law. For each document and each field, write down the correct value by hand. Twenty documents times five fields gives you one hundred data points. That takes an afternoon, and it is the only ground truth you will have.

Run the documents through the tool. Compare. What you are looking for is recall: of the hundred known-correct values, how many did the tool get right?

Expect the clean ten to score well and the rest to disappoint. The interesting number is not the average. It is the spread. A tool with 95% recall on clean documents and 60% on scans is a different purchase decision than one with 85% overall, evenly distributed.

What counts as a miss. An empty field is an obvious miss. A confidently wrong value is worse — a renewal date that is off by a year looks identical to a correct one in a summary table. Track those separately, because they define how much verification your team still has to do.

Test two: the clauses that break things

Recall on standard fields tells you whether the tool can read. This test tells you whether it can think.

From your negotiated documents, find three or four clauses that a junior lawyer would flag as unusual: a liability cap expressed as a multiple of fees rather than a fixed sum, an auto-renewal that triggers only on written notice, an indemnity carved out for a specific category of claim. These are where extraction turns into interpretation.

Run them. Then look at two things: did the tool extract the clause at all, and did it characterise it correctly? A tool that returns the clause text but labels a fee-multiple cap as "uncapped" is more dangerous than one that returns nothing, because the label is what your team will read.

What you want to see is a tool that knows when it does not know — one that flags a clause as needing review rather than forcing it into the nearest matching bucket.

Test three: format robustness

This one is quick, and it eliminates tools faster than either of the others.

Take one clean agreement. Produce four versions: the original PDF, a scan of a printed copy, a version with heavy tracked changes left in, and a photo taken on a phone at a slight angle. Run all four. Compare the extracted fields.

If a tool handles the original and the tracked-changes version but collapses on the scan and the photo, you now know exactly which intake channel it can serve, and which documents still go to a human unassisted. Many teams find that the scanned back-catalogue — often the largest pile — is precisely what fails.

Setting an acceptance bar that means something

Trials usually end with a vague feeling. Replace that with two numbers.

Metric	What it measures	How to set the bar
Recall on critical fields	Share of the five key fields correctly extracted	Set it against the cost of one missed renewal or liability term
Silent-error rate	Share of fields where a wrong value was returned without a flag	This is the number your review time actually depends on

Recall gets the attention, but the silent-error rate decides whether the tool saves you hours or merely moves them. A tool that misses a field and leaves it blank sends a reviewer to check. A tool that fills it with a plausible wrong number sends nobody, until the renewal passes unnoticed.

Ask the vendor directly how their tool signals low confidence, and whether your team can set the threshold. If the answer is a percentage in a dashboard rather than a flag on the document, the review step stays as large as it was.

What to do with the results

Three outcomes, three decisions.

Good recall, low silent errors: buy, and start by routing the clean-template documents through it. Keep the negotiated and scanned piles on human review for a quarter, then retest.

Good on clean, poor on scans: this is the most common result. It is not a reason to reject the tool. It is a reason to fix the intake channel — request native files from counterparties where you can, and treat OCR-heavy documents as a separate queue. Many teams get more accuracy improvement from asking for native files than from any software change.

Poor on both, or a high silent-error rate: walk. A tool that guesses confidently is worse than no tool, because it quietly removes the check that used to exist.

The part nobody tests

One last thing worth measuring during the trial, because it does not appear on any feature list: how long it takes a reviewer to verify the output. Time three reviews with the tool and three without, on documents of similar complexity. If verification takes almost as long as the original read, the tool is optimising the wrong step.

That number — minutes saved per document, measured rather than claimed — is the one that decides whether a subscription pays for itself. Everything else is a demo.