Your AI Is 'Pretty Accurate'. That's Not a Number — and That's the Problem.
Ask a business running AI how accurate it is, and you typically get one of two answers: a shrug, or an anecdote — 'it handled the Johnson case beautifully'. Neither is a number, and the absence of the number is the actual problem. Accuracy you don't measure, you can't improve — and you certainly can't trust it with customers, prices, or anything signed. The good news is that measuring AI accuracy is neither mysterious nor expensive. It's a discipline borrowed from quality control, it takes days to set up, and it converts every future argument about 'is the AI any good?' into a look at a dashboard. This page is the method, start to finish.
Step 1: Build a test set from reality, not imagination
A test set is 50–100 real inputs your AI actually faces — genuine customer questions, actual documents you process, real orders — each paired with the verified correct output, confirmed by a human expert on your team. The sources are already in your business: the support inbox, last quarter's processed documents, the order archive. What matters is that it's real. Real phrasing contains typos, vagueness, halves of thoughts. Real documents contain the weird formats. Synthetic tests written in a meeting will flatter your system; reality won't. Effort estimate: one to two days of domain-expert time, once. This artifact becomes a permanent company asset — every future change to the AI gets scored against it, which means every future vendor claim gets tested against it too.
Step 2: Score it, and categorize every error
Run the test set through your system and score each output: right, wrong, or partially right. You now possess your baseline — for most companies, the first honest accuracy number they've ever had, and frequently an uncomfortable one. Then the diagnostically crucial step: categorize every error. Was the wrong information retrieved? Was the input misunderstood? Did the system reason incorrectly? Did it format the answer wrongly? Or was the necessary data simply absent upstream? The distribution across these categories is your repair plan in embryo, because each category has a different fix, a different cost, and a different owner. Without categorization, teams 'improve the AI' generically and hope. With it, you fix the two categories that account for 80% of errors and watch the number move. In production, keep the discipline alive with random sampling — five to ten outputs a week, human-checked, fifteen minutes — so drift is caught in days rather than discovered via complaint.
Step 3: The escalation ladder — cheapest fix first
When accuracy falls short, the industry reflex is to reach for expensive technology first. Do the opposite. Most accuracy problems break at the cheap rungs of this ladder.
| Fix | Cost/effort | When it's the right answer |
|---|---|---|
| Better instructions/prompting | Hours | Format errors, misunderstanding of task, missing refusal behavior |
| Retrieval/knowledge base repair | Days | Right information exists but isn't found, or content is stale |
| Task decomposition + validation | Days–weeks | Complex reasoning: split into steps, check each before proceeding |
| Model upgrade | Config change + retest | Reasoning ceiling genuinely reached after the above |
| Fine-tuning | Weeks | Consistent style/judgment needed across thousands of similar cases |
| Custom model training | Rarely justified | Almost never the answer for SME use cases |
What's achievable — and what 'enough' means
For bounded business tasks — answering from your documents, extracting from your standard forms, classifying your request types — accuracy of 95% and above is an engineering target, not a hope. Systems in this class are running in production across ordinary businesses now. The more useful question is what to do about the remaining few percent, and the answer separates mature deployments from fragile ones: you design for it. Low-confidence outputs route to humans with the AI's draft attached; the human resolves in seconds what would otherwise be a customer-facing error. '95% with graceful handover' outperforms '99% with silent failure' in every real-world measure, because trust is won or lost on how a system fails, not on how often. Define per use case what an acceptable number is — an internal summarizer and a customer-facing pricing bot deserve different bars — write it down, and manage to it.
Related pages
- Chatbot gives wrong answers
- AI ROI not visible
- AI readiness assessment coming soon
- Case studies
Frequently asked questions
How do we measure accuracy without technical staff?+
The test-set method needs only a domain expert marking answers right or wrong in a spreadsheet. We set up the harness in the first week of an engagement.
How often should accuracy be measured?+
The full test set after every change to the system; light random sampling (5–10 items weekly) continuously in production, to catch drift.
Is 95% accuracy enough to automate a process?+
Depends what's at stake — but 95% with human review of low-confidence cases is usually the right operating point. Design the handover; 100% doesn't exist.
Why did our AI's accuracy degrade over time?+
Usually drift: the business changed (new products, new phrasing) and the system didn't. Weekly sampling plus periodic test-set refreshes catch this in days.