Short answer: Calibrating AI QA against human evaluators works the same way as calibrating two people. Pick a set of real calls, have your best evaluators and the AI score them blind on the same form, compare answers question by question, and fix the form wherever they disagree. Then keep doing a smaller version of it on a schedule so nobody drifts, including the AI.

Every QA leader has lived this. Two evaluators score the same call. One gives it an 88. The other gives it a 74. The agent hears about both and decides QA is a coin flip.

Calibration is how you fix that. And when you add AI to the mix, calibration doesn't go away. It gets more important, because now you have a third "evaluator" scoring a lot more calls than any person could.

Here's the process I'd use.

What Calibration Actually Means

Calibration is getting everyone who scores calls to reach the same answer on the same call, for the same reason.

That last part matters. Two evaluators can both give a call 85 for totally different reasons. That's not calibration. That's luck.

So you calibrate at the question level, not the total score. "Did the agent verify the caller before discussing the account?" Yes or no. Did everyone agree? If not, why not?

Why AI Needs Calibration Too

Some people assume AI scoring is automatically consistent, so there's nothing to calibrate. Half true.

AI applies the same logic each time it scores. That's the upside. But it applies the logic it was given. If a form question is vague, the AI will be consistently interpreting it one way while your team interprets it another way. That's a consistent disagreement, and agents will notice.

So the goal isn't "is the AI right?" The goal is "does the AI score this question the way our best evaluator would, for the same reason?"

If you're still choosing a tool, the same idea applies to testing AI QA on your own calls before you buy.

Step 1: Pick the Right Calls

Calibration falls apart when the calls are too easy. If the calls are all clean passes, everyone agrees and you learn nothing.

Pull a mix:

Pick them before anyone scores anything. Don't let the person who picked a call score it first and "explain" it to the room.

How many? Enough that each question on your form gets tested more than a couple of times. A question that only applies to 3 of your calls hasn't really been calibrated.

Step 2: Score Blind

Everybody scores the same calls on the same form, separately. Your human evaluators don't see each other's answers. They don't see the AI's answers either.

Write down three things for every question:

  1. The answer (yes, no, N/A)
  2. The evidence (what was said, and roughly where in the call)
  3. Any note on why it was a tough call

The evidence is what makes the next step work. Without it, calibration turns into an argument about who's more experienced.

Step 3: Measure Agreement Question by Question

Now line up the answers. For every question, count how often the scorers agreed.

The simplest measure is percent agreement: the number of times two scorers gave the same answer, divided by the number of calls they both scored.

It has a weakness, though. If a question is almost always "yes," two scorers will agree most of the time just by saying yes. Cohen's kappa is a common statistic that adjusts agreement for the agreement you'd expect by chance. This NIH-hosted article explains percent agreement and kappa in plain terms, including its limits.

You don't need a stats degree for this. A spreadsheet with one row per call and one column per scorer per question is plenty.

What the numbers tell you

Here's a made-up example. Say you scored 40 calls and you're looking at three questions:

Example question Times scorers agreed (out of 40) What it probably means
Used the approved greeting 39 Clear question. Leave it alone.
Verified the caller before discussing the account 36 Mostly clear. Look at the 4 misses.
Showed empathy 27 Vague question. Rewrite it.

Every number in that table is an example, not a benchmark. Your team should set your own bar for "good enough" based on how much a wrong answer costs you. Compliance questions deserve a higher bar than soft skills.

Step 4: Fix the Form, Not the People

This is where most calibration sessions go wrong. They spend an hour debating who was right, and nobody changes the form.

When scorers disagree, it's usually one of these:

Write the fix into the question's definition, with one example of a pass and one of a fail. Then re-score the calls that disagreed and see if the gap closes.

When you change a form question, tell the agents too. They're being scored on it.

Step 5: Keep Calibrating After Go-Live

One calibration session at launch isn't enough. People drift. Scripts change. New call types show up. A form that was clear in January can get fuzzy by June.

A simple rhythm:

  1. On a regular schedule, pull a small fresh set of calls and score them blind again, humans and AI.
  2. Track agreement by question over time. One question sliding is a signal. Look at it before agents complain. It belongs on your monthly QA metrics report.
  3. Re-calibrate any time the form changes, a new program launches or the script changes.
  4. Use disputes as calibration data. Every time an agent disputes a score and wins, that's a question worth looking at.

Who Should Be in the Room

Keep it small and keep it the right people:

Don't bring every evaluator to every session. Calibrate your best people first, then use them to calibrate everyone else.

Where OttoQA Fits

OttoQA scores calls against your own QA forms. That makes calibration a practical step, not a sales pitch. You're comparing the AI to your team on the questions you already use.

OttoQA's scoring accuracy is 96%. That's our number. Calibration is how you keep scoring on your form consistent over time, for people and AI alike.

Our 200 to 500 call proof of concept is a good place to start: we score 200 to 500 of your calls against your form.

No seat licenses, no IT, and setup takes about 2 weeks. There's a $1,000 one-time setup fee.

Bottom line... if your evaluators don't agree with each other, they won't agree with the AI either. Fix the form first, then calibrate everyone to it.

FAQ

What is QA calibration in a call center?

It's the process of getting everyone who scores calls to give the same answer on the same call for the same reason. Evaluators score the same calls separately, compare answers question by question, and fix any form questions that cause disagreement.

Do you need to calibrate AI QA scores?

Yes. AI applies the same logic every time, but it scores what your form asks. If a question is vague, the AI and your team can read it differently. Calibrate the AI against your best evaluators before you rely on its scores, and keep checking after go-live.

How do you measure agreement between AI and human evaluators?

Start with percent agreement per question: how often the two gave the same answer. For questions that are almost always "yes," also look at Cohen's kappa, which adjusts for agreement expected by chance. A spreadsheet is enough to do both.

How often should you run QA calibration sessions?

On a regular schedule, plus any time the form, script or program changes. Track agreement by question over time so you can spot drift before agents do.

What should you do when evaluators disagree?

Fix the question, not the person. Split double questions, turn ratings into observable behaviors, define N/A, and add a pass and fail example. Then re-score the calls that disagreed to check the fix worked.

About the author

Tom Laird is the founder of OttoQA and CEO of Expivia, with about 30 years in contact centers. He hosts the Advice from a Call Center Geek podcast. Connect with Tom on LinkedIn.