Marcus got an 84 last month. One of his misses was address verification, on a single call. He swears he did it. He wants the points back.
Dana, his QA manager, pulls the recording. She scrubs back and forth. He confirms the street but not the zip. Or maybe the customer read it off first. Twenty minutes later she isn't sure either.
You've seen this fight. If you've worked in a contact center for more than a week, you've probably been in it.
I've been Dana. I've built QA programs for 30 years. I've sat across from a hundred Marcuses and defended that number in rooms full of angry agents. I was wrong about what it was for.
Here's what took me way too long to admit. Marcus is right to fight.
Not because he verified the address. Maybe he didn't. He's right because the number we handed him can't carry the weight we put on it. It decides his bonus. His schedule pick. Whether he gets promoted or put on a plan.
And it came from eight calls.
Your QA score doesn't measure your agents. It measures which calls you happened to pull.
The dispute isn't the problem. The dispute is a symptom. The problem is that our whole industry runs on a number that was never built to do what we ask it to do.
This is the long version. Why the single QA score has to die. And what should take its place.
To understand why, you have to go back to where it came from.
The QA score was a workaround for a problem we no longer have
When I started, quality monitoring meant a supervisor with a headset. You plugged in next to an agent or listened in from your desk. Later we got recordings. That helped. It didn't change the basic math.
A human can listen to maybe 30 or 40 calls a day if they do nothing else. A 50 seat center takes thousands. So we sampled. We had to.
Then we needed a way to make the sample useful. Different analysts heard calls differently. Leadership wanted something they could put on a slide. So we built a form. Yes or no questions. Points for each. Add them up.
The score was a compression tool. We took an eleven minute conversation and squeezed it down to one number. Because one number was all anybody had time for.
That was smart. It was the right answer to scarcity.
The scarcity is gone. AI can listen to every call you take, tonight, for less than you pay one analyst. But we kept the compression.
We kept the ration card after the shortage ended.
Think about it this way. We have every word of every conversation now. And we're still boiling it down to an 84.
I coach basketball. For a long time players were judged on points per game. Now we look at deep analytics because the technology is here. Points weren't wrong. They just weren't the full story.
The QA score isn't wrong either. It just isn't the story. And it was never as accurate as we pretended.
Same dispute(ish). BUT...Every call.
Back to Marcus and his disputed call. Now picture the same dispute with a different scorecard. Not eight calls. All 400 of Marcus's calls from the last 30 days, with every question answered on every one.
Look at question 2.1. Marcus verified the address on about 70 percent of his calls. He skipped it on roughly one in four. The call he's disputing doesn't matter anymore. The month already answered the question.
And the next row shows he skips the callback number about as often. That's not a dispute. That's a coaching conversation.
Nobody scrubs a recording for twenty minutes. Nobody has to win.
So if a month of calls settles the argument, what are eight calls really telling us?
The math nobody checks
Here's what the old way would have shown.
Say Marcus's true rate is three out of four, close to what his month showed. Now score just eight of his calls. What will the scorecard say?
It could say anywhere from 38 percent to 100 percent. Same Marcus. Same habits. Same month. The only thing that changed is which eight calls Dana happened to pull.
Read that chart again. On eight calls, Marcus could look like a superstar or a disaster. And he wouldn't have changed a thing.
At 30 calls the range is still 60 to 90. At 400 calls it tightens to 71 to 79. That's a number you can actually coach to.
And that chart is the best case. It assumes the eight calls were picked at random and scored perfectly. In real life, neither is true. There are two more kinds of noise.
Which calls get picked
We say the sample is random. It almost never is. Analysts pull calls that are easy to find. Calls that aren't 40 minutes long. Calls a supervisor flagged. Every one of those choices bends the score.
Who does the scoring
Every QA manager has run a calibration session. Three good analysts score the same call. They come back with three different numbers. Then everybody spends an hour arguing about whether "I can help with that" counts as ownership.
Put those together. A small sample. A sample that isn't random. Scored by people who don't fully agree with each other.
Then we tie bonuses to it. We put agents on plans because of it. We send it to the VP as proof the program works.
That's not a measurement. That's a guess with a decimal point.
The call mix trap: what the 84 is hiding
This one should bother you more than the sample size.
An 84 sounds like one fact about Marcus. It isn't. It's six different jobs blended into one number.
Here's his month, split by call type.
On password resets, order status and address changes, he's one of the best on the floor. On cancellations, billing disputes and refunds, he's struggling. Billing disputes are the second biggest thing he does, 70 calls, and he's scoring 64 on them. The 84 hides all of it.
Now think about what that number does next month. If billing sends him more disputes, his 84 drops, even if he gets better at every call type. If he gets moved to an easy queue, it climbs, even if he learns nothing. The score moves with the calls he's given, not just with how well he handles them.
Statisticians have a name for the worst version of this. They call it Simpson's paradox. An agent improves on every call type and his overall score still goes down. Contact centers call it "Marcus had a bad month."
It works the other way too. Put a mediocre agent on an easy queue and they look like a star. Route your toughest calls to your best people, and the score quietly punishes them for it.
An average of different things isn't a fact. It's a recipe. Change the ingredients and you change the number. Even if nobody cooked any differently.
What the "old" scoring does to people
Bad math would be bad enough. But a number tied to money doesn't just measure behavior. It changes it.
There's an old rule called Goodhart's law. When a measure becomes a target, it stops being a good measure. QA might be the purest example of it I've ever seen.
It teaches agents to perform for the form
If the form checks for the customer's name, agents say the name three times. If it checks for an empathy statement, they say "I'm really sorry to hear that" in the same flat voice on every call. Box checked. Customer unmoved.
I wrote a book about false hustle. Looking busy instead of getting results. The QA form is one of the worst offenders. We trained a whole industry to hit checkboxes instead of helping people.
It turns QA into a courtroom
When eight calls decide your bonus, every point is worth fighting for. So agents fight. QA teams build dispute processes. Supervisors sit in the middle playing judge.
Nobody learns anything in a courtroom. They just try to win.
It wastes your best analysts
A good QA analyst notices a lot on every call. The moment the customer calmed down. The system screen that froze. The policy that made no sense. The customer who almost canceled and didn't.
Then they write down an 84. Everything else they noticed goes in the trash. We pay smart people to learn things and then throw away what they learned.
It makes coaching vague
A score tells you that something went wrong. It doesn't tell you when, or what it sounded like, or what to do instead.
That's how you get coaching notes like "be more empathetic." That's the whole note. The agent nods. Nothing changes. How could it?
It gives leaders false comfort
The dashboard says 90. It's green. Everybody relaxes.
Meanwhile customers are calling back about the same issue. A new fee is making people angry. Cancellations are creeping up. None of that shows up in a 90. The score says the agents followed the form. It says nothing about whether the business is okay.
The half of the call nobody scores
Here's the thing that should keep every CX leader up at night.
The customer talks for about half of every call. And every QA form ever written is about the agent.
Your contact center is the biggest focus group your company runs. Thousands of customers a day tell you exactly what's broken. Why they're mad. What confused them. Who they're thinking about switching to. They tell you for free.
And we point the form the other way.
Marketing walks into Dana's office and asks what customers are upset about this month. Dana has 500 scorecards about agents. She has nothing about customers.
Now imagine a second form. Same calls. Every question is about the customer.
Across 25,000 calls, 22 percent mentioned the new monthly fee. Of the customers who wanted to cancel, more than half said price was the reason. More than a third were calling about something they'd already called about. Almost half had tried the website or the app before they picked up the phone.
That's not a QA report. That's a board meeting.
The fee is a pricing problem. The repeat calls are a resolution problem. The website number is a product problem. Not one of them is an agent problem. A score aimed at the agent would never have found any of it.
And it's not just what customers say. It's how they feel. About 5 percent of customers answer your survey. You hear from the thrilled and the furious. Everybody in the middle hung up and went on with their day. When you read sentiment on the calls themselves, you hear from everybody.
What rises from the ashes: the new rules of QA
The score doesn't disappear. It gets demoted.
The scorecard stops being a report card. It becomes a tool to collect data. You still answer the questions on every call you score. The number still falls out at the end. It's just a byproduct now. The point is everything you learned on the way to it.
Build it yourself or buy it. Doesn't matter. These are the rules.
Rule 1: Patterns over points
When you score two calls a week, one call is worth fighting over. When you score hundreds, it isn't.
The question stops being "did Marcus verify the address on Tuesday." It becomes "does Marcus verify addresses." Over a month, the data answers that without a meeting. Single calls go back to being what they should be. Exceptions.
Disputes don't get resolved. They just stop happening.
Rule 2: Questions over totals
Stop reporting the total. Report each question. What percent of the time does this agent verify the callback number? Thank the customer? Avoid talking over them?
A total tells you something is off. A question tells you what. And split every one of them by call type. Then the trap from earlier can't hide anything.
Rule 3: Moments over phrases
Stop scoring whether the agent said a magic sentence. Start finding the moment the call turned.
On a good call you can watch it happen. Minute two, the customer bottoms out. Minute four, something changes. By minute nine she's happy. Go find what the agent said at minute four. In one call I watched, it was five words. "How are you holding up?"
Now coaching is easy. That thing you said right there. Do more of that. That beats "be more empathetic" every single time.
Rule 4: Process over people
This is the rule that changes the most.
When you only score a few calls, every miss looks like an agent problem. When you score enough calls, you can see the whole floor at once. And you start noticing something. Some questions fail everywhere.
Look at phone number verification on that form. It failed on more than half of all calls. That isn't sixty agents who need coaching. That's one process problem. Maybe the field is buried on the screen. Maybe the script skips it. Maybe nobody ever trained it.
You don't fix that with sixty one on ones. You fix the screen.
Score enough calls and the form stops being a grade. It becomes a map of where your operation is broken.
Rule 5: Counts over feelings
Most one on ones start with "how do you think it's going?" Because nobody had forty minutes to prep.
Now the prep can be done before the meeting. "Missed the callback number on 9 of 12 calls." That's not a feeling about the agent. It's a count. Agents can argue with an opinion. It's a lot harder to argue with 9 of 12.
And count the good stuff too. "Kept a calm, patient tone on 11 of 12 calls." Start there. The meeting shouldn't open with a list of faults.
Rule 6: Reps over notes
When Marcus blows a call, he gets a coaching note and a hope. Then he waits for the next angry caller and tries to remember what Dana said.
Nobody gets better from a note. I've coached basketball for 16 years. Players get better from reps.
Now you can give agents reps. Rebuild the customer from the real call. Same complaint. Same attitude. Let the agent take it again. And again. Practice the ugly calls on purpose, before the real ones show up.
Rule 7: Predicted outcomes over surveys
If 5 percent of customers answer the survey, your CSAT number is a guess about the other 95.
Predict it on every call instead. Then ask the question that should scare you a little. Which calls score high on your QA form and low on predicted CSAT?
Those are the calls where your agents did everything you asked and the customer still left unhappy. That's not an agent problem. That's your form measuring the wrong things.
The seven rules
- Patterns over points
- Questions over totals
- Moments over phrases
- Process over people
- Counts over feelings
- Reps over notes
- Predicted outcomes over surveys
The objections, answered honestly
Every time I make this argument, I hear the same pushback. Most of it is fair. Here's how I answer it.
"Our bonuses are tied to the score."
Then you're paying people based on eight calls somebody else picked. That should bother you more than changing the plan.
Tie pay to question level results over enough calls to mean something. Split by call type so nobody gets punished for the queue they were put on. Run the old way and the new way side by side for a quarter. Show agents both. I'd bet they pick the one built on more calls.
"Compliance requires a score."
Compliance is the best reason to score more calls, not fewer.
A compliance miss shouldn't be averaged into an 84. It should be found, on the call where it happened, every time. Keep your compliance questions. Make them pass or fail. Just stop burying them inside a total where a great greeting can cover up a missed disclosure.
"AI gets it wrong sometimes."
It does. So do people. Go sit in your next calibration session and watch three humans disagree about the same call.
The difference is that a machine tends to be wrong the same way over and over. That means you can find it and fix it. Demand the reasoning behind every single answer. If a tool can't show you why it scored a question the way it did, don't trust it.
You can calibrate AI to be more accurate and consistent than a human.
And ask what model is doing the scoring. Some vendors use the cheapest one they can find and keep the difference. Full disclosure, I sell this stuff. Hold me to the same questions.
"So we have to score 100 percent of our calls?"
No. And this is where some vendors are going to get mad at me.
Two percent was never enough. You saw the chart. But 100 percent isn't automatically the answer either. A lot of "full coverage" is sold per seat, run on a cheap model, and called complete. You get every call scored and a lot of them scored wrong. That's the seat license fallacy: paying for coverage instead of accuracy.
You need enough calls to see the pattern. That's a math question, not a sales question. Going from 8 calls to a few hundred per agent changes everything. Going from a few hundred to every single call usually changes your invoice more than your answer.
The exception is compliance. If a missed disclosure is a legal risk, you want to catch every one.
A few hundred calls scored right beats every call scored wrong.
"Our agents will hate being watched on every call."
What agents hate is being judged on eight calls they didn't pick by a person they can't argue with.
Show them the counts. Show them the praise side. Give them reps instead of notes. Being seen accurately is a lot less scary than being guessed at.
Marcus, one more time
Let's go back to Marcus. Same Marcus. New way.
He doesn't get an 84. He gets this.
You verify the address on about three out of four calls. Let's get that up. You're one of the best on the floor at password resets. Billing disputes are where you're struggling, and it's mostly two questions. Here's a call from Thursday where you turned an angry customer around at minute four. Whatever you said there, do more of it. Want to practice a dispute call before lunch?
There's nothing to dispute. There's just something to work on.
That's the whole point. The score was never supposed to be the thing. It was the best we could do when we could only hear a few calls. We can hear all of them now.
The score was the output. Now it's just a byproduct.
Let it go. And go find out what your customers have been trying to tell you.
Pull your last month of QA scores. Next to each one, write how many calls it came from. Then decide if you still trust it.
If nobody on your team ever disputed a score again, what would you do with all that time? Tell me in the comments. I read every one.
I build this stuff at OttoQA. You can build it yourself too. Either way, stop fighting over the 84.
Interested in hearing more about this...check us out at ottoqa.com!
Originally published on LinkedIn: The QA Score Is Dead. Here's What Should Rise From Its Ashes.