What this episode is about
Most of you have the same Excel scorecard you've been checking off for 15 years. A lot of people come to us and say, our form isn't great, let's redo it. We tell them not yet. Keep the form you have so you can benchmark against it and see whether the AI calibrates to your team. Then change it.
When you do rebuild it, get rid of the false hustle. Saying the customer's name three times and calling it engagement. A scripted "I'm really sorry to hear that" and calling it empathy. Those questions exist because a human scorer had no other way to grade those things. Now you can ask whether the agent actually listened, controlled the call, and whether the customer felt heard.
The hardest thing for us early on wasn't accuracy. It was consistency, the model scoring the same call 20 times and getting it right 18. We fixed that with prompting: ask the question, ask it again, and if the answers conflict, score it again. If you're building this yourself, plan for that work.
Once it's consistent, you see patterns you never could before. An agent who skips a rebuttal on a big chunk of calls. Someone whose empathy drops on Mondays. You can also add zero-scored questions about why customers are calling or canceling, and use the form as a cheap analytics tool.
Key takeaways
- Keep your current scorecard at the start. It's your calibration benchmark.
- Replace scripted-phrase questions with questions about outcomes: did the agent listen, control the call, make the customer feel heard?
- Test consistency, not just accuracy. Score the same call many times and see if the answer holds.
- Two calls per agent per week can't tell you what to train. Patterns across many calls can.
- Unscored questions about customer behavior turn QA into a voice-of-customer source.
In Tom's words
“everybody just has kind of that excel spreadsheet that they've been checking off for the last 15 years”Tom Laird · 01:36
“we don't want to really reinvent your scorecard you know on day one”Tom Laird · 02:19
“if you're doing two calls per agent per week you have no idea what to train”Tom Laird · 11:19
Transcript excerpt
These excerpts come from an automatic transcript and are lightly edited for readability (filler and obvious transcription errors removed). For exact wording, listen at the timestamp.
[01:14]The first kind of lesson that we've learned with customers and clients when they come to us is a lot of them say my QA scorecard isn't great, or hey, I'd like to update my scorecard, and we tell them not to do that yet, right? So we don't want to really reinvent your scorecard on day one. And the reason is because we want to have a benchmark for calibration, and I think if you're using any type of AI tool, it's really, really important that you have at least something that you've been using to benchmark off of, and then you can kind of see the progression as we change your form.
[03:23]False hustle is kind of the things in the QA world that we had to do because human beings didn't know how to score. We have to say the customer's name three times during the call and call that engagement, right? Having scripted phrases like I'm really sorry to hear that and calling that empathy. All of those kind of things can go away with AI-powered QA. You can define engagement by looking at sentiment, by looking at outcomes of the call. Did they really connect? Did the agent listen? Did they control the call? Did the customer feel heard?
[09:17]The hardest thing that we found with OttoQA was not the accuracy, so to speak. It would be scoring a call 20 times and being correct 18, but two times, right, we had like a crazy number. And again, that was kind of frustrating. You start to see patterns that humans couldn't find before. This agent on a Monday, 62 percent of the time, doesn't show empathy, right? Maybe she just doesn't like Mondays. You can start to see things in the aggregate on your agents and on your teams that you couldn't ever see.
[10:49]If you're doing two calls per agent per week, you have no idea what to train, right? That might have been an outlier, and now you're hammering that agent on something they know they just screwed up one time.