What this episode is about
When we started OttoQA, the question I wanted answered was how much value we could get out of a single evaluation. What I see most often, especially with in-house builds, is a team that gets calls scored at scale and thinks it's finished. Getting calls scored is the first step, not the finish line.
First, push back on 80 to 90 percent accuracy. That isn't good enough, and there are prompting techniques that get you further. Second, ask what you're doing with the data. Saying you scored 30,000 calls isn't an answer. Did coaching change? Did agents get better? If your form is weak, scoring it at scale just scales the problems.
Be honest about the limits too. AI still has trouble with tone of voice, sarcasm needs its own prompting, and wide rating scales like 1 to 10 are hard because humans can't agree on them either. And the AI can't see the agent's screen, so it can't tell you whether the CRM got updated.
The bigger opportunity is treating the call itself as the survey, and building coaching into the platform. That means a few things the agent did well and a few they struggled with on each scored call, and letting agents redo a bad call against an AI version of the same customer.
Key takeaways
- Scoring at scale is step one. Ask every vendor what comes after it.
- Don't accept 80 to 90 percent accuracy as good enough.
- A weak form scored by AI is still a weak form, just at a bigger scale. Fix the form.
- Know the limits: tone of voice, sarcasm, wide rating scales, and anything that happens on the agent's screen.
- Collect coaching and customer data from day one, even before you use it.
In Tom's words
“everybody's acting like scoring calls is the kind of the revolutionary endpoint, but it's it's not”Tom Laird · 03:55
“the problem is you're just scaling garbage”Tom Laird · 07:32
“Are we just checking boxes at scale?”Tom Laird · 07:09
“the call basically is the survey”Tom Laird · 15:24
Transcript excerpt
These excerpts come from an automatic transcript and are lightly edited for readability (filler and obvious transcription errors removed). For exact wording, listen at the timestamp.
[03:34]There's this kind of misconception, and I think it's mostly when people are building the tool internally, right? So you have an internal team that is trying to help the contact center out, they're building out their tool, and they get to a point and they think, we've done it. It's done, voila, right? And everybody's acting like scoring calls is the revolutionary endpoint, but it's not. It's the beginning step, right? You have to be able to score where they are to be able to do all the really cool other things. But most people I see are stopping there.
[06:25]There's this kind of volume versus value track that we're falling in, where people are talking about, hey, we scored 30,000 calls, we scored 50,000 calls. And my question is, well, what did you learn? Maybe there's a dashboard and you can see some lower reps or some higher reps. But again, are you coaching differently? Are the agents getting better? Is performance up? Are we just checking boxes at scale? If you have one of those forms that is not great, then I think the problem is you're just scaling garbage.
[09:13]Human beings are extremely subjective. If you have a form in your contact center where you're saying, hey, rate this on a scale of one to 10, I guarantee you, you're going to have 14 different answers from all the people that are scoring those calls. And the other thing is the AI currently, without some other special tools, can't see into the screen. So if you have a Salesforce instance, I can't see your screen to see if the agent did what they're supposed to in their CRM.
[14:48]No matter what that customer is asking for, what they need, if we're asking questions of it and we know how to listen for it, every single call is a survey. Because those post-call surveys, they're noisy, they have extremely low response rates, they don't scale.