What this episode is about
I've been saying this for a while and I wanted to put it on the record. The 80, the 72, the 100 your analyst gives a call doesn't mean much. That score was how our human brains had to handle QA when we did it by hand. We listened, rewound, gave points, and the score was the output.
When you're looking at a big pile of calls for every agent, the score drops from the most important thing to one of the least important. Disputes go away, because nobody is fighting over one call anymore. If Marcus didn't verify the address on a quarter of his calls last week, that's the conversation. Arguing over whether he did it on one call misses the point.
The same goes for empathy. Most forms still score whether the agent said "I'm really sorry to hear that." That's scoring a phrase. With AI you can ask a better question: where did the customer have a pain point, and did the agent's response raise or lower the sentiment? Then roll that up. What share of an agent's calls start negative and end positive? What share go the other way? Look at it by call type too, because an agent with a 90 overall can be in the 70s on refunds and billing disputes.
This isn't a sales pitch. You can do some of this yourself with an LLM. But if you're shopping for an AI QA tool, these are the questions I'd be asking the vendor.
Key takeaways
- With a large sample per agent, one call is just one data point. Coach to the percentage of time a behavior happens.
- Score outcomes, not phrases. Did the agent move the customer's sentiment after a pain point?
- Sentiment timelines on single calls are mostly noise. Track how often each agent turns negative calls positive, and how often it goes the other way.
- Break scores out by call type. An overall 90 can hide a 78 on refund requests.
- Summaries aren't there for you to read one by one. They're data you roll up into a coaching plan.
In Tom's words
“One call is just an insignificant data point.”Tom Laird · 05:13
“we're scoring words and phrases, we're scoring what they've said.”Tom Laird · 08:50
“At what point the call went negative, I think is a meaningless stat.”Tom Laird · 10:56
“Now you're summarizing 300 calls for an agent.”Tom Laird · 12:37
Transcript excerpt
These excerpts come from an automatic transcript and are lightly edited for readability (filler and obvious transcription errors removed). For exact wording, listen at the timestamp.
[04:33]Disputes, right? This is the number one thing that now kind of goes away. I know so many of you out there have, because I get this question every time: hey, can the agent actually dispute the call for OttoQA? And again, disputes were important when you were scoring and maybe incentivizing your agent off of three calls per agent per week, maybe nine calls a month, but that's done now. Now we don't have to dispute things because we have mass quantities of data, and with the mass quantities of data, we can look at the bigger picture. So one call doesn't mean anything anymore. One call is just an insignificant data point.
[05:38]So again, if you had an agent, you scored one call, and that agent said, hey, you know what? I did verify that customer's address. And maybe you have a conversation on that. But now when we look at Marcus and his 200 calls from last week, we can see that almost 25% of the time, he did not do that behavior. So disputing is done. And disputing doesn't make any sense because you're arguing and fighting over little scraps of QA, of things that we thought were important.
[08:01]How many of you guys have a question on your form that says, was the agent empathetic, right? Did they use these empathetic terms like I'm really sorry to hear that. We coach our agent to quote unquote be more empathetic. But the problem is when we're doing human scored QA, we're scoring words and phrases, we're scoring what they've said. And even if an agent says something like, Mrs. Smith, yeah, I'm really sorry to hear that, because they've said it 432 times, right? We still give them points.
[10:56]At what point the call went negative, I think is a meaningless stat. But the overall macro of which of your agents are actually turning negative into positive, and then which ones are turning positive into negative? Those are important data points, things that you can very easily pick up with AI.
[12:11]We don't have to prep for a coaching session. All that information is already there. You're not summarizing three calls. Now you're summarizing 300 calls for an agent. And now you're coming up with the four or five main points that they struggled with last week, and the three or four main points that they did awesome with.