What this episode is about
I do an episode on this about once a year, because the question keeps coming back. When you look at AI QA, there are two ways vendors bill: per seat, or per call scored. The per-seat model pushes everyone toward scoring huge volumes, and running the best large language models at that scale gets expensive, so the trade-off often shows up in accuracy.
The other route is sampling. It's the same law of large numbers that lets pollsters call an election from a sample, or lets drug trials work without testing everyone. Apply it to QA and you can size a sample that gives you a known confidence level and margin of error, then score that sample with the most accurate model you can get.
Sampling only works if it's done right, and most manual QA teams don't do it right. They pick shorter calls, they listen to complaints, they pull extra calls from the agent a supervisor doesn't like. That sample is skewed before anyone scores anything. You also need a minimum number of calls per agent, not just for the center overall.
If you're evaluating a QA tool, ask what model they use, what accuracy they'll stand behind, what happens if your form isn't accurate, and how they set up your questions. And if you have compliance items, you can score just those sections on a larger set of calls.
Key takeaways
- The law of large numbers applies to QA: a correctly sized random sample gives a reliable read with a known margin of error.
- Sample at two levels: the whole center and each individual agent.
- Manual QA samples are often skewed by call length, complaints and supervisor bias.
- Accuracy per score matters more than raw volume of scores.
- Ask vendors which model they use, what accuracy they commit to, and how they handle a weak form.
In Tom's words
“They don't want the 12 minute call.”Tom Laird · 15:21
“So your sample size right there is already screwed up, right?”Tom Laird · 15:40
“Ask what type of large language model they're using.”Tom Laird · 17:44
“What happens if your form isn't accurate?”Tom Laird · 17:51
Transcript excerpt
These excerpts come from an automatic transcript and are lightly edited for readability (filler and obvious transcription errors removed). For exact wording, listen at the timestamp.
[06:11]This math that I'm about to tell you has been backed by about 300 years of proof. There's a statistical principle called the law of large numbers, and it's the same thing that allows pollsters to predict elections. They can get pretty close, within a margin of error, with about a thousand responses. It's the way pharmaceutical companies can prove drugs without testing every human being. We can apply these same principles to your QA.
[15:14]Your QA cherry-picks calls. We all know this. They pick calls based on duration. They don't want the 12 minute call, they'll take that three and a half minute call. They'll say we're just going to listen to customer complaints. Or a supervisor maybe doesn't like an agent, so we're going to listen to more of their calls than anybody else. So your sample size right there is already screwed up.
[17:44]Ask what type of large language model they're using. Ask the level of accuracy that they're going to give you. What happens if your form isn't accurate? Do they change it for you? Do they change questions? How do they set up your questions? All of those things you should be asking, to make sure you're getting the most accurate data, data you can use to make agent changes.