What this episode is about
This was recorded about a year into building our AI QA tool, first for our own BPO customers and then as a product. A lot of the lessons were about models. A cheaper new model cut our costs in half and, for our use case, cut our accuracy too. So we tested pretty much everything available.
The big one: if a model is right eight or nine times out of ten and then gives you something weird, it isn't really accurate. Consistency is part of accuracy. That's why we built checks into the system prompt, including reiteration, where the model answers a question, holds the answer, answers it again, and we compare.
We also learned we couldn't trust the models with the math. They're okay at math, not great. So the model answers the questions and the scoring is done programmatically on the back end. And we score for intent and outcome rather than keywords.
Calibration: the customer sends scored calls. We line up the human score and the AI score for each question, flag where we think one or the other is more correct, and tune the prompt toward the company's scoring culture. Some companies are strict on certain things and give leeway on others. Scaled ratings like one to ten are harder than yes/no, and we were getting good at those too. Screen recordings weren't ready for production QA yet.
Key takeaways
- Test models on your own calls. Cheaper or newer isn't always more accurate for QA.
- Consistency is part of accuracy. Ask the same question twice and compare.
- Let code, not the language model, calculate the final score.
- Calibrate to the company's scoring culture, not to an abstract right or wrong.
- Scaled (1–10) questions are harder to automate than yes/no, but workable.
In Tom's words
“if something's accurate and it's not consistent is it really accurate”Tom Laird · 04:28
“we can't trust any of the large language models to actually do the scoring”Tom Laird · 08:24
“They're okay at math. They're not great at math”Tom Laird · 08:47
“It's being right or wrong compared to the scoring culture of the company”Tom Laird · 10:59
Transcript excerpt
These excerpts come from an automatic transcript and are lightly edited for readability (filler and obvious transcription errors removed). For exact wording, listen at the timestamp.
[03:38]We tested pretty much all of the models. It's kind of an oxymoron to say, well, if something's accurate and it's not consistent, is it really accurate? The answer is no. We would see, out of 10 questions, it would be right eight or nine, but then one it would just give us something weird. That's where we learned a lot from a prompting standpoint. We have checks within our system prompt to make sure we're being as accurate as possible.
[08:14]We found that we can't trust any of the large language models to actually do the scoring. So we do the scoring programmatically on the back end. We know that's always going to be spot on. We tried to do it with the language models, but they're okay at math. They're not great at math.
[10:44]We would tag it blue if we thought the AI was a little more correct, and red if we thought the human was a little more correct. It's not about being right or wrong. It's being right or wrong compared to the scoring culture of the company. They might be really strict on certain things and have more leeway on others, where the AI initially will be really strict. So we dial that back and match the scoring culture of every organization.