What this episode is about
This one is the 30,000-foot view of how we fully automate QA with AI, and how you could think about building something basic yourself. In our own center we had a team of QA people scoring calls. Today none of them are scoring calls. They're coaching, on the phones, doing the human work.
The mechanics: a system prompt in the back, then your form, with a specific prompt built for each question. We take the recording, chat, email, SMS or ticket, transcribe it, and run each question. The system prompt does things like reiteration, re-asking the model to check its own answer, which is why a call takes a little while to score.
Calibration is the part people underestimate. Before rollout, you score a set of calls yourself. We put your scores next to the AI's question by question, and where they differ, the prompt engineers decide which is right and change the prompt. Sometimes the AI turns out to be more right than the human, and those conversations can get a little dicey.
The other big thing: an auto QA platform is really a tool for asking questions of your data. Not just did the agent do this, but why are customers canceling, did they mention a competitor, why are they calling at all. And keep it simple enough that a frontline supervisor can click a question and see why it was scored that way.
Key takeaways
- Build one prompt per form question, and score to your organization's culture, not a generic standard.
- Self-checking steps in the prompt improve accuracy at the cost of a few seconds per call.
- Calibrate question by question against your own scorers before rollout.
- Supervisors should be able to click any question and see the reasoning.
- The same engine can answer analytics questions: cancel reasons, competitor mentions, call intent.
In Tom's words
“it's scoring it to the culture of the organization”Tom Laird · 05:56
“we're looking for intent of the agent now”Tom Laird · 11:04
“what is an auto QA platform, it's a platform that asks questions of data”Tom Laird · 22:39
“please don't trick us yet”Tom Laird · 28:31
Transcript excerpt
These excerpts come from an automatic transcript and are lightly edited for readability (filler and obvious transcription errors removed). For exact wording, listen at the timestamp.
[05:34]We don't ever say right or wrong. Is it scoring something properly, is it right or is it wrong? No, it's scoring it to the culture of the organization. There are going to be questions that are very black and white. There are going to be questions where you say, we give the agent leeway on that. We can put all of that cultural aspect right into the form, just like a human does.
[10:52]Find out how many times an agent or a customer brought up a pain point, and then did the agent respond in kind? Was there something that raised the tone, raised the sentiment, made the customer laugh, made the customer calm down? So we're looking for intent of the agent now, true intent, and we can look for the outcome of what's happening on the call. We no longer have to just worry about trending keywords.
[27:47]Before we roll this out, we're going to let you score 100, 200 calls on your own. When we're calibrating and building your form, please don't trick us yet. Tell us exactly how everything should go. We'll have a live spreadsheet that shows: here are all your questions, this is how you scored them, this is how the AI scored them. Our prompt engineers look at those and we physically change the prompt so it matches how you score.