What this episode is about
This was a live session about eight months into our work automating QA for smaller contact centers. Lots of smaller centers still score calls on Excel or Google Sheets, and part of the idea was that you can get a long way with a large language model if you understand how it thinks.
Some practical things we learned. Early on, with a long form, the model wouldn't always return every question. Specific wording in the prompt fixed that. For accuracy and speed, tell it where to look. If the greeting has to happen at the start of the call, say look in the first few lines of the transcript.
Binary questions, did the agent do this or that, any model can handle. The real test for anybody doing this is the judgment questions: empathy, or whether the agent handled something appropriately across several parts of the call. We were testing every prompt against our own BPO's human-scored calls, all day long.
We also talked about scoring things you can't directly see, like dead air while an agent hunts for information, and asking for a rationale whenever the model scores a no or N/A so a human can check it. A listener described having the model flag calls that are too complex so a quality analyst reviews them, which is a good idea.
Key takeaways
- Start with a system prompt that sets the role, the scoring rules and the output format.
- Write one prompt per form question and tell the model where in the transcript to look.
- Judgment questions like empathy are the real test of an AI QA setup.
- Require a rationale for every no or N/A so humans can review it.
- Test every prompt against human-scored calls before trusting it.
In Tom's words
“is to tell chat GBT where to look for certain things in a transcript”Tom Laird · 15:28
“how do they handle the empathy questions”Tom Laird · 16:25
“as long as something is in the transcript, we've been able to figure out really”Tom Laird · 22:50
Transcript excerpt
These excerpts come from an automatic transcript and are lightly edited for readability (filler and obvious transcription errors removed). For exact wording, listen at the timestamp.
[14:57]The other thing we found, for accuracy and for speed, is to tell the model where to look for certain things in a transcript. So for the greeting, for this specific client, the agent must say thank you for calling customer service, and we tell it to look for that in the first five lines of the transcript.
[16:16]The black and white binary questions, did the agent do this or that, everybody knows the model can handle that. But the question for any of these companies trying to do this is how do they handle the empathy questions. Did the agent do something appropriately throughout the call, where it's not black and white but takes a thought process across multiple sections of the call. I think it can be done.
[28:20]When it comes to the actual summaries, we ask for what we're calling the rationale. If it scores a yes and gives a full point, we don't say anything. But if it scores a no or an N/A, there's a little question mark next to the question where we can look, and it tells us why it scored it as a no.