What this episode is about
This was a special episode after some LinkedIn posts about how we'd gotten to a proof of concept for automating QA in our own contact center, using each client's actual scoring forms. I said it plainly: I'm not an expert on this, we're not a software company, and it was a work in progress.
The approach: we already had transcripts from our analytics platform. We wrote prompts that go through the QA form like an analyst would. Yes is five points, no is zero, scored against the forms clients had benchmarked for years, so a 90 out of 100 still means what it always meant.
Two things you need. Access to the data the question depends on, and people who know the form, the QA and floor managers, to write robust prompts with all the if-thens. Small wording changes in a prompt can change the output a lot. That's why I thought vendors would struggle unless they sat down with each customer and worked through the prompts.
Where does that leave QA teams? Scoring is a job AI can handle, like password resets. Give it good data and it gives you good data. The human work moves to coaching and to deciding what to do with all that scored data. My advice: build a prompt, test it in the desktop app, have your QA team score the same calls, and compare. Watch security, too: mask PII and think through HIPAA and PCI.
Key takeaways
- Keep the scorecards you've already benchmarked. Automate them rather than replacing them.
- You need transcripts and access to whatever data each question depends on.
- Prompt wording matters. Expect real trial and error.
- Compare AI scores to your own team's scores on the same calls before trusting them.
- Mask customer data and think through HIPAA and PCI before you start.
In Tom's words
“it's really easy to do if you can figure out the prompt”Tom Laird · 05:06
“This is a work in progress. This is a proof of concept. This is not a finished product.”Tom Laird · 06:31
“give it good data in and it's going to give you good data out”Tom Laird · 25:36
Transcript excerpt
These excerpts come from an automatic transcript and are lightly edited for readability (filler and obvious transcription errors removed). For exact wording, listen at the timestamp.
[05:06]It's really easy to do if you can figure out the prompt. It's very difficult to do if you can't, and you don't understand the prompt. So what we've done is take all of the chat transcripts, and recently all of the transcripts in our interaction analytics platform. If you don't have transcripts, this becomes more difficult, and it becomes extremely expensive if you have to go get your voice calls transcribed.
[07:23]We're talking about scoring off of actual scorecards that clients have used for the last ten years, that you benchmarked, that you know work well. You know that 90 out of 100 is going to correlate to a pretty good customer experience. Then you can write a prompt to say, look at the transcript and answer these questions.
[21:28]Take that first step. Develop your prompt, practice it in the desktop app. Find a prompt that works, then go to your QA team and have them score the same call. The example I gave the other day: the AI scored it at an 85 and we scored it at an 84.5. I'm okay with that. And if I have five people in our QA department all score a call, I guarantee we're going to have similarities, but it's not always going to be perfect.