What this episode is about
This was our 200th episode, a live debate with Chris Mounce from Evaluagent about what's real with AI in QA. I'd been building prompts to score our own clients' forms for about three weeks at that point, and I came in more bullish than Chris.
My argument: the technology to score is there, and the hard part is prompting. A small BPO can sit down and write a great prompt for one client's form: if the customer says cancel, it's a retention call, use this verification, answer these 15 questions, check this disclosure. Doing that at scale, for every customer's form, is a completely different ballgame for a software company.
Chris's point, which I agree with, is that building those prompts forces you to be far more explicit about what you're measuring than most evaluator guidelines ever were. On calibration, I don't care much if it's an 84 versus an 82.5. I care if it's a 90 versus a 54 and an auto fail.
We also talked about security (masking sensitive data before it goes to a model), needing transcripts in the first place, and who owns your contact center data. And we ended in the same place on coaching: sending an agent a score and a snippet and saying fix this is cold. That part needs a human.
Key takeaways
- Scoring was the easy part. Prompting each specific form well is the hard part.
- Writing prompts forces you to define exactly what each question measures.
- Small score differences matter less than big misses like a missed auto fail.
- Mask sensitive data and confirm how your transcripts are handled before sending them to a model.
- Coaching stays human. A score and a snippet isn't coaching.
In Tom's words
“It's very difficult right to prompt”Tom Laird · 07:29
“you care if it's a 90 to a 54 and an auto fail”Tom Laird · 12:18
“here was your call here was your score fix this do this”Tom Laird · 45:17
“AI will never be able to truly show things like you know empathy”Chris Mounce · 44:22
Transcript excerpt
These excerpts come from an automatic transcript and are lightly edited for readability (filler and obvious transcription errors removed). For exact wording, listen at the timestamp.
[07:57]From a small BPO, I can take the time and write an awesome prompt for that customer that takes into account the type of call it is. If somebody says I want to cancel, I know this is a retention call, it's this type of form, I need this kind of verification, then these 15 questions get answered yes or no or one through five, and then I need a disclosure. To scale that is a whole different ballgame.
[12:10]Sometimes we're on, sometimes we're off, but it's 84 to 82.5. When you're in QA you don't care so much about that, that it's that close. You care if it's a 90 to a 54 and an auto fail. That's a big deal.
[44:39]I think from the overall coaching aspect, it's cold. It's one thing to score a call and just send them the score, which I don't even really like to do. Here was your call, here was your score, fix this, do this, and here's even a snippet of the call. I don't know if that would resonate. There's a human element there for sure.