What this episode is about
This was a live session from our earliest days, when we were building for centers under 50 seats that score QA in spreadsheets. The goal was to give away enough that you could start calibrating and automating QA yourself with a paid ChatGPT account the same afternoon.
Two prompts are about the form itself. First, paste in your form and ask what it's actually measuring. A lot of times we think a form measures customer experience and it's really scoring security steps and disclosures. Second, ask how to phrase fuzzy criteria, like was the agent empathetic, so a model can score them consistently.
For scoring, you need transcripts first. Then give the model as much information as you'd give a new QA person: what does "handled the opening correctly" actually mean for you? Define what earns a five. Ask for a confidence level, because some transcripts are hard to read. Ask for the reasoning behind each score so your team can trust it.
Outputs matter too: call summary, score, areas of improvement, sentiment and why. If you want to build reporting on top, ask for a strict JSON format. And use the disagreements: when a human and the model score differently, give the model the human's reasoning and see whether it agrees. My goal from the start was letting people use their exact form, not a rewritten one.
Key takeaways
- Ask the model to summarize what your form actually measures. It may surprise you.
- Define each question as you would for a brand-new QA analyst.
- Ask for a confidence level and the reasoning behind each score.
- Specify structured output if you want searchable reporting.
- Use human-vs-AI disagreements to refine prompts, not to pick winners.
In Tom's words
“Because a lot of times we think our form is doing something, but maybe it's not.”Tom Laird · 03:17
“can you provide a confidence level for each of your assessments”Tom Laird · 12:58
“my goal is to have you be able to use your exact form”Tom Laird · 36:04
Transcript excerpt
These excerpts come from an automatic transcript and are lightly edited for readability (filler and obvious transcription errors removed). For exact wording, listen at the timestamp.
[03:04]Let's say you have an Excel form with 15 questions. Throw it in there and ask it to summarize the areas of what it's actually measuring. Because a lot of times we think our form is doing something, but maybe it's not. I've put a form in from a client where we're trying to look at customer experience, but really the only thing it's scoring is security aspects and making sure we read disclosures. It misses the mark on what the form really wants to be.
[12:56]It might not be good. So you want to ask for a confidence score: when evaluating, can you provide a confidence level for each of your assessments, based on the clarity and information available in the transcript? Sometimes the transcript is very difficult to read. Sometimes it's not very high quality.
[36:04]As long as you're getting the same outputs and it doesn't change the question a ton, I think that's fine. But my goal is to have you be able to use your exact form, the one you've used for 10 years and feel comfortable with. You know that a 90 is a 90 and an 85 is an 85, and an auto fail is there.