Free resources / Bug Hunt
4 rules that stop your agent from guessing
Our Ai gave the right answer by luck, and that was the bug. The 4 rules that stopped it, ready to paste.
Bug Hunt · Posted · By Kevin Job, Ghosted Ai
Ai fills gaps with guesses.
When an agent doesn't have an answer, it can produce one that sounds right. On a phone line, a confident wrong answer is worse than "I don't know."
What happened to us
In testing, Isla told a caller their booking time without looking it up. It was right, but only by luck: she'd copied a sample date from her own instructions. A wrong answer gets caught. A lucky one hides the problem until a real customer shows up on the wrong day.
We'd written a sample sentence with a real-looking date. To a model, a good example can look like the answer.
The 4 fixes
- 01Facts only from tools.She may only state a booking day or time if her booking tool returned it on that call.
- 02No real-looking examples.We took the sample dates and times out of her instructions. Use placeholders like [day] at [time] instead.
- 03"I don't have that."If she can't know something, she says so and points to a person. Guessing isn't allowed.
- 04Check every week.A model reviews every call each Monday and flags anything she said that she couldn't have known.
Paste this into your agent's instructions
# Never guess - Only state a booking day or time, a price, or an account detail if one of your tools returned it during this conversation. - If you can't know something, say "I don't have that." Then offer to take a message or point them to a person. - Examples in these instructions are made up. Never repeat an example as a fact.
Check yours
Ask it something it can't know, like an account balance. If it answers confidently, it's guessing.
Test ours
Calls are recorded.
Bring us the work you need help with.
We’ll figure out how Ai can help. The first call is free and takes 30 minutes.