All posts
AI agents10 min readUpdated November 17, 2026By

How to test an AI agent before it talks to your customers

You test an AI agent by writing down 20 or so real conversations as scenarios, running every one through the real model, and scoring each reply three ways: plain code checks for the exact rules, a second model as judge for everything that needs reading, and a person who reads the report. Then every bad reply becomes a new scenario before anyone touches the fix.

That's the method we use on a WhatsApp sales agent we built for a service business. It caught things a demo never would have, like replies the owner's own staff called "too pushy" and answers that slid past the question the customer had actually asked. It also told us which model to run the agent on.

Last updated: November 17, 2026

How do you test an AI agent?

You replay realistic conversations through the agent and score every reply against facts you approved in advance. A few chats in a demo tell you very little, because the agent will handle the easy ones well and the demo rarely includes the hard ones.

Our loop has 4 parts:

  1. A set of written scenarios. Each one is a customer's side of a conversation, message by message, plus a note on what a good outcome looks like.
  2. A run through the real model. Same prompt, same tools, same facts file the live agent uses. A test against a simplified copy tests the copy.
  3. Scoring. Code checks first, then a judge model on the rest, then a person reads the summary.
  4. A rule for fixes. Nothing changes in the agent until the problem exists as a failing scenario.

If you're still deciding whether you need an agent at all, our guide to AI agents for business covers that first. This post assumes the agent exists and you're about to put it in front of customers.

Where do good test scenarios come from?

The best scenarios come from real conversations: your inbox, your WhatsApp history, your sales team's memory of the customers who made them sweat. Read a stack of real transcripts before you write a single criterion, because the transcripts tell you what actually goes wrong.

Our 20 scenarios came from 3 places.

The first one was the founder's own first test conversation with the agent. He played a customer the way his customers actually write, and that chat became scenario 1.

Six came from a simulator we set up before launch. The business owner and two staff members chatted with the agent as if they were customers and left notes on any reply they didn't like. They wrote 31 notes. Some were short: "too pushy", "that's wrong", "didn't answer my question". We grouped them, and 6 of the groups turned into scenarios.

The rest came from past conversations the business had already had with real customers, rewritten so nothing identifies a person.

We now recommend the simulator step to every business we work with. Your own people know your customers better than any test we can invent, and an hour of them trying to break the agent is worth more than a week of us guessing.

Which failure paths should you test on purpose?

Test the conversations that go wrong, because the happy path almost always works. An agent that answers a polite, single question well tells you nothing about Tuesday afternoon.

These are the ones we write a scenario for on every agent:

  • The haggler. Asks for a discount 3 times, in 3 different ways. The agent should hold the price and know when to bring in a person.
  • Four questions at once. "How much, do you have it Thursday, can you deliver, and do you take cards?" The agent should answer all four or say which ones a person will follow up on.
  • The one who goes silent. Writes twice, then nothing for a day. What does the agent send next, and does it send anything at all?
  • Someone abusive. The agent stays polite and hands off.
  • The competitor question. "Company X does it cheaper." The agent shouldn't badmouth anyone or invent a comparison.
  • The question outside the facts. Something the facts file doesn't cover. The agent should say someone will check, and never fill the gap itself.

Some of these will end in a handoff. That's correct. A test set where every scenario ends with the agent closing the deal on its own is a test set that's too kind.

How do businesses know if an AI agent is following the user instructions?

You check every reply against a written list of criteria and the approved facts, and you do it automatically on every change. In our setup a second model, the judge, reads each reply with the price table and the facts file in front of it and scores the reply against 9 criteria.

CriterionWhat the judge looks for
Matches approved factsNothing in the reply contradicts the facts file
No price a tool didn't supplyEvery number traces back to the price tool's answer
Answers the question askedThe customer's actual question gets an answer in this reply
Doesn't push what wasn't asked aboutNo add-ons or upsells the customer didn't raise
Hands off at the right momentA serious buyer, an upset customer or an unknown question reaches a person
Links only to the business's own siteNo outside links, no invented URLs
Tone fits the businessSounds like this business on its best day
Length fits a chat messageShort enough to read on a phone without scrolling
Clear next stepThe customer knows what happens now

Two details make the judge trustworthy. It runs on a fixed model id, so its standards don't drift when a provider updates a model quietly. And it gets the same reference material every time (the facts file and the price table), so it grades against your business.

A person gets the last word. The judge is good at spotting a pattern across 20 conversations, and then someone who knows the business reads the report and decides whether a flagged reply is a real problem.

What should a code check catch, and what needs a judge?

Anything that has an exact answer goes to a code check. Anything that needs reading goes to the judge. Anything that needs taste or business judgment goes to a person.

Code checks are cheap and never have an off day. If a rule can be written as "this must never appear" or "this must always come from there", write it in code and stop asking a model about it.

What to checkHow to check itWhy
Empty or cut-off replyCode checkExact, and it happens more than you'd think
Links point only to the business's domainCode checkA domain list is exact
Every price in the reply came from the price toolCode checkCompare numbers to the tool's output
Handoff tool was called when the scenario expects itCode checkThe tool call is in the log
Reply matches the approved factsJudge modelNeeds reading the reply in context
Answered the actual questionJudge modelNeeds understanding the conversation
Pushy, rambling or off-topicJudge modelNeeds the whole reply in context
Tone fits this businessPerson, helped by the judgeThe owner knows what sounds right
Whether a flagged reply mattersPersonBusiness judgment

The code checks in the test run are the same ones that sit in front of the live agent and block a reply before it's sent. You write them once and they protect both places. Our guide to building an AI agent explains why that final check should stay small.

How do you know which version of the agent produced a result?

Record the model id and a fingerprint of the prompt in every test report. Then any result can be traced to the exact version that produced it.

This sounds like bookkeeping, and it is. But without it you end up with a report that says 19 out of 20 passed and no way to tell whether that was before or after Tuesday's prompt change. The fingerprint is a short hash of the prompt and facts file. If it changed, the agent changed, and the old results no longer count.

We also keep every report. When the owner asks "was it always doing that?", the answer is already in a file.

How do you compare two models for the same agent?

Run the same scenarios with the other model, changing that one setting and nothing else, and read the two reports side by side. The scenarios, the judge and the criteria stay fixed, so any difference comes from the model.

That's how we chose the model for this agent. A cheaper model passed most scenarios but kept missing small signals: a customer who was clearly comparing options, a question asked a second time because the first answer had slid past it. Each miss looked minor. Side by side with a stronger model, the pattern was obvious, and we switched. For an agent that talks to buyers, one lost lead costs more than the difference.

The same side-by-side works for any change you're nervous about: a rewritten prompt, a new tool, a big update to the facts.

What should happen when the agent gets a reply wrong?

A bad reply becomes a failing scenario first. Then you change one thing in one place, and the change ships only when that new scenario and every older one pass.

The order matters. If you fix first and test later, you have no proof the fix worked, and no warning when it breaks something else. We learned this on an earlier version of this same agent, where quick fixes stacked up until a change for one conversation broke two others.

So when the owner sends us a screenshot with "that's wrong", the next steps are always the same:

  1. Write the conversation up as scenario 21.
  2. Run it and watch it fail.
  3. Make one change: the facts, a principle in the prompt, a tool or the model.
  4. Run all 21. Ship only on green.

Which of those 4 places the fix belongs in has its own table in our guide to building an AI agent. The test set is what makes that discipline possible, because it's the thing that tells you the fix held.

Should you A/B test an AI agent on live customers?

For a small business, no. Replaying real past conversations offline gives you a better answer, faster, without risking a single customer.

A/B testing needs volume. A business that gets, say, 40 sales conversations a week would wait months for a meaningful difference between two versions, and half its customers would talk to the worse one the whole time. Replaying 20 well-chosen conversations through both versions gives you a side-by-side report in minutes.

Live A/B tests make sense for companies with thousands of conversations a day. Most of the businesses we work with have a few dozen, and for them every conversation is a real lead.

How do you keep testing an AI agent after launch?

Read a sample of real conversations every week, and turn every complaint into a new scenario. The test set should grow for as long as the agent runs.

Launch is when the real material starts arriving. Customers write things your team never thought of. Some of those conversations will go well, some will go sideways, and the sideways ones are the next scenarios.

Our routine after launch:

  • Every week someone reads 10 to 15 real conversations, picked at random plus any that ended in a handoff.
  • Every complaint becomes a scenario, whether it comes from a customer, the staff or the owner. Same rule as before launch: failing scenario first, then the fix.
  • Any change to the prompt, facts, tools or model reruns the whole set.

It helps a lot when every conversation lands somewhere your team already looks. If the agent writes each lead and its transcript to the CRM, the weekly sample is a filter in a screen you already use. We cover that setup in AI agents and your CRM.

FAQ

How many test scenarios does an AI agent need?

Start with about 20 real conversations, including the hard ones on purpose. That's enough to catch most patterns before launch. The set then grows by one every time a real reply goes wrong, so a mature agent's test set is usually much larger than the one it launched with.

Can the same model judge its own replies?

It can, but we'd rather it didn't. Use a separate judge model with a fixed model id and give it the approved facts and price table as reference. A fixed id keeps the judge's standards steady between runs, and the reference material keeps it grading against your business.

Does a non-technical business owner need to be part of testing?

Yes, and they're the most useful tester you have. In our case the owner and two staff members wrote 31 notes in a simulator before launch, and 6 of the scenarios came from those notes. All they do is chat with the agent and say what sounds wrong.

How often should you rerun the tests?

Every time anything changes: the prompt, the facts file, a tool or the model. Run the whole set, every time. Most regressions show up in a scenario that had nothing to do with the change you made.

Is testing an AI agent different from testing a chatbot?

Partly. A chatbot only produces text, so you score the text. An agent also takes actions (looks up prices, saves leads, hands off to a person), so you also check that the right tool was called at the right moment. Our chatbot vs AI agent comparison explains the difference.

Next step

If you have an agent that's about to meet customers, or one that already does and keeps surprising you, talk to us. We'll help you build the first 20 scenarios from your own conversations and tell you what they find. Our AI agents service and WhatsApp agents pages show what we build.

More from the blog

We use cookies to make your experience smoother, faster, and a little more fun. If you stick around, we'll take that as a thumbs-up. Learn more