A/B test changes on real customers
Split new conversations between the live version and your draft, see which one Dazza handles more of on its own, and publish the winner in one click.
Not sure a new tone, rule or instruction makes Dazza better? Test it on real customers before everyone gets it.
Start a test
- In Agent, change what you want to test, such as the tone, a guidance rule or the instructions.
- Press Test on customers at the top.
- Choose how to split new conversations: 50 / 50, 80 / 20 or 90 / 10 (the second number is your draft’s share).
- Press Start test.
Your draft is saved and frozen as a new numbered version. New conversations are shared out between it and the live version, and each customer keeps the same version for good, even if they come back. Conversations already under way carry on with the live version.
The welcome message, suggested questions, name, colours and nudges stay as they are live during a test, because everyone sees them before they write. A test compares how Dazza answers.
While a test runs you can keep editing your draft, but publishing waits until the test ends.
Read the results
The test card shows each version side by side:
| Measure | What it means |
|---|---|
| Handled by Dazza | Answered without your team, and never said it didn’t know |
| Handed to the team | Handed over, or taken over, at any point |
| Rated good | Share of rated chats the customer rated Good |
| Answer quality | Average 1 to 5 score for Dazza’s replies |
| Conversations | How many each version has had |
At the top, a plain verdict:
- Too early to tell: each version needs at least 30 conversations, or the two are still close. It says roughly how many more conversations, and at the current pace how many days, it should take.
- Leaning towards version …: one is ahead, but not by enough to be sure.
- Version … looks better: it handles clearly more conversations on its own (a difference that size is very unlikely to be chance), and its ratings and answer quality are no worse.
- Mixed results: one handles more on its own, but customers rate it lower or its answers score lower. Read a few of its conversations first.
- No clear difference: after plenty of conversations, the two do about as well. Publish the one you prefer.
The chance figure is how likely the draft is to handle more conversations on its own than the live version, given what’s happened so far.
End the test
- Publish version … makes your draft’s version live for everyone, straight away.
- Stop and keep version … ends the test and keeps the live version.
Past tests are listed under A/B tests, and the versions list shows which version was tested and whether it won. Starting, publishing and stopping are recorded in the audit log. In the inbox, a conversation in a test shows its version and side in the side panel.
A/B tests are on Growth and above. Only admins can start and end them. If your plan changes, a running test pauses: everyone gets the live version until you’re back on Growth.