Practice tests for your AI agent

• Practice tests are under Evals in the menu on the left. Users and Owners can use them.
• Evals appears only if your clinic has an active subscription, or if a partner manages your clinic’s subscription. If you don’t see it, ask the person who manages your clinic’s subscription, or write to support@allomia.com.
• The practice test screens are only in English.
• Practice tests use the agent’s saved instructions. If you just changed them, click Save Settings first.
• Practice tests are written, not spoken. They check what the agent answers, based on its instructions. They don’t check its voice, how well it hears patients, or answers that come from your clinic documents or actions. For those, use Speak with Agent. See Trying your AI agent yourself.
• Practice tests don’t place phone calls, so they don’t use call minutes. Each run you start with Run is counted in your plan’s usage, like Chat.

1. In the menu on the left, click Evals, then click the Create tab.
2. In Eval Name, type a name, such as “Booking request: asks for name and date of birth”. In Description, say what the test checks.
3. Under Organization & Agent, choose your clinic in Organization, then the AI agent to test in Agent. The agent’s instructions appear below, so you can check that it’s the right one.
4. Under Conversations, click Conversation, then + User. Type what the patient says, for example: “Hi, I have a sore throat and I’d like to see a doctor this week.”
5. Click Conversation again, then + Agent Eval. This card describes a good answer. In the list at the top of the card, choose LLM-as-a-Judge: an AI reads your agent’s answer and decides if it passes.
6. In Pass Criteria, write what the answer must do, for example: “Asks for the patient’s full name and date of birth.” In Fail Criteria, write what it must never do, for example: “Suggests a treatment or a medicine, or says what the illness might be.”
7. To try the test before saving it, click Test, next to Organization & Agent. The result opens on the right.
8. Click Save Evaluation. Your test now appears in the Evals tab.

To check a longer conversation, add more pairs: a + User message, then a + Agent Eval card. Your AI agent answers each patient message that is followed by a + Agent Eval card. You don’t need the other choices, such as + Agent Mock or + Tool Response, or the Evaluator settings.

1. Click the Evals tab. On your test’s row, click Run.
2. In the Run Evaluation window, choose your clinic in Organization and the agent in AI Agent. Tick Navigate to runs page after completion to go to the result when it’s ready.
3. Click Run, and wait until the run is over.
4. In the Runs tab, look at the Status column: Passed, Failed, or Error if the test couldn’t finish.
5. Click View. Conversation shows what your AI agent said. Turn Results shows each check and, under Judge Reason, why a check failed.
6. If the test failed, change the agent’s instructions, click Save Settings, and run the test again. See When your AI agent gives a wrong answer.

A suite is a group of practice tests that you run in one click, for example after each change to your agent’s instructions.

1. Click the Suites tab, then New Suite.
2. In Name, type a name, such as “Before each change”. Under Evaluations, select the tests to include.
3. Click Create Suite.
4. On the suite’s row, click Run. Choose your clinic and the agent, then click Run suite. The tests run one after the other, and the results appear in the Runs tab.

• Evals isn’t in the menu: your clinic needs an active subscription. Ask the person who manages it, or write to support@allomia.com.
• The status is Error: the test couldn’t finish. Check that you chose an agent, then run it again. If it happens again, write to support with the name of the test.
• The test fails, but the answer looks right to you: the criteria may be unclear. Write each criterion as one simple thing the answer must or must not do. To change a test, click the pencil on its row in the Evals tab, then click Update Evaluation.
• The agent passes the test but answers differently on calls: the agent’s words can change from one conversation to the next, and practice tests are written, not spoken. Also try a call with Speak with Agent.