Make a clear, safer operating decision.
You will be able to create a small but meaningful evaluation set before customers encounter the workflow, including the cases most likely to expose unsafe behaviour.
Good agent work is useful before it is impressive.
A few satisfying demos are not evidence of reliability. An evaluation set defines what acceptable behaviour means for the actual job and makes it possible to compare versions without relying on memory.
Make the relationship visible.
The language that keeps the work clear.
Work through the decision in order.
- 01
Collect representative cases
Use consented, de-identified, or synthetic cases based on normal work and known exceptions.
- 02
Write expected behaviour
Define the routing, answer, action, or stop condition before running the system.
- 03
Score quality
Use a simple rubric for correctness, completeness, tone, policy alignment, and handoff quality.
- 04
Set a launch gate
Decide which failures block launch, which require a narrower scope, and which can be monitored in a pilot.
A realistic, bounded implementation.
A real-estate inquiry agent is tested with a straightforward buyer request, a vague request with missing location, an existing client asking about a contract, and a prompt-injection-style message that asks it to ignore policy.
The expected outcome for the normal case is a complete record and assigned agent. The contract query routes to a person; the malicious instruction does not change the agent’s boundary; the vague request receives one clarifying question and then a handoff if it remains unclear.
The team records actual results, reasons, and fixes. It does not call the build ready because the normal demo looked polished.
Use this copyable working template.
Adapt it to the client’s evidence, policy, people, and tools. Do not treat placeholders as approved instructions.
Put the operating system around the agent.
Use controlled test contacts, tasks, conversations, workflow history, and reporting to run evaluations without mixing them into live client records.
Before you move on
- Write ten cases: six normal, two edge, one sensitive, and one adversarial or failure case.
- Score each case before editing the agent.
- Choose the failure that would most damage trust and fix that first.
- Cases reflect the real workflow.
- Expected behaviour includes safe stopping.
- A scorer can explain why a result passed.
- Launch criteria are agreed before launch.
Put the learning to work
Build an AI service people can trust.
Create a free Spacebrain account and use the operating layer around your AI service.