Evals are the new unit tests
If an agent isn't gated by a regression suite, it isn't in production — it's in rehearsal. How we wire pass-rate gates into CI.
Every agent we ship is gated by an eval suite the same way application code is gated by unit tests.
The gate
A deploy is blocked unless eval.suite(agent).pass_rate >= 0.97 on the regression set. The set grows with every incident: each production miss becomes a permanent test case.
What to measure
- Task success against golden answers
- Tool-call correctness (right tool, right arguments)
- Cost and latency budgets per task
- Refusal correctness on out-of-policy requests
If you only remember one thing: agents drift, suites don't. Pin your quality to the suite.