Field notes: shipping an AI agent that talks to real customers
Guardrails, evals and the escalation path matter more than the model. What we learned putting one into production.
1 min read
The model choice took an afternoon. The escalation path — when the agent should stop and fetch a human — took three weeks and was the reason the project succeeded.
Evals are the deliverable
We now write the eval suite before the prompt. If you cannot measure a regression, you cannot ship changes with confidence, and an agent that cannot be changed safely is a liability.
