Support
212 customers, refunded twice in 38 minutes
A business ran a customer support agent with a refund tool. Before launch, the agent passed an offline eval at 96%. The eval used 400 real, anonymised conversations, each run three times. One prompt line said: if a tool call fails, retry once. In production the payments system was slow. Its typical slow response was 4.8 seconds, and the tool timed out at 3 seconds. So calls marked failed had in fact gone through, and the retry paid a second refund.
How it wentIn 38 minutes at full traffic, 212 customers got two refunds each. Error rates looked normal. A customer ticket and a finance check found the problem, not the monitoring. The business is unnamed in the post.
Try it · based on the source's workflow
Before any agent at [business name] can use a tool that moves money, run this check. Test the tool with real response times, including slow and timed-out calls. Each conversation may call a money tool at most once, and every call carries a unique key so a retry cannot pay twice. Retries live in the tool layer, never in the prompt. Launch to [5%] of traffic first, and show me the test results and the first day's refund log before any wider launch.
A starting template based on the workflow the source describes. Review it before you use it; you are responsible for what it does.
212 customers, refunded twice in 38 minutes — from The SmallOffice Brief, a daily brief of real AI cases for small businesses: https://smalloffice.space/case-212-customers-refunded-twice-in-38-minutes