The 7-Day AI Agent Trial: What to Measure Before the Second Invoice

You signed up for a month. Maybe a two-week pilot. The vendor sent a Loom, a Notion doc, and a Slack invite. Now the agent is running, and in three weeks you owe them another $2,000.

You do not need three weeks to know if it works. You need seven days and a spreadsheet.

Most trials collapse into vibes. The agent "seems to be working." The vendor sends a recap that lists tickets closed and hours saved. Nobody checks the numbers against what a human would have done in the same week. That is how businesses end up on month four of an agent that quietly costs more than the intern it replaced.

Here is the actual checklist we use with clients on a 7-day pilot, and what each row is trying to catch.

#Day zero: write down what "working" means

Do this before the agent runs a single task.

Pick one job. One. Not "handle support." "Answer tier-1 refund questions inside 15 minutes during 9-to-5, in the same tone as our best rep." That sentence is your success bar.

Then write three numbers next to it:

1. Volume you expect. Roughly how many of these tasks per day.
2. Baseline cost. What it currently costs you (staff time, contractor rate, or the deals you lose because nobody replies).
3. Break-even. The point where the agent stops being cheaper than the current setup.

If you cannot fill in those three lines, you are not ready to trial anything. The vendor will happily fill them in for you. Do not let them.

#Day 1 to 3: watch the failure mode, not the wins

The first thing to measure is not what the agent does well. It is what it does when it is wrong.

Wrong on an AI agent has three flavors, and they are very different problems:

#Flavor 1: it says "I don't know" and stops

This is the good flavor. The agent hit its confidence threshold and handed off. You get a queue of escalations. You reply to them. Nothing burns down.

Track the escalation rate. If it is above 40% in week one, the agent is not doing the job. It is a router with a chatbot glued to it. That is fine if the vendor priced it as a router. It is not fine at $2,000 a month.

#Flavor 2: it makes something up and sends it

This is the flavor that kills trust. Refund confirmed for the wrong order. Meeting scheduled for a timezone that does not exist. Invoice sent with the vendor's demo values instead of yours.

You cannot spot these by watching a dashboard. You need to sample. Pull ten random completed tasks per day, by hand, and compare them to what actually happened. If one in ten is wrong in a way the customer would notice, the agent needs a human review layer before it touches real work. That layer is not free. Ask the vendor who pays for it.

#Flavor 3: it does the right thing, badly

The invoice went out. The invoice was ugly. The customer got a reply. The reply did not sound like you. The lead got a follow-up. The follow-up burned the lead.

Grade a sample by hand every day of the trial. Pick five interactions. Rate them 1 to 5 for tone, accuracy, and next-step. Do it on paper. If the median is under 4 by day four, you have a taste problem that a system prompt will not fix.

#Day 4: check the drift

By mid-week, a healthy agent should be more accurate than on day one. It has seen more of your data. If the vendor built the pilot right, they have been tuning behind the scenes.

If accuracy is flat or worse, ask one specific question:

"What changed between Monday and Thursday, and who changed it?"

A good answer includes a diff. A file. A prompt version. A named person. A bad answer is "we made some tweaks." Tweaks are what vendors say when nothing changed and they hope you will not notice.

The reason this matters is that at the end of the pilot, you will decide whether to renew. If nobody was iterating in week one, nobody will iterate in month two either. You are paying for maintenance you are not getting.

#Day 5: the cost math nobody runs

By day five you should have enough data to run the actual number. Not the vendor's number. Yours.

Add up:

• Agent monthly fee. Whatever the invoice will say.
• Your time reviewing its work. How many hours a week did you or your team spend correcting, escalating, or cleaning up. Multiply by your loaded hourly cost.
• Cost of the errors it made. Refunds you should not have issued. Deals you lost because a reply went sideways. Even ballpark it.

Compare that total to what the same work cost you before the trial. If the agent is not saving you at least 30% on that combined number in week one, do not renew at the pilot price. Week one is when the agent is running against the easiest slice of the work. It will not get more efficient in month two. It will get more of the hard cases.

Vendors will push back on this math. They will say "we need three months to show ROI." Sometimes that is true, for complex integrations. For a tier-1 support agent, a scheduling agent, or a lead follow-up agent, seven days is enough to see the shape of it.

#Day 6: talk to the humans it displaced or unblocked

Whoever's job the agent is doing part of, sit with them for 30 minutes. Not on Slack. In person or on video, with your camera on.

Ask three things:

1. What is the agent doing that they would have done differently.
2. What is it doing that they could not have done at all (faster response, night shift, volume spike).
3. What is it doing that is making their job harder.

The third one is the one you will not hear on a metrics dashboard. Agents create work upstream and downstream: cleanup of bad outputs, retraining when your process changes, awkward handoffs where the customer already talked to the bot and now has to re-explain. If your team has a list of pain points that outweigh the wins, the agent is a net negative even if the numbers look fine.

#Day 7: the "would we notice if you turned it off" test

On the last day of the trial, ask yourself one question with the whole team in the room:

"If we turned this agent off tomorrow at 9am, what would break?"

Three possible answers:

• "Nothing. We would just go back to doing it ourselves." The agent has not become load-bearing. It might still be worth the money if it is buying back real hours, but you have leverage to renegotiate. You are not locked in.

• "We would drop the ball on 20 leads a day." It is doing real work. Renew, but negotiate an exit clause and a data-portability clause first. You do not want to be trapped.

• "We would not even know how to run this process anymore." Stop. That is a red flag, not a green one. The agent has absorbed institutional knowledge that lives nowhere else. Before you renew, get the vendor to hand over the workflow, the prompts, and the data flows in a form you own. If they refuse, the agent is not a tool. It is a hostage situation.

#What we tell our own clients

At BrainAI Team, when we run a 7-day trial with a business, we agree on the three success numbers before day zero. We share the sample-audit worksheet from day one. The mid-week iteration diff is a doc the client can read. Cost math on day five is a spreadsheet, not a slide.

Half the time, week one tells us the agent is a fit and we go monthly. About a third of the time, we come back and say the scope is wrong and we rework it. And sometimes, we come back and say we would not renew this if we were you.

That last one is the honest answer that vendors trained on retention metrics almost never give.

#The soft rule

If a vendor will not run a trial this way, that is the data. You do not need to sign to learn what you needed to know.

If you are staring at a renewal invoice this month and cannot answer these questions, we will run the audit for you. Free, one week, no upsell. You get the spreadsheet at the end. Start here.