The One KPI Your AI Agent Should Be Graded On (And 5 That Just Look Impressive)

Your AI agent vendor sends a monthly report. It looks great.

432 tickets closed. 1.2 million tokens processed. 89% first-contact resolution. Response time down 71%. A little pie chart of "sentiment." A screenshot of a happy customer.

You read it, nod, and pay the invoice.

Six months in, none of those numbers has told you the one thing you actually need to know: is the agent making you more money than it costs, or not?

Vendors do not report on that number because it is bad for them. Every KPI on that monthly PDF is chosen to make the agent look busy. Busy is not the same as valuable. Here is the one KPI you should be tracking instead, and the five that are wasting your attention.

#The one that matters: net margin per unit of work

For every job the agent does (a reply, a booking, a lead handled, an invoice sent), calculate two things:

1. Revenue attributable to that unit of work. Not gross revenue. Not "the customer who touched the agent at some point." The margin on outcomes that would not have happened if the agent had not been there.

2. Fully loaded cost of that unit of work. Agent fee, cloud/API costs, your team's review time, cleanup labor, and the cost of every mistake the agent made this month (refunds, lost deals, unhappy customers who churned).

Divide (1) by (2). That number is either above 1 or below 1.

If it is above 1, the agent is a business. If it is below 1, the agent is a hobby you are paying for.

That is the only KPI that decides whether to renew. Everything else is noise.

Yes, this is harder to calculate than "tickets closed." That is the point. The reason vendors do not report it is that it forces the honest question. If your vendor will not help you calculate it, calculate it yourself. If they will not share the data you need to calculate it, that is your answer.

#Now here are the 5 that just look impressive

#1. Tickets closed / tasks completed / actions taken

The classic vanity metric. It says "the agent did stuff." It does not say the stuff needed doing, that the outcomes were correct, or that a cheaper tool could have done the same. A spam filter closes millions of tickets a day. Nobody thinks a spam filter deserves a $2,000 retainer.

Better version: "Tickets closed correctly, without human rework, that a customer confirmed were resolved." That number is usually 30 to 60% of the raw count. The gap is what your vendor is not telling you.

#2. Tokens processed / API calls / model runs

This one is worse than useless. It rewards the agent for being chatty. A well-designed agent should use fewer tokens over time as prompts and retrieval get sharper. If your vendor is proudly reporting rising token counts, they are effectively billing you for their own inefficiency.

Better version: "Cost per successful task, tracked monthly." Should be flat or falling. If it is rising, either the work is getting harder (fine, investigate) or the agent is regressing (not fine).

#3. First-contact resolution rate / accuracy percentage

Almost always self-graded. The agent decides what "resolved" means. The vendor's dashboard rolls it up. Nobody re-reads the actual conversations.

Sample it. Pick 20 "resolved" tickets per week at random and read them. In most rollouts, at least a quarter of them are not actually resolved. The customer gave up, or the reply was technically correct but useless, or the ticket was closed by the agent without a real answer.

Better version: "First-contact resolution, spot-checked weekly by a human reviewer against a rubric, with the sample published in the report." If the rubric is not written down, the number is fiction.

#4. Response time / speed of reply / latency

Fast is not the same as helpful. A robot that replies in 400ms with a wrong answer is worse than a human who replies in an hour with the right one. Customers know the difference.

Speed only matters below a threshold ("respond within business hours," "under 24 hours for tier-1"). Anything faster than that threshold is vanity. Once you are inside the window, further speed does not improve outcomes and might hurt them (customers dislike suspiciously instant answers on emotional messages).

Better version: "Percentage of interactions inside the response-time promise you made to your customer." If you have not made a promise, response time is a number without a scale.

#5. Customer sentiment / CSAT / NPS impact

Almost impossible to attribute cleanly to a single system change, and vendors know it. Sentiment shifts with pricing, product bugs, seasonality, ad campaigns, and the mood of the market. Yet the report will confidently claim "sentiment improved 8% since agent rollout" as if the agent were the only variable.

If you must measure it, measure it against a control group. That means: some customers keep interacting with the old process, some interact with the agent, and you compare. Almost nobody does this because it is uncomfortable. Without it, the number is a story.

Better version: "Complaints, refunds, and churn attributable to agent-handled interactions, tracked against a control group." Uncomfortable, precise, honest.

#The trap: aggregating theater

Notice what all five vanity KPIs share. They roll up. They average. They summarize.

The moment you aggregate, you hide the failures. 89% accuracy sounds great. The 11% is where the money goes. One botched refund on a $5,000 account costs more than 50 correctly-handled $30 support tickets.

The net-margin-per-unit KPI does not aggregate away failures. It forces them into the numerator or the denominator. Every bad outcome shows up as either lost revenue or added cost. There is nowhere for it to hide.

That is the discipline. Do the harder calculation once a month. Skip the pretty dashboard. Ask one question of your data: "did this month of the agent running earn back its cost with margin left over."

#What to ask your vendor this month

Copy and paste this into an email:

Can you share (a) the raw list of every task the agent handled in the last 30 days, (b) which of those tasks were reviewed or reworked by a human, (c) the outcomes for each (customer response, downstream conversion, refund, churn), and (d) your best estimate of revenue tied to those outcomes?

I want to run a net-margin calculation. Happy to do the math if you can send the data.

Watch what comes back.

If you get a clean CSV within a week, you have a real partner. If you get a "we don't track it that way," or a re-pitch of the monthly PDF, you have your answer.

#What we do at BrainAI Team

We run this calculation for every client, every month. Not because they ask for it. Because the number decides whether we keep the retainer or shrink it or hand it back.

Half the numbers on our monthly reports are the ones above (vanity metrics), because they are still useful for spotting trends. But the top row of every report is net margin per unit of work. If that number is under 1, we tell the client and propose to change scope, price, or shut it down. We have shut down two of our own agent deployments this year for that reason.

If you are running an AI agent right now and cannot answer whether it is net-positive on cost, we will help you run the math. One week, no charge. You keep the spreadsheet. Start here.