The 5-Minute Weekly Review That Keeps Your AI Agent From Slowly Getting Worse

Software fails loudly. It throws an error, or it times out, or the site goes down and someone gets paged.

AI agents fail quietly. The reply is slightly off. The tone drifted a degree. The booking got confirmed but for the wrong week. The customer got a coherent-sounding answer that happens to be wrong. Nothing errors. Nothing pages. The dashboard is green.

Three weeks later, a customer complains. You look back, and the last four Fridays of interactions have been getting steadily worse. Nobody noticed because nothing broke.

This is the drift problem, and it is the single thing that turns a working AI agent into an expensive liability. The fix is not a better model or a smarter prompt. The fix is a 5-minute weekly review that a human runs, on paper, every Friday.

Here is the exact review.

#Why weekly, why 5 minutes

Daily is too often. You will skip it after 3 days.

Monthly is too rare. By the time you notice drift, three weeks of bad interactions have already gone out.

Weekly is the sweet spot: catches drift in under 7 days, small enough that you'll actually do it, and short enough that it doesn't require anyone's calendar.

Five minutes is the maximum. If it takes longer, you're doing the wrong review. This is a smoke test, not an audit.

#What you need before you start

Two things:

1. Access to a raw sample of what the agent did this week. Not a summary. The actual messages, replies, tickets, or actions. If your vendor's dashboard only shows aggregates, this alone is a red flag.
2. A rubric from week one. Three to five criteria that describe what "good" looks like for your specific agent. Tone, accuracy, structure, whatever. Written down. If you don't have it, spend 10 minutes right now writing it. That is the last long meeting you will need.

Everything below assumes you have both.

#The review, minute by minute

#Minute 1: pull 5 random interactions from the last 7 days

Not the last 5. Random. Random is important because your eye will drift to the ones that look weird, and you'll miss the ones that look fine but aren't.

Cheapest way to make it random: pick a random hour of the day, then take the 5 interactions closest to it, sorted by time. Or use RAND() in a spreadsheet. Whatever. The point is: no cherry-picking, no self-serving sample.

If your vendor won't give you exportable interaction logs, you can't do this. That is a bigger problem than drift.

#Minute 2: read them

Actually read them. Not skim. Not open-and-scroll. Read as if you were the customer.

Three questions, per interaction, running in your head:

1. Does this answer the actual question? Not the surface question. The one behind it.
2. Does this sound like us? Same tone, same vocabulary, same brand.
3. Is anything factually wrong? Prices, dates, product names, policies, links.

You're not grading yet. Just reading.

#Minute 3: score each on your 3-to-5-item rubric

One row per interaction, one column per rubric item. Score 1 to 5. Do it fast.

Example rubric for a support agent:

• Correctness (is the answer right)
• Tone (does it sound like us)
• Completeness (did it end the ticket or leave it open)
• Escalation (did it hand off when it should have)

Do not overthink the scores. Trust your gut. This is a smoke test.

Add up the total. Divide by 5 interactions × 4 criteria × 5 max = 100. You get a percentage.

#Minute 4: compare to last week

Write the number next to last Friday's number in the same spreadsheet.

Three possibilities:

• This week ≥ last week. Great. Continue.
• This week is down 5 points or less. Noise. Continue, but note it.
• This week is down more than 5 points. Drift. Investigate.

The 5-point threshold is arbitrary but works. Adjust it once you have 6 weeks of data.

#Minute 5: write one sentence

If the score dropped, write one sentence about what you noticed. Not a full analysis. One sentence.

Examples:

• "Three of five had a wrong price for the new tier we launched Wednesday."
• "Two replies referenced our old refund policy from before June."
• "Tone was noticeably more formal this week; may be a prompt tweak I didn't approve."

If the score held, write nothing. Move on.

That's it. Five minutes.

#What you do with the sentence

You do not immediately email the vendor. You wait for pattern.

If the same kind of drift shows up two weeks in a row, then you email the vendor with both weeks' notes. That is signal. One week could be a bad sample. Two weeks in the same direction is drift.

When you do email, be specific. "Our review flagged wrong pricing in 3/5 samples in week of X and 2/5 in week of Y, both referencing the discontinued tier. Can you check the source data and prompt version?" Not "the agent seems off." Vague reports get vague responses.

#Why this catches what dashboards miss

Vendor dashboards aggregate. Aggregation hides drift. If 92% of interactions are fine and 8% are subtly wrong, the dashboard says 92%, which sounds great, and the 8% is where you lose money.

A rubric-based random sample doesn't aggregate. Every interaction gets read on its own. Bad ones stand out. Good ones don't hide the bad ones.

The dashboard is not lying. It is answering the wrong question. It answers "how much work did the agent do." The review answers "was the work correct."

#Who should do it

Not the vendor. Never the vendor.

Ideally the same person every week (consistency of grading matters), and someone senior enough that their instinct for "does this sound like us" is reliable. Founder, ops lead, head of the function the agent serves. Not the intern. Not a rotating volunteer.

If you cannot spare that person 5 minutes a week for something running unattended in your business, the agent has bigger problems than drift. It has no owner.

#The bigger point

The reason agents drift is not that models get worse. It's that:

• Your data changes (new products, new pricing, new policies).
• Your prompts get quietly tuned by the vendor.
• Model versions rotate under you.
• Retrieval indexes go stale.
• Edge cases that used to be rare become common.

None of those show up in a dashboard until they show up as a customer complaint. The Friday 5-minute review is the earliest signal you can build in without hiring anyone.

If your vendor has not offered you something like this, ask why. If you don't want to run it yourself, at BrainAI Team we run this Friday review for every client we host an agent for, and we share the sheet. If you're running an agent from someone else and want the format we use, we'll send it over. Start here.