You set up your AI agent. You trained it. You delegated tasks. For a week or two, it was magic — emails triaged, follow-ups sent, questions answered. You saved hours.

Then, quietly, the drift started. The agent started answering questions wrong. It missed a policy change you made last month. It fell back on old instructions you forgot to update. The magic faded — not because the agent broke, but because you stopped maintaining it.

AI agents aren't set-and-forget. They're more like a garden than a machine. Left alone, they grow weeds. A weekly 15-minute audit is the difference between an agent that compounds in value and one that slowly degrades into noise. Here's exactly what to check.

Why Agents Drift

Before the checklist, understand the problem. Agent drift happens for three reasons:

  • Context decay. Your business changes. New products, updated policies, shifting priorities. The agent's instructions become stale because they reference old realities.
  • Knowledge base rot. Documents get outdated. Links break. Information that was true in January is false in August. The agent faithfully retrieves and repeats information that's no longer accurate.
  • Interaction drift. The agent's behavior subtly shifts over time. It starts being too verbose, or too terse. It interprets ambiguous instructions differently than it did at launch. These small changes compound.

The fix isn't to rebuild the agent every week. It's to run a systematic checkup that catches drift early, before it causes real damage.

The 15-Minute Weekly Audit

Set a recurring calendar reminder. Block 15 minutes, same time every week. Here's what to do with them.

Minutes 1–3: Review the Failure Log (3 minutes)

Every agent platform should have some kind of interaction log — a record of what the agent was asked and what it answered. If yours doesn't, build one. A simple "agent log" in a spreadsheet or Notion database where you record any interaction that didn't go well is infinitely better than nothing.

Scan the last week's interactions. Look for three things:

  • Wrong answers. The agent gave incorrect information. Flag it.
  • Escalations that shouldn't have happened. The agent punted to a human for something it should have handled. Fix the gap.
  • Confused responses. The agent was unsure, gave a hedging answer, or asked for clarification on something it should know. That's a knowledge base gap.

If you don't have a formal log, spend these three minutes scanning your support inbox, chat history, or wherever the agent's output lands. You'll spot the pattern quickly.

Minutes 4–6: Check for Policy Changes (3 minutes)

Think about what changed in your business this week. A new pricing tier? A shipping delay? An updated return window? A new team member with different responsibilities?

Now ask: does your agent know about this change?

If the answer is no, you have two options:

  • Quick fix: Add the change to your knowledge base immediately. This takes 30 seconds.
  • System fix: Update your onboarding process so that whenever a policy changes, updating the agent's knowledge base is part of the rollout checklist. This prevents the problem from recurring.

Most people skip this step because "it's just one small change." But small changes add up. A policy that shifted 2% each week is 104% different after a year — and your agent is still operating on the original. Three minutes a week prevents this entirely.

Minutes 7–10: Run the Smoke Test (4 minutes)

This is the most important part of the audit. Ask your agent 3–5 questions that represent your core use cases. The questions should be the same every week — that's what makes it a test. You're not looking for different answers; you're looking for consistent, correct ones.

Here's a sample smoke test for a small business agent:

1. "What's our return policy for electronics?"
2. "How do I escalate a VIP customer complaint?"
3. "What's the pricing for the Pro plan?"
4. "What's the process for a new team member onboarding?"
5. "What hours are we open on weekends?"

Run through them. Check each answer against your actual current policy. If any answer is wrong, outdated, or incomplete, fix it immediately. Then re-ask to confirm the fix worked.

The key insight: by keeping the same questions, you build a baseline. You can spot drift the moment an answer changes. A question that was answered correctly last week but wrong this week is a red flag that something in your setup changed unintentionally.

Minutes 11–13: Review New Edge Cases (3 minutes)

This is the proactive part. Think about the last week and ask: what situations came up that the agent wasn't prepared for?

Maybe a customer asked about a rare edge case — a bulk order, an international shipping question, a refund for a subscription that was paused 6 months ago. Your human team handled it, but the agent would have failed.

Add these edge cases to your knowledge base or decision trees. Each one you add is a future escalation you prevent. Over time, this builds an agent that can handle the long tail of situations — the ones that are rare but costly when they happen.

Minutes 14–15: Tune the Tone (2 minutes)

Read the last 3–5 responses your agent sent. How do they sound? Too robotic? Too casual? Too long? Too short?

Agent tone drifts. I've seen agents that started friendly and professional gradually become brusque and clipped. I've seen the opposite — agents that start concise and become verbose, overwhelming customers with information they didn't ask for.

If the tone has shifted, add a quick directive to the agent's system prompt. Something like: "Keep responses under 3 sentences unless the customer asks for details. Use a professional but warm tone." Two minutes of tone tuning prevents a month of awkward interactions.

The Monthly Deep Dive

Once a month, extend the audit to 30 minutes and add two more checks:

  • Full scenario replay. Run 10–15 questions instead of 3–5. Cover every major category your agent handles. Verify each one against your actual current policies.
  • Knowledge base health check. Scan your knowledge base documents for outdated information. Check dates, links, product names, and pricing. Remove or archive entries that are no longer relevant. Consolidate entries that have grown redundant.

The monthly deep dive catches the stuff the weekly audit misses — the slow rot, the accumulated drift. It's especially important in the first 90 days, when your agent is still finding its footing and your knowledge base is still being shaped by real interactions.

A Real-World Example

I worked with a small e-commerce business that had an AI agent handling customer support. The agent worked well for about a month. Then it started approving returns for products that were past the return window.

The weekly audit caught it: the smoke test question "What's our return policy for electronics?" returned an answer that was two weeks out of date. The business had changed its return window from 30 days to 14 days for electronics — a policy change that the team had discussed in Slack and never documented.

Two minutes to update the knowledge base. Problem solved. Without the audit, they would have processed three more weeks of incorrect returns before someone noticed the pattern.

The Bottom Line

An AI agent that you never audit is a liability. An AI agent that you audit weekly is a compounding asset. The 15-minute investment compounds: every issue you catch early prevents a cascade of wrong answers, frustrated customers, and wasted human time later.

Set the reminder. Build the checklist. Run the smoke test. Your future self — the one who isn't cleaning up after a month of agent drift — will thank you.

This post is part of our ongoing series on practical AI agent management. For more, read The Feedback Loop: How to Train Your AI Agent to Get Better Every Day and How to Build a Knowledge Base Your AI Agent Can Actually Use.