Your AI Agent Doesn't Need a Bigger Context Window

Everyone is excited about 1M-token context windows. Longer conversations. Whole codebases in a single prompt. Attach your entire quarterly plan and ask questions.

I run every one of my agents on 1M context. I still don't recommend you upgrade for it. Not for the reason you think you're upgrading.

The pitch for bigger context is that your agent can "remember more." It can't. It can hold more, which is a different thing.

#Holding is not remembering

Give a model 800,000 tokens of chat history and ask it what happened in message 4,000. It will confidently make something up. Not because it's a bad model. Because attention over 800k tokens is a lossy compression of a lossy compression. Recall stays sharp for the last few messages and the very first few. Everything in the middle blurs.

You already know this if you've built agents. The 2024 "lost in the middle" papers were the first look at it. The 2026 versions of these models have not fixed the fundamental problem. They have made it less visible. That is not the same thing.

Try it right now on whatever model you use. Paste in a long meeting transcript. Ask it about a decision made two-thirds of the way through. Watch it confabulate a plausible-sounding answer that never happened.

#What you actually want

You don't want the agent to hold everything. You want the agent to look things up when it matters.

• Notes on disk, in a directory the agent knows the shape of.
• A search index (BM25, vectors, both) that the agent can query on demand.
• A daily note file where the agent writes what it did today.
• A memory file where the agent writes what it learned about you, your business, and your prior decisions.

None of this needs 1M context. Most of it works on 32k. The magic is in the retrieval layer, not the window size.

Every one of our production agents has less than 50k tokens of live context at any moment. What they have instead is a filing system with a search interface. When the agent needs the answer to "what did the founder say last Tuesday about pricing," it doesn't try to remember. It runs a search over its own notes and reads the top match. That takes 200 milliseconds and one small API call. It costs a fraction of a cent.

#Where bigger context does help

Two places, both narrow.

1. First-pass code review of a large file. Reading a 15,000-line file in one shot beats chunking. Real gain here, no argument.
2. Debugging a long conversation trace. When something went wrong three hours ago in a run, dropping the full trace in and asking "what changed at message 2200" saves you from writing a query manually.

Both of these are analyst work, not agent work. They are moments when you, the human, want the model to be an oracle over a big blob. That is fine. That is what oracles are for.

The agent that runs your business does not need to be an oracle. It needs to be a diligent clerk with a filing cabinet.

#The cost you are not measuring

Longer context is more expensive. Everyone knows this. What you may not know is that the cost is linear in input tokens but the recall improvement is not. You are paying two to five times the API bill to get a 10% better memory of the middle of your conversation. That is not a trade you would make in any other engineering domain.

Two years ago we optimized token budgets. We looked at every prompt like an accountant. Then the bigger-context models arrived and everyone stopped optimizing. That was the mistake. The cost per query went up, the quality of the middle-of-context recall barely moved, and people started calling it progress.

#What to do this week

• Look at your agent's longest conversation. Ask it something concrete from the middle of it. If it invents an answer, you have a recall problem. It will not be fixed by upgrading to a 5M-token model.
• Take the middle-of-conversation content that matters and move it to a file. Have the agent read the file when it needs to.
• Give the agent a daily note where it writes down what happened. On the next run, have it check the note before it answers.
• Watch what happens to the bill and to the answers.

The upgrade that fixes your agent is not a bigger window. It is a smaller working set with better retrieval. That is boring and unfashionable. It is also what actually works.

If you want us to look at your agent's memory and retrieval stack, start here.