The Hidden Cost of Thinking With AI

Every senior developer I know has had this moment: you check your AI API bill and something feels off. You did maybe three hours of serious work. The number on the screen suggests you ran a small server farm.

You didn't make a mistake. You just didn't know how the pricing actually works.

The external observation

LLM context caching — the mechanism that lets models like Claude reuse previously computed token representations — was designed to save money. And it does. Dramatically. Anthropic's Sonnet 4.6 charges $3 per million input tokens at base rate, but only $0.30 per million for cache reads. A 10× reduction.

The math looks compelling on paper. A session with 167 million cached tokens costs ~$50 in reads rather than ~$500 without caching. You save $450.

What the pricing page doesn't tell you: that $50 is also $50 more than you needed to spend if you'd managed your session differently.

The gap

The tooling we use for AI-assisted work was not designed for knowledge work. It was designed for task completion.

You open a session, load context, do some work, close. The model forgets everything. Next session, you load it all again — MEMORY.md, CLAUDE.md, every file you read last time, the conversation history. Each token re-enters the context. Each turn, the model reads it all from cache.

What's missing isn't a better model. It's a system for what to keep, what to discard, and what to actually learn from the work you've done. Most AI sessions are expensive and amnesia-inducing at the same time.

What I built

Today I built a research repo — a personal knowledge base designed to compound over time. The structure is simple:

research/
├── topics/[slug]/
│   ├── summary.md      # findings with inline confidence markers
│   ├── sources.md      # all sources with access dates
│   ├── notes.md        # raw data, LLM deltas, contradictions
│   └── prompts.md      # single consolidated cross-LLM prompt
└── concepts/           # evergreen notes, reusable across topics

The governance layer is what makes it work. Every investigation gets a YAML frontmatter with lifecycle status (seed, incubating, mature, needs-refresh, superseded), a confidence rating, and a review_by date. Every finding gets an inline confidence marker: [alta confiança], [média confiança], [especulativo]. Every investigation requires a Contradições e Dissensão section — not optional, mandatory.

The cross-LLM prompt is a single consolidated query I run on 2–3 other models after completing an investigation. Not to validate — to find the blind spots. Today, running it on three different models surfaced: OpenAI's pricing structure (no write premium, but ~50% cache hit rate), a bug in my thinking about Gemini's minimum token threshold (conflicting data: 1K–4K tokens vs. 32K tokens across sources — still unresolved), and a cost trap I hadn't considered.

The unexpected insight

The write-heavy cost trap: if you pause for more than 6 minutes between prompts, Anthropic's 5-minute TTL on cache writes expires. Your next turn pays the write premium (1.25× base price) without ever benefiting from the read discount (0.1× base price). In a session with frequent pauses, you can pay more than in a session of equivalent size with continuous work.

But the deeper insight isn't about the pricing mechanics. It's about what 167 million cached tokens actually represents.

That number is the weight of every decision made in a session: every file read, every message sent, every tool call result. Most of it is dead weight by turn 50. The system prompt and MEMORY.md are loaded fresh every turn. File contents that were relevant 2 hours ago are still there, silently increasing the cost of each subsequent thought.

Context is not an archive. It's working memory. Clear it before it clears your thinking.

The philosophical takeaway

There's a principle in knowledge management called progressive summarization: you don't keep everything, you keep what survives repeated distillation. Each pass removes noise and amplifies signal.

AI sessions fail in the opposite direction. They accumulate. Every turn adds tokens; nothing is removed. By hour three, you're paying to re-read the scaffold of decisions that should have been digested hours ago.

The research repo inverts this. It's designed to extract signal from each investigation and discard the rest. The summary.md is the distilled version. The notes.md holds the raw material. The concepts/ folder holds what's genuinely reusable across topics. The session context is expendable; the repo is not.

The rule I'm now applying: sessions ≤ 90 minutes for focused tasks, /compact at 45 minutes, /clear between distinct work streams. Not because it's cheaper (though it is), but because a fresh context produces better work than a cluttered one.

Do you actively manage context in your AI sessions, or do you let it accumulate? And if you do manage it — what's your system?

Sources

1. Anthropic — Prompt Caching Documentation. https://platform.claude.com/docs/en/build-with-claude/prompt-caching. Accessed 2026-04-02.
2. Anthropic — Claude Pricing. https://platform.claude.com/docs/en/about-claude/pricing. Accessed 2026-04-02.
3. Sankalp — How Prompt Caching Works (vLLM / Paged Attention). https://sankalp.bearblog.dev/how-prompt-caching-works/. 2025.
4. Atlan — LLM Context Window Limitations in 2026. https://atlan.com/know/llm-context-window-limitations/. 2026.
5. 16x Engineer — LLM Context Management: How to Improve Performance and Lower Costs. https://eval.16x.engineer/blog/llm-context-management-guide. 2025–2026.