Cutting LLM API Costs 60%: A Production RAG Post-Mortem

This post describes a client system, not my own site. Every figure below is one I recorded at the time. Where I cannot show my working, I say so.

The invoice was $4,200 for one week of LLM calls. The system was live, the users were active, and a rewrite was not on the table. The brief was narrow. Make it cheaper without making it worse.

Six weeks and two sprints later the same week cost $1,680.

Cost per query is a product decision, not a billing line. Nobody can make it smaller without first being able to see it.

What did this RAG system cost before and after?

Week 0 billed $4,200. Week 6 billed $1,680, a 60% reduction, with no database migration and no downtime. Cost per query fell from $0.08 to $0.032. Every number in this table was recorded on the same dashboard, six weeks apart.

Latency fell alongside cost, which surprised me. It should not have. A call that never reaches the model returns in milliseconds, and a call carrying 1,200 tokens of context finishes sooner than one carrying 8,000.

Continue Reading

Building Production AI Agents That Actually Work

12 min

20 Years in the Trenches: From Banking Code to AI Agents

4 min

What was the system, and where was the money going?

A retrieval system answering questions against a client's internal knowledge base. Next.js on the front, Convex as the backend, Qdrant for vectors, OpenAI for inference. It demoed well and held up for the first hundred users.

Four things were wrong at once, and none of them were visible.

Vector search returned 10 chunks per query, each over 800 tokens. The system prompt had grown to roughly 420 tokens of assistant boilerplate. There was no cache, so the same question cost full price every time it was asked. There was no routing, so a yes-or-no lookup paid the same rate as a multi-step reasoning task.

Nothing in that list is exotic. Every one of them is invisible until somebody instruments the call and reads the number.

Which five changes produced the cut, and what did each one save?

Five changes, applied in order, each one resting on the visibility the previous one created. These are the shares I attributed at the time.

Those five shares add to 65%, and the invoice moved 60%. They do not add, they stack. Change 2 only ever runs on the calls change 1 failed to catch, so its 15% is 15% of a smaller number by the time it lands. I am quoting the attributions as I recorded them. The 60% is the invoice, and the invoice is the number I trust.

How does a semantic cache cut LLM spend?

By never calling the model. An incoming query is embedded, the embedding is hashed, and the hash is looked up in Redis. A hit returns the stored answer. A miss calls the model, then writes the answer back under a SETEX with a seven-day expiry.

// packages/ai/cache.ts
const CACHE_TTL = 60 * 60 * 24 * 7;

export async function getCachedResponse(
  embedding: number[]
): Promise<string | null> {
  return redis.get(`rag:cache:${hashEmbedding(embedding)}`);
}

The similarity threshold is 0.92 cosine. Above that, two phrasings of the same question are treated as one question. Below it, they are not. I set the target hit rate at 40% and it settled at 62%.

A cache entry ages out on a seven-day timer and nothing invalidates it early. If the underlying answer changes on day two, the cache is wrong for five days.

That is the trade I took, and it is only safe because the knowledge base changed slowly. Event-driven invalidation is the harder half of the same problem, and I wrote up how I do it for this site's own cache in the cache handshake. Different cache, different system, same question. When do you evict?

What did prompt and context compression actually remove?

Two things: the system prompt, and the retrieved context around it. The prompt went from roughly 420 tokens of politeness instructions to 47 tokens that say what the model must do.

The context was the larger win. Ten chunks at 800-plus tokens each meant 8,000 tokens of retrieval before the user's own question was added.

Read your rendered prompts, not your templates. The template looks lean. The string that reaches the model is the one you are paying for, and it is usually full of things nobody chose.

Why does dropping low-relevance chunks improve answers?

Because low-scoring context does not sit there harmlessly. It competes. Qdrant exposes a score_threshold on search, so a chunk below 0.78 cosine is never returned rather than returned and then trimmed. Average relevance across the surviving chunks went from 0.65 to 0.89.

Hallucinations dropped when the noisy chunks stopped arriving. That was not the goal. The goal was tokens, and the quality improvement came free, which is the opposite of how cost work usually goes.

A relevance floor is a quality control that happens to save money. Cutting chunk count without a floor is a cost control that happens to hurt quality. They look identical on the invoice and behave nothing alike.

The pipeline also carries a max_context_tokens ceiling. If query plus context exceeds it, the oldest chunks are truncated first. It has never been the binding constraint since the floor went in, which is the point of a backstop.

When is a cheaper model the right model?

When the query does not need reasoning. A definition lookup and a multi-step analysis were paying the same rate, and a query classifier fixed that. Classification runs on a cheap embedding compared against labelled examples, so the routing decision costs close to nothing.

Roughly 40% of queries ended up on the cheap path. Routing was worth about 8% of total spend, which is smaller than it feels like it should be, and that is worth saying out loud. Caching moves four times more money than routing does.

The cost is a slight quality dip on edge cases where the classifier guesses wrong. I accepted that because it is measured, not assumed. Check current rates on the OpenAI pricing page before copying this table. The relative gap between tiers is what makes routing pay, and that gap moves.

Why cap output tokens on every call?

Because output tokens bill at the same rate as input tokens, and nothing about an uncapped response is predictable. One query returned 2,400 tokens because a user typed "tell me everything about". Nobody chose that cost.

Every call now carries a maxTokens value from that table. When the model stops early, the user gets a "Continue?" button rather than a truncated paragraph, so the cap is visible in the product instead of hidden in the response.

This was the smallest of the five changes, worth about 3%. It is also the one I would ship first in a new system, because it costs an afternoon and it removes an entire class of surprise from the bill.

What do you need to see before you change anything?

Cost per query, per call, with the model name attached. I spent the first day instrumenting and changed nothing, and that day paid for itself several times over. Langfuse carried the tracing.

Each call records input tokens, output tokens, total cost, model, latency, cache hit or miss, user and session id, query classification, and the rendered prompt. The rendered prompt matters more than it sounds. It is the only way to tell whether compression actually shipped.

The dashboard refreshes every five minutes and shows cost per query, cache hit rate, the split across models, and latency at p50, p95 and p99.

You cannot optimise what you cannot see. Every one of the five changes above started as a number on a dashboard that I did not like.

The same instinct drives the agent work I wrote up in building production AI agents that actually work. An agent you cannot trace is an agent you cannot debug or bill for.

What did not work?

Three attempts, all rolled back. I am listing them because the failures shaped the final design more than the wins did.

The third one is the one I would have shipped if nobody had asked who sits at those desks. It was technically sound and completely wrong for the deployment, which is the kind of mistake you only avoid by knowing the client's floor plan. I wrote about how much of that context matters in what actually works in enterprise banking.

How do you keep every LLM call on one path?

One wrapper, no exceptions. Every model call in the system goes through a single function, so caching, routing, guardrails and cost tracking cannot be skipped by a new feature written in a hurry.

// packages/ai/index.ts
export async function generateRAGResponse(
  query: string,
  context: Chunk[]
): Promise<AIResponse> {
  const embedding = await embed(query);

  const cached = await getCachedResponse(embedding);
  if (cached) return cached;

  const modelConfig = await routeQuery(query);
  const prompt = buildCompressedPrompt(query, context);
  const validated = await applyInputGuardrails(prompt);

  const response = await callLLM({ ...modelConfig, prompt: validated });

  await trackCost(response.usage, modelConfig.model);
  await cacheResponse(embedding, response.text);

  return response;
}

The module splits into cache, router, guardrails, cost tracker and shared types. None of that is clever. Its whole value is that there is one place to change a threshold and one place to read a bill. I use the same shape at much smaller scale on my own site, described in how I built an AI agent for my portfolio.

What would I do differently?

Instrument on day one of the build, not day one of the cleanup. The $4,200 week was not caused by a bad decision. It was caused by six good decisions nobody could see the price of.

Demos do not show you where a system bleeds. A hundred users will not either. Cost curves bend late, and by the time the invoice is loud enough to notice, the habit is already in the codebase.

Two smaller lessons held up. The "no rewrite" constraint forced better thinking than an open brief would have, because every idea had to survive the question of what it touched. And the boring change won. A Redis cache is not novel and it moved more money than the three clever experiments combined.

The system still runs. Spend flattened while usage kept climbing, and there have been no more surprise invoices.

Measured with

• Stack: Next.js, Convex, Qdrant, OpenAI, Redis, Langfuse
• Figures recorded week 0 and week 6 of the optimisation, two sprints apart
• Client system, not yabasha.dev. Figures are from that engagement's dashboard

Questions: @bayyash · Code: composable-ai-stack