Résumé
All writing

7 min read

Stop Chasing the Best LLM — Find the Bottleneck Instead

AI Agents · LLMs · ChatGPT · AI

I have recently been seeing some version of the same question. Should I use GPT or Claude? Is Gemini better than DeepSeek? Which embedding model has the highest MTEB score? What’s the fastest inference provider?

These questions though repetitive are reasonable questions. However, they’re also, sometimes the wrong question to be asking first.

The assumption underneath such question is that a better application comes from choosing a better model. In practice, most AI systems don’t fail because a wrong LLM was used. They fail because someone optimized a part of the system that was not the actual bottleneck of the system.

A support chatbot giving bad answers usually doesn’t mean a smarter model is needed. Sometimes it might need a better retrieval. A coding assistant that feels slow usually doesn’t mean a fast inference provider is needed. Sometimes it might need fewer LLM calls in its loop. A RAG system hallucinating over your own documents usually doesn’t need another benchmark leader sitting behind it. It might need better embeddings or cleaner context reaching the model in the first place.

The job of an AI engineer isn’t choosing the smartest model anymore — benchmark websites and even LLMs themselves can help narrow that choice. The real job is identifying the weakest part of the system and improving it.

Every engineering hour spent optimizing the wrong component is an hour not spent improving the actual bottleneck. Spending days comparing GPT and Claude is wasted effort if your retriever is only finding the right document 60% of the time. Before optimizing anything, identify what’s actually limiting the system.

Every AI application is the same pipeline

Whether you’re building a chatbot, a coding assistant, a document search tool, or the agentic RAG loop, they’re all variations on the same shape: a user’s request goes in, something retrieves relevant context, that context gets assembled and checked, an LLM generated from it, and a response is provided. Every stage in that chain is a place the system can be quietly failing, and the LLM is only one of them.

Fig 1 — The same pipeline underneath almost every AI application, with the five places it actually tends to break.

You don’t upgrade the engine in a car when the tires are flat. In same way, LLM cannot be changed when retrieval is broken.

Bottleneck #1: The model

You should spend real time comparing models only after everything is ruled out. And even then, the first question isn’t “which model is best”. It’s “what’s the minimum model that clears my quality bar”. If a smaller, cheaper model answers 95% of your production queries correctly at a third of the cost and half the latency, that’s usually the better engineering decision, not a compromise.

Before you compare anything, eliminate hard constraints like budget per request (not per month — per request is what actually decides whether a design is viable at your volume), a p95 latency ceiling (the number 95% of your requests beat, which is what a user actually feels, not your average), a privacy or data-residency floor, and the context window you genuinely need. Then only you should start comparing reasoning quality, coding performance, or tool use.

One mention worthy mistake: picking a model because it advertises a million-token context window. Every frontier model claims one now, but advertised context and effective context are different numbers — current long-context benchmarks show multi-fact retrieval accuracy dropping 30 to 60 points once you’re past roughly 200k tokens, even on models that headline a 1M window. If your workload actually lives in that range, test it there specifically. Don’t take the spec sheet’s word for it.

In addition, public benchmarks alone cannot be trusted to make final call. They’re useful for narrowing a shortlist, not for picking a winner. Run 100 to 200 real examples from your own workload through your finalists and compare the actual outputs before you commit to anything.

Bottleneck #2: Retrieval

This is one of most underrated bottleneck. For many RAG applications, improving retrieval quality — through better embeddings, chunking, reranking, or indexing — often produces larger gains than switching to a stronger LLM. On a typical RAG workload, the embedding model alone can shift retrieval precision by 20 to 30%. If the wrong documents get retrieved in the first place, it doesn’t matter how capable the model downstream is. It can’t invent the right answer when a wrong context is provided.

What’s actually worth evaluating are: multilingual support if you need it, retrieval quality on benchmarks matched to your task, dimensionality, inference speed, storage cost, and price per token.

Dimensionality is the trade-off most teams get wrong by not thinking about it at all. Doubling embedding dimensions doesn’t double retrieval quality — for most applications, 768 to 1024 dimensions is close to practical ceiling for what you gain, while going from 1536 to 3072 dimensions costs roughly 6x the storage for a marginal quality improvement. If your vector database is holding millions of documents, that’s not a rounding error.

The other thing worth knowing before selecting: swapping LLMs might take you few minutes. Whereas, swapping embedding models means rebuilding your entire vector index from scratch, which might take hours or even days depending upon the corpus size. Choose this one more carefully than you’d choose the model, not less — it’s the harder decision to undo.

One area that’s often overemphasized is prompt engineering. A better prompt can absolutely improve outputs, but its impact is bounded by the information reaching the model. If retrieval returns the wrong documents or incomplete context, no amount of prompt tweaking will consistently recover the right answer. Prompt engineering should refine a good system, not compensate for a broken one.

Bottleneck #3: Orchestration

Most people assume latency comes from slow models. However, many times it comes from too many model calls.

Taking a simple agentic RAG loop that: decomposes the question, retrieves, validates what came back, reformulates and retries if it wasn’t good enough, then finally generates. That will already have five LLM calls for a single user question, before you ever added a router, a reranker, or a second validation pass. Swapping in a model that’s 30% faster barely moves the needle if the loop itself is making five calls it didn’t need to make.

The highest-leverage fixes here usually aren’t model changes at all: prompt and context caching (skipping this specifically is one of the most common unforced errors — it can add on the order of 80% in avoidable cost), routing simple requests to smaller model instead of running everything through the frontier one, parallelizing retrieval calls that don’t actually depend on each other instead of looping through them sequentially, and cutting validation loops that aren’t earning their keep.

A fast system isn’t the one running the fastest model. It’s the one doing the least unnecessary work.

Bottleneck #4: Agent design

One of the more persistent misconceptions between the AI community is that more agents make a better system. Usually what they make is a more expensive and harder to debug. Every additional agent adds more prompts, more coordination, more latency, more tokens, and more places for the whole thing to fail quietly.

Production benchmarks put independent multi-agent setups at roughly 58% extra token overhead compared to a well-tuned single agent doing the same task, and centralized orchestrator topologies at closer to 285% extra for something a single agent handles just as well. Teams that start with 3 to 5 agents and scale only when the data supports it consistently outperform teams that launch with 10 or more.

One of the good practice would be to start with one well-scoped agent. Add a second only when there’s a specific, earned reason: the work genuinely decomposes into parallel sub-tasks, different sub-tasks need different tools or models, or two parts of the workflow need different permission boundaries — an agent that drafts a refund and a separate one that’s actually allowed to issue it is a legitimate split. “It felt like it should be multi-agent” is not a specific, earned reason.

Bottleneck #5: Infrastructure

Eventually you do have to pick a provider — OpenAI, Anthropic, Google, Groq, Fireworks, self-hosted, or some mix. By this point the model itself is often the smaller part of the decision. What actually matters more: rate limits, regional availability, SDK maturity, the pricing structure (cached input pricing and batch discounts can move your real cost more than the headline per-token price), observability, and uptime.

The decision framework

Fig 2 — Diagnose the symptom before you touch the model.

Whenever you’re tempted to ask “which LLM should I use,” ask this instead: what’s actually limiting this system right now?

If the answers are wrong, inspect retrieval. If they’re slow, inspect orchestration. If costs are high, inspect routing. If retrieval itself is poor, inspect embeddings. Only once those check out should you spend time comparing models — and even then, on your own workload, not a public leaderboard.

AI engineering has changed

A few years ago, success in AI meant building better models. Today, it means building better systems. Tomorrow, the best AI engineers won’t be defined by the models they use, but by their ability to diagnose, optimize, and ship reliable AI applications.

Sources