Model routing, semantic caching, prompt compression, and better prompts: what each one saves, where it goes wrong, and how to test it before it reaches users.

The quickest way to cut agent spend is to swap every call to a smaller model. The quickest way to lose a customer is the same move. Cost work only counts if quality holds, so each lever below comes with a way to check that it did.

Before touching anything, get an eval set. Even fifty real inputs with known good outputs is enough to catch the worst regressions. Without it you are guessing, and every change below becomes a gamble.

Route by task, not by habit

Most teams pick one strong model at the start and never revisit it. That model is doing a lot of work that a cheaper one could handle: classification, field extraction, short rewrites, yes or no checks. Multi-step reasoning and long-form writing are where the larger model earns its price.

A router picks the model per request. Start simple. Tag each step of your agent with a task type and map task types to models by hand. Only move to a learned router once the manual map is stable and you can measure it.

Semantic caching

Support agents answer the same ten questions all day. A semantic cache stores the answer and returns it when a new question means the same thing, even if the wording differs. A hit costs almost nothing and comes back in milliseconds.

Caches fail in quiet ways, so set these up on day one:

• Scope entries by tenant and permission level. A cached answer must never cross a customer boundary.
• Tune the similarity threshold per route. A returns-policy answer can tolerate looser matching than a pricing or legal one.
• Expire entries when the source changes. A cached answer about last quarter's pricing is worse than no answer.
• Log hit rate and spot-check hits. A high hit rate with falling user ratings means the threshold is too loose.

Prompt compression

Look at what your biggest prompts actually contain. Usually it is retrieved documents and chat history, and a lot of it repeats or does not matter to the answer. Compression drops the low-value tokens and keeps the ones the model leans on. Research such as LLMLingua showed large reductions on long contexts with small quality loss, though results depend heavily on your data.

The cheaper trick comes first: retrieve less. Cutting top-k from ten chunks to four often saves more than any compression model, and it is easier to reason about.

Fix the prompt before buying a bigger model

When an agent fails, the reflex is to upgrade the model. Read ten failed traces first. You will often find an ambiguous instruction, a missing output format, or an example that teaches the wrong thing. Fixing those costs nothing per request and can remove whole retry loops.

Stop the requests that should not run

Some of the most expensive calls are the ones nobody wanted: a retry storm, a tool that loops, a prompt injection that sends the agent off task, a run that kept going after the budget was gone. Limits on steps, retries, and spend per run catch these before they reach the invoice.

What each lever costs you

In RAW all of these run in the Optimise layer and write to the same trace, so you compare cost and accuracy on the same runs instead of two dashboards that disagree.

FAQ