Semantic Caching and Live Internet Search: Tools That Let AI Cache Answers to Save Money, and Pull Data From the Web When It Needs To

Posted on Oct 30 • Originally published at next4i.com

Say you open the logs of your chat system and they are full of questions like these:

All three mean the same thing. But the system sent each one to the AI, had it write a fresh answer three times, and paid for the tokens three times. Multiply that by thousands or tens of thousands of users, and the API bill gets... sigh.

At the same time, there is another kind of question the AI cannot answer from memory at all: "What's the gold price today?", "Is it going to rain today?", or "What promotions are on this month?" These answers change every day, and the data a model was trained on stops at a certain date, its knowledge cutoff. Anything new after that date, the model simply does not know.

This article covers two techniques, one for each side. Semantic caching lets an AI remember answers to questions that keep coming back with the same meaning, so you stop paying for them again. Live internet search lets the AI pull fresh data from the web, or a search engine, when a question needs it. And we will look at how to decide which questions should be "remembered" and which should be "looked up again."

A call-center agent who has been on the job for a while knows customers ask the same things over and over, so they write the answers down in a notebook. When someone calls and asks "I lost my card, what do I do?", they open the notebook and answer right away, without digging through the policy manual again. The idea here is pretty much the same.

A cache is a place where you keep results you have already worked out, so next time you can grab them instead of doing the work again. Websites and apps have used this technique for a long time.

With AI it pays off even more, because every time a model writes an answer, you pay per token and wait several seconds. Every word you send in and every word that comes back costs money, and output tokens cost more than input. If you want the background, I wrote an earlier article on tokens.

When you see "10x cheaper," part of it usually comes from that 0.1x price on prompt caching. But that discount only covers repeated input. It is not 10x off the whole bill.

The clearest win: on a hit, you pay nothing for the model and you do not wait for it, so users get their answer much faster. Turning the question into an embedding and searching the cache cost far less than having the AI answer.

To get a feel for it, let's do some simple math. If we ignore the cost of embeddings and the cache system itself, your AI cost comes down to the questions that miss.

To get 10x, about 90% of requests need to hit. That means almost every question has to repeat the meaning of one you have seen before, like a bot that answers FAQs.

The GPT Semantic Cache paper (Regmi and Pun, 2024) tested repetitive, customer-service-style questions. It got cache hit rates of about 61.6% to 68.8%, cut API calls by up to 68.8%, and over 97% of the answers served from cache were correct, by the authors' own measurement. Run that through the table above and it works out to roughly 3x cheaper. (More in the references at the end.)

"How do I cancel my credit card?" and "How do I cancel my debit card?" differ by one word. Their embeddings may be very close, but the answers may be completely different. The same goes for sentences that differ only slightly, by a number, a date, or even a different context.

Redis's documentation says it plainly: semantic caching does not guarantee that a hit is always correct. The vCache paper (Schroeder et al., 2025) makes the same point from another angle: using one fixed similarity threshold for every question does not guarantee correctness either.

To lower the risk, start with a strict threshold, then loosen it step by step, and spot-check the answers that hit the cache along the way. Test with real questions in the languages your users actually write in, because every embedding model understands each language a bit differently.

Some things have already changed, a promotion for example, but the cache keeps serving the old answer. The fix is to give each answer an expiry time (TTL) that matches how often the source data changes. Redis gives the example that real-time data like stock prices or weather should expire within 15 to 30 minutes, while FAQs that rarely change can be kept for days or weeks. And when a source document gets edited, clear the related cache entries right away.

This one is the most serious. If a customer asks "What's my outstanding balance?" and the answer gets saved into a shared cache, the next customer who asks something similar could get the first customer's answer. That is bad. Redis gives a multi-tenant example too: Company A's policy could end up answering Company B if the cache is not separated.

There is also research (Song et al., 2024) showing that by watching for responses that come back unusually fast, because they hit the cache, someone may be able to guess what other people asked. So a cache shared across users is a privacy issue as well, not only an accuracy issue.

The fix is to separate the cache by user, organization, or permission level, and to not cache answers built from personal data at all unless you really need to.

Requests like "Help me come up with a caption" or "Give me name ideas for my shop" should get a fresh answer each time. Serving a repeat from the cache feels weird. GPTCache itself skips the cache more often when the temperature is set high.

The same question asked at a different time or in a different context may need a different answer. So the cache key should be tied to its context, for example:

If any one of these changes, the old answer may no longer be up to date.

Even with a RAG system that can search your company's documents (I wrote an article explaining RAG, if you want to go back to it), some questions still need the latest data: news, prices, newly issued regulations, or software versions. That information is neither inside the model nor in your internal documents.

A typical live internet search flow looks like this:

There are mostly two ways to do it: 1. use the web search tool your model provider builds in, or 2. use a separate search API service and wire up the steps yourself.

To lower the risk, you can restrict domains for topics that need accuracy, for example only searching official sites or a whitelist of trusted ones, and show sources users can check. Cap the number of searches per question. And do not give an AI that reads the web permission to do important things at the same time, because it may end up running or doing something unsafe.

Put a router in front that sorts the questions first.

Semantic caching really can lower costs and make AI answer faster. But caching has to be checked carefully and tightly. You need to weigh the benefit against the risk for each kind of question. If the AI gives a wrong answer without having any idea it is wrong, the savings may not be worth it compared with simply calling the LLM or the API again.

Live internet search lets an AI look things up to answer current questions, but it costs both money and time. And the key point is that we cannot control anything about content that comes from the web, so we need a way to handle that too.

Once you have a solid step that sorts questions into "cache" or "search again," your AI starts working like a great call-center agent: quick to answer, remembers what rarely changes, checks or looks up new information when it makes sense, answers accurately, does not leak data, stays safe, and saves money. If you can get all of that, that is the complete system everyone wants.

Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i-devto-en

Originally published at: https://www.next4i.com/dev-notes/en/semantic-caching-and-live-internet-search-dev-en