Why a Search API Is Often the Wrong Abstraction for AI Agents

If you've built an AI agent that uses web search, you've probably run into this. You ask it a simple question like so: "What do these three project management tools charge for their team plans today?"

The search tool returns ten links for each. Great. But now what?

I ran into this problem while building agents that need to work with information from across the web. Search was rarely the difficult part. Everything that came after it was. And that got me looking for a better way to handle web research. That is when I came across Olostep, an API platform that offers more than just search.

In this article, I'll look at why search alone often falls short, how Olostep approaches the problem, and how to use its APIs to build a more complete web research workflow with Python. I'll also cover when a plain search tool is still all you need.

Let us circle back to the pricing question from the intro. For each of the three tools, the agent has five jobs to finish after the search returns:

Search has only given the agent candidate URLs. It still has fifteen pieces of work to complete across the three tools, and without additional web capabilities, each step becomes something you have to implement yourself.

Think of a search API as a library card catalog. It tells you which shelf holds the book. Someone still has to walk to the shelf, open the book and copy out the table. A person does that without noticing. An agent needs each step written down.

The following properties of search results cause most of the trouble for agents.

A snippet is a sentence or two. Pricing lives in tables, feature matrices and footnotes, and a snippet cuts all of those. The model fills the gap with a guess, and the guess reads like a fact.

Web search ranks pages by what a person is likely to click. A page that reads well to a person can be hard for code to parse, for example when prices render only after a tab click. The best source for your agent may sit on page two, and an agent that reads the top three results never sees it.

A person reads "$12 per seat per month, billed annually" and moves on. Code needs:

{ "plan": "Team", "price_per_seat_usd": 12, "billing": "annual" } Enter fullscreen mode Exit fullscreen mode Going from the first form to the second is an extraction job, and search leaves it to you.

A search call returns links with no connection between a claim and the page behind it. But how does the agent know the number it quoted came from the page it read? It has no way to know. Verification needs a step that keeps the claim and its source together.

TL; DR: You need your agents to go from search results to a reliable structured output that likely looks like so:

Now that you've understood the problem, you can think about web research as more than just finding links. An agent needs to move through a few distinct stages to get from a question to an answer it can rely on.

These stages can be broken down into four parts:

Find: Discover relevant pages and sources that might contain the information the agent needs. This could mean searching the web for a topic or discovering URLs across a domain you already know.

Fetch: Retrieve the actual content of those pages. Search results only give you a preview; the agent needs the full page, including content that might be buried in the page or rendered dynamically.

Extract: Turn that content into structured information the agent can work with. For example, instead of returning a paragraph about pricing, extract the plan name, price per seat and billing period as individual fields.

Verify: Check the extracted information against its source. The agent should be able to trace an answer back to the page it came from and recognize when the information cannot be verified.

This is also how Olostep approaches working with the web. Rather than treating search as the entire workflow, it provides APIs that address each of these stages. Here is how these stages map to Olostep's APIs:

Searches, scrapes, maps and answers return a result in the same call. Batches and crawls run as asynchronous jobs: you create the job, check its status, then list and retrieve the results. A search call can also scrape each result in the same request through scrape_options. The examples below keep the jobs separate so each one is visible.

URL discovery is useful when you already know the site and need its pages. Search helps when you do not know the site yet. Both belong to the "find" job.

Let us try to build this simple workflow.

We'll start by searching for a project-management tool's pricing page, then fetch the page and extract its plan details in the same scrape call.

First, install the SDK and set your API key:

pip install olostep export OLOSTEP_API_KEY = "your-api-key" Enter fullscreen mode Exit fullscreen mode For this example, we'll search for project management software pricing pages and use the first result as our candidate URL. In practice, you'd evaluate the returned URLs to identify the most relevant pricing page before scraping it.

import os from olostep import LLMExtract, Olostep # Initialize the client with your API key client = Olostep ( api_key = os. environ [ " OLOSTEP_API_KEY " ]) # 1. Search for candidate sources search = client. searches. create ( query = " team plan pricing project management software " ) # Extract the URL from the first link dictionary in the list first_url = search. links [ 0 ][ " url " ] # 2. Fetch and extract structured data in one call page = client. scrapes. create ( url_to_scrape = first_url, formats = [ " json " ], llm_extract = LLMExtract ( schema = { " type ": " object ", " properties ": { " plans ": { " type ": " array ", " items ": { " type ": " object ", " properties ": { " name ": { " type ": " string " }, " price_per_seat_usd ": { " type ": " number " }, " billing ": { " type ": " string " }, }, }, } }, } ), ) # 3. page.json_content is a pre-parsed dictionary in the Python SDK plans = page. json_content. get ( " plans ", []) # Iterate and safely print extracted values for plan in plans: print ( plan. get ( " name " ), plan. get ( " price_per_seat_usd " ), plan. get ( " billing " )) Enter fullscreen mode Exit fullscreen mode Let's parse what each part does:

Each LLM extraction call costs 10 credits per scrape according to the documentation, so check the current pricing before running it across a large number of URLs.

The fields you get back depend on the page. Try the script against a pricing page you know well and compare the output with what you see in the browser. That check takes two minutes and tells you whether the schema matches the page.

Having structured data doesn't automatically mean it's correct. A page might have changed, a price might be missing, or the extraction might have picked up the wrong value.

This is where Olostep's Answers API comes in handy. It lets you ask a question and receive a response grounded in live web sources, giving you another way to check the information your agent extracted:

reply = client. answers. create ( task = f " What does the team plan on { first_url } cost per seat per month? ", json_format = { " price_per_seat_usd ": "" }, ) print ( reply. json_content ) print ( reply. sources ) Enter fullscreen mode Exit fullscreen mode json_format describes the fields you want returned, with empty values as placeholders. The response includes sources, allowing you to inspect the URLs behind the answer. If a field cannot be verified, the API returns NOT_FOUND.

You can compare this result with the price extracted earlier. If they agree, you have additional evidence supporting the result. If they disagree, the agent has a reason to investigate before responding.

The example above handles one URL. But what if you need to run the same workflow across hundreds of pages?

Olostep supports asynchronous batch scraping, allowing you to submit multiple URLs in a single job. The API handles the request in the background and returns a unique retrieve_id for each page. You can poll the endpoint or configure webhooks to fetch the markdown data once processing completes.

The single-page example above gives you the basic pattern. Batch processing lets you apply it at a larger scale without having to manage every scrape individually.

None of this means you should stop using plain search APIs. Search is still the right tool for many tasks, particularly when you only need to discover relevant pages or retrieve a quick fact.

The distinction is whether your agent needs to do anything with the page after finding it.

Use the stages/jobs approach outlined here when:

If the agent's next step is to choose a page, search is enough. If the next step is to read, process or extract information from a page, it needs more than search.

Search is a useful starting point, but finding a page and getting an answer from it are quite different problems.

An agent that works with web information needs access to page content, a way to extract structured information, and a way to ground its answers in sources. And Olostep helps with its APIs for search, scraping, extraction, URL discovery and grounded answers.

The important shift, therefore, is to stop thinking of web search as the entire workflow. For many agentic tasks, it is just where the workflow begins. So if you're building agents that need to do more than find links, try exploring Olostep.