Prepare web content for Amazon Bedrock Knowledge Bases with Crawl4AI

When you build Retrieval Augmented Generation (RAG) applications, answer quality depends on the data that you ingest. Amazon Bedrock Knowledge Bases is the fully managed RAG capability of Amazon Bedrock. With Amazon Bedrock Managed Knowledge Base, you connect your data sources, and Amazon Bedrock manages ingestion, storage, indexing, and retrieval for you. You don’t provision or scale a vector store, and embedding and reranking use service-managed models by default.

Some sources need more control over what reaches the knowledge base. You might want to index only the main content region of each page, remove navigation and other boilerplate, or attach your own metadata attributes to every document. For that class of source material, you can extend the reach of your knowledge base with a preparation step built with Crawl4AI, an open source Python crawler. The step writes Markdown and a metadata sidecar for each page to Amazon Simple Storage Service (Amazon S3), and the knowledge base ingests both through its Amazon S3 data source.

In this post, we show you how to choose the appropriate web ingestion path and implement custom preparation for content that needs additional processing. You then start ingestion into a managed knowledge base and validate retrieval quality. The companion aws-samples repository provides the complete workflow as a sample implementation.

This is batch ingestion of known web properties, not real-time web access at inference time such as Web Grounding with Amazon Nova.

Start with the Amazon Bedrock Web Crawler when it can reach and index the pages you need. It respects robots.txt, and after the first full crawl, each sync captures only added, updated, and deleted content.

Use the Crawl4AI preparation path when public source material needs one or more of the following before ingestion:

Because managed and customer-managed knowledge bases both support Amazon S3 data sources, the Crawl4AI path works with either type. This post uses a managed knowledge base and notes where a customer-managed knowledge base differs.

The rest of this post follows the companion sample. Treat its architecture and defaults as a starting point, not as requirements of Amazon Bedrock Knowledge Bases.

By default, the sample runs the AWS Fargate task in a public subnet, where it reaches your configured websites through an internet gateway. Crawl4AI’s role ends once it has prepared the source documents and metadata sidecars. From ingestion onward, Amazon Bedrock Knowledge Bases handles parsing, chunking, embedding, indexing, and retrieval. The following diagram shows the Crawl4AI path from the schedule through to retrieval.

Figure 1: The Crawl4AI-to-Amazon S3 ingestion path for Amazon Bedrock Knowledge Bases

The numbered steps in the diagram map to the following workflow:

The task reads its non-secret pipeline configuration from AWS Systems Manager Parameter Store at startup.

The workflow processes public pages only and doesn’t implement authenticated sessions. For sites that require sign-in, evaluate the Amazon Bedrock Web Crawler authentication options first, which include basic, form-based, and SAML authentication. Extending the sample with authentication requires session handling, credentials in AWS Secrets Manager, and a security review.

The AWS Cloud Development Kit (AWS CDK) app packages the implementation in a container image during deployment. When the schedule fires, the container runs scripts/container_entrypoint.py and executes crawl, upload, and ingestion in sequence. The following three steps explain the core logic.

The crawl step performs bounded crawling and writes a Markdown file with a metadata sidecar for each successfully prepared page. The following excerpt is simplified. The module src/crawl4ai_bedrock_kb/crawl.py contains the complete seed pacing, URL filtering and scoping, page limits, and redirect checks:

async def crawl_and_prepare(config: PipelineConfig, output_dir: str) -> list[str]: written, seen = [], set() pages_crawled = 0 browser_config = BrowserConfig(headless=True, java_script_enabled=True) async with AsyncWebCrawler(config=browser_config) as crawler: for seed in config.seed_urls: remaining = config.crawl.max_pages - pages_crawled if remaining <= 0: break strategy = BFSDeepCrawlStrategy( max_depth=config.crawl.max_depth, max_pages=remaining, filter_chain=FilterChain([ SeedScopeFilter(seed, config.crawl.scope), # Optional include/exclude URLPatternFilter objects are added here. ]), ) run_config = CrawlerRunConfig( deep_crawl_strategy=strategy, markdown_generator=DefaultMarkdownGenerator( content_filter=PruningContentFilter() ), css_selector=config.crawl.css_selector, # Optional page region. check_robots_txt=config.crawl.check_robots_txt, mean_delay=config.crawl.request_delay_seconds, max_range=0, semaphore_count=1, ) results = await crawler.arun(url=seed, config=run_config) for result in results: if pages_crawled >= config.crawl.max_pages: break pages_crawled += 1 requested_url = result.url source_url = getattr(result, "redirected_url", None) or requested_url if not SeedScopeFilter(seed, config.crawl.scope).apply(source_url): continue # Recheck the effective destination after redirects. canonical = _canonical_url(source_url) if canonical in seen: continue seen.add(canonical) if not result.success or not result.markdown: continue md_result = result.markdown markdown = ( getattr(md_result, "fit_markdown", None) or getattr(md_result, "raw_markdown", None) or (md_result if isinstance(md_result, str) else "") ) if not markdown: continue # The complete implementation adds title, crawl timestamps, content # type, and configured custom attributes to this metadata. written.extend( _write_document( output_dir, _base_name(source_url), markdown, {"source_url": source_url}, ) ) return written Amazon Bedrock Knowledge Bases supports several document formats for Amazon S3 data sources, including plain text, HTML, Markdown, PDF, and DOCX. The workflow writes Markdown because it preserves headings, lists, and tables without carrying all raw HTML into ingestion.

The upload step sends each prepared Markdown document and its matching metadata sidecar to the configured S3 bucket, under the workflow’s s3Prefix. For page.md, the sidecar must be named page.md.metadata.json, the complete source filename with.metadata.json appended. The workflow stores each sidecar in the same S3 location as its document. Amazon Bedrock limits each metadata sidecar to 10 KB. If a sidecar would exceed that limit, the run stops with an error before it writes either file.

Each attribute in the sidecar uses the typed format that both managed and customer-managed knowledge bases accept for Amazon S3 data sources. The following abbreviated sidecar, formatted for readability, shows three of the attributes that the workflow writes:

{ "metadataAttributes": { "source_url": { "value": {"type": "STRING", "stringValue": "https://www.example.com/services/permits"} }, "crawled_at_epoch": { "value": {"type": "NUMBER", "numberValue": 1790640000} }, "content_category": { "value": {"type": "STRING", "stringValue": "web"} } } } The pipeline uploads only the paths produced by the current crawl, so stale local files are not sent.

After content is in S3, an ingestion job hands each document to the data source’s ingestion pipeline. By default, a managed knowledge base parses each file with its managed parser, which automatically selects a parsing approach for your content. You select the chunking strategy when you connect the data source. You can’t change it afterward, so evaluate the options before you create a production data source. A managed knowledge base offers the following chunking strategies:

Chunk boundaries and retrieval results depend on the content and the selected strategy, so compare the options with representative pages and evaluation questions. To change strategies later, create a new data source. For more information, see Customize ingestion for a data source. A customer-managed knowledge base also offers semantic and hierarchical chunking and additional parsing options. For details, see How content chunking works for knowledge bases.

The workflow then starts an ingestion job so the knowledge base processes the new documents. The following simplified excerpt from src/crawl4ai_bedrock_kb/ingest.py shows the call:

def start_ingestion_job(config: PipelineConfig) -> str: # Start an Amazon Bedrock knowledge base ingestion job and return its ID. client = boto3.client("bedrock-agent", region_name=config.region) response = client.start_ingestion_job( knowledgeBaseId=config.knowledge_base_id, dataSourceId=config.data_source_id, ) return response["ingestionJob"]["ingestionJobId"] The same StartIngestionJob and GetIngestionJob calls work for both knowledge base types, and you can also monitor progress on the console.

In this section, you deploy the workflow into your AWS account with AWS CDK. Only the knowledge base prerequisites differ between the two knowledge base types. When you’re finished, follow the steps in the Clean up section to remove the resources.

Before you start, make sure that you have:

You need the workstation only to build and deploy. After deployment, the schedule runs the task in AWS.

Clone the companion repository and install the deployment dependencies:

git clone https://github.com/aws-samples/sample-crawl4ai-bedrock-knowledge-base.git cd sample-crawl4ai-bedrock-knowledge-base/deploy python -m venv.venv source.venv/bin/activate # Windows PowerShell:.venv\Scripts\Activate.ps1 python -m pip install -r requirements.txt Keep Docker running. Set AWS_DEFAULT_REGION to your knowledge base’s Region first, because cdk deploy uses the Region from your environment or AWS profile, not the one in the bootstrap command. Bootstrap the target account and Region once, if needed. Then deploy the stack with your knowledge base, data source, S3 bucket, and authorized seed URLs:

export AWS_DEFAULT_REGION=REGION # Windows PowerShell: $env:AWS_DEFAULT_REGION = "REGION" cdk bootstrap aws://ACCOUNT_ID/REGION cdk deploy \ -c knowledgeBaseId=ABCDEFGHIJ \ -c dataSourceId=KLMNOPQRST \ -c s3Bucket=amzn-s3-demo-bucket \ -c 'seedUrls=["https://www.example.com/services"]' \ -c maxDepth=1 \ -c maxPages=25 Replace every example value, including ACCOUNT_ID, REGION, and the four required context keys, with your own. The example values are syntactically valid, so an unedited command deploys a stack that points at a bucket you don’t own. You can also set these keys in cdk.json.

The maxDepth and maxPages values limit the first crawl to one link level and 25 pages. Raise them after you review the prepared output. To preview which pages a crawl prepares, you can also run the pipeline locally with --dry-run, as described in the README. The stack name is fixed, so another cdk deploy in the same account and Region updates this stack.

The sample builds an ARM64 container image by default. If Docker on your x86_64 workstation can’t build ARM64 images, add -c cpuArchitecture=X86_64 to the cdk deploy command.

If your account turns on VPC Block Public Access, the task can’t reach the internet from the default public subnet. To check the mode, run aws ec2 describe-vpc-block-public-access-options. In ingress-only mode, deploy with -c egressMode=nat, which adds a NAT gateway with hourly charges. Bidirectional mode also needs an exclusion, as the deployment guide describes.

The stack doesn’t create or change your knowledge base or data source.

With the default rate(24 hours) schedule, the first run starts when the deployment creates the Amazon EventBridge rule. To follow that run, use the command in the TailLogsCommand output, which cdk deploy prints when it finishes. A successful run logs job_status: COMPLETE. To start another run outside the schedule, use the command in the RunTaskCommand output after the current run finishes. To list the outputs later, run aws cloudformation describe-stacks --stack-name Crawl4aiBedrockKbCrawler --region REGION --query "Stacks.Outputs".

The deployment guide covers the remaining configuration, including crawl boundaries, the schedule, networking, container architecture, monitoring, and on-demand runs.

After ingestion, test representative questions with Retrieve before you add generation. A managed knowledge base uses hybrid search. When it uses the default managed embedding model, it also reranks results with its managed reranker. You can narrow results with the metadata from the sidecars. The sample’s default configuration adds content_category set to web to every page. The following example returns up to five chunks from those pages that were crawled in the last seven days. It requires the AWS SDK for Python (Boto3) 1.43.32 or later, which you can install with python -m pip install "boto3>=1.43.32". Replace the Region and knowledge base ID with your own values:

import time import boto3 client = boto3.client("bedrock-agent-runtime", region_name="us-east-1") week_ago = int(time.time()) - 7 * 24 * 60 * 60 response = client.retrieve( knowledgeBaseId="ABCDEFGHIJ", retrievalQuery={"text": "How do I apply for a building permit?"}, retrievalConfiguration={ "managedSearchConfiguration": { "numberOfResults": 5, "filter": { "andAll": [ {"equals": {"key": "content_category", "value": "web"}}, {"greaterThan": {"key": "crawled_at_epoch", "value": week_ago}}, ] }, } }, ) for result in response["retrievalResults"]: print(result.get("metadata", {}).get("source_url"), result["content"]["text"][:200]) The example passes an integer timestamp, which both knowledge base types accept in number filters. A managed knowledge base doesn’t support the startsWith and stringContains filters. For a customer-managed knowledge base, pass the same filter in vectorSearchConfiguration instead of managedSearchConfiguration.

The operators it supports depend on its vector store.

Metadata filtering can improve retrieval precision, but it is not an authorization boundary. If different users should see different pages, the Amazon S3 connector for managed knowledge bases supports document-level access control with access control list (ACL) files that you provide. The sample doesn’t create ACL files.

To generate responses from a managed knowledge base, use agentic retrieval. The AgenticRetrieveStream operation returns the retrieved results and a generated response with citations to those results. For a customer-managed knowledge base, use RetrieveAndGenerate. In both cases, each cited result includes the source_url metadata from its sidecar, which your application can present as the user-facing source.

Evaluate retrieval relevance, source-page coverage, and citation quality whenever you change content preparation, chunking, or retrieval settings. For the retrieval settings that you can tune, see Configure and customize queries for managed knowledge bases.

Each scheduled event launches one standalone Fargate task. The stack doesn’t prevent overlapping runs, so choose an interval longer than the expected crawl and ingestion duration. A failed run isn’t announced, because the sample adds no retry policy, dead-letter queue, or alarm. An ingestion job can also report COMPLETE when some documents fail to ingest, and the task then exits with code 0. The task logs the job’s failure reasons. Add the monitoring you need, including an alarm on those failure reasons, before you rely on the schedule.

Each run incurs Fargate charges for the configured task size, which defaults to 2 vCPU and 4 GB. The task keeps running while it polls the ingestion job for up to 30 minutes, so a slow ingestion job extends billable task time. If ingestion takes longer, the task exits with a nonzero code, and the ingestion job keeps running.

The stack turns on Container Insights for the cluster, and flow logs for an Amazon Virtual Private Cloud (Amazon VPC) that it creates. Both add Amazon CloudWatch charges. The stack’s AWS KMS key has a monthly charge, and the task’s public IPv4 address is billed while the task runs.

With egressMode=nat or privateEndpoints=true, the NAT gateway and interface endpoints are billed hourly, even when no task is running. Knowledge base usage is billed separately. For details, see Amazon VPC pricing and Amazon Bedrock pricing.

Uploads are additive: removing a URL or missing a page during a later crawl doesn’t delete its existing S3 objects. To remove stale content, delete the obsolete document and matching sidecar under the configured prefix, then start ingestion again. If you turn on the document deletion safeguard (deletion protection) for a managed knowledge base data source, check its threshold. A sync that would delete more than that percentage of documents skips its delete phase, so those documents stay indexed. To start ingestion automatically whenever objects in the bucket change, see Build and deploy an automatic sync solution for Amazon Bedrock Knowledge Bases.

Retrieved passages become the context that a foundation model uses to generate a response. Apply the following controls before you run this workflow in production:

Review the following considerations before you run this workflow against production content:

To avoid ongoing charges, delete the resources created during this walkthrough:

In this post, you learned how to extend Amazon Bedrock Knowledge Bases to authorized public web content that needs preparation before ingestion. A scheduled Fargate task uses Crawl4AI for bounded crawling, prepares Markdown with page.md.metadata.json sidecars, uploads them to an Amazon S3 data source, and starts ingestion. The knowledge base then handles parsing, chunking, embeddings, indexing, retrieval, and grounded responses with citations.

Start with the Amazon Bedrock Web Crawler in a managed knowledge base for most public websites. Use the Crawl4AI path when you need control over which part of each page is indexed or which metadata travels with it. To get started, deploy the scheduled Fargate workflow from the companion sample-crawl4ai-bedrock-knowledge-base repository, and then validate retrieval quality and citations with representative content before scaling.

For more information about the services and patterns in this post, refer to the following resources:

This product includes software developed by UncleCode as part of the Crawl4AI project.

Dax is a Solutions Architect at Amazon Web Services (AWS), where he works closely with customers and partners on key cloud engagements. He helps organizations build and migrate applications on AWS, with a focus on data processing and distributed computing solutions. Dax is an AI advocate who designs and delivers generative AI solutions on AWS, helping customers apply AWS AI services to automate workflows, improve operational efficiency, and accelerate decision-making at scale.

Bhavesh is a Principal Consultant at Amazon Web Services (AWS) with 25+ years of deep database, modernization, and technology leadership experience. He focuses on helping enterprise customers with AWS cloud adoption and modernizing legacy workloads, successfully delivering large-scale cloud transformation programs across multiple industries. Bhavesh is an AI advocate and leads the development of AI-powered solutions and agentic accelerators that solve complex customer use cases, leveraging AWS AI services to drive automation, operational efficiency, and intelligent decision-making at scale.

Abhishek is a Senior Delivery Consultant with AWS Professional Services, where he leads technical workstreams for enterprise cloud migration and modernization engagements. He specializes in defining DevOps strategy, architecting scalable delivery frameworks, and aligning cross-functional engineering teams around technical outcomes. His interests include generative AI, agentic architectures, and building reusable patterns that raise the delivery bar across customer and internal teams, enabling organizations to move faster while maintaining operational excellence.