Building AI Website Audits with Python: 5 Lessons About Reliability

Building an automated website audit sounds straightforward: fetch a page, run checks, and generate a report.

But creating an audit that developers can actually trust is a different challenge.

While working with Python, FastAPI, Playwright, and automated website analysis, I’ve been exploring an important question:

How do we distinguish a real website problem from something our audit simply couldn’t verify?

Here are five engineering principles that make automated audits more reliable.

A common mistake is treating every unsuccessful check as a website failure.

For example, a request timeout does not prove that a website is broken.

This distinction prevents unreliable conclusions.

HTTP response codes, HTML elements, redirects, and response headers can be checked using conventional Python code.

import requests def check_robots_txt(domain: str): url = f"https://{domain}/robots.txt" try: response = requests.get(url, timeout=10) except requests.RequestException: return {"status": "not_tested"} if response.status_code == 200: return {"status": "pass"} if response.status_code == 404: return {"status": "fail"} return {"status": "not_tested"}

This example only checks whether robots.txt is accessible. It does not validate the file’s content.

For subjective assessments, LLMs can help interpret evidence, but they should not invent missing observations.

A useful finding should answer three questions:

An audit should identify a specific, verifiable condition and explain its relevance.

A system can appear highly accurate when developers repeatedly test it against the same websites.

This creates the risk of optimizing for known examples rather than general reliability.

A better approach is to test against previously unseen websites, classify false positives, and document the results before changing the implementation.

A trustworthy audit does not need to produce an answer for every check.

Not enough evidence to determine the result.

That is more useful than a confident but unsupported warning.

The challenge in AI-assisted auditing is not generating more findings.

It is making sure those findings are reproducible, explainable, and supported by evidence.

Reliable automation should make uncertainty visible rather than hide it behind a score.

I’m Ozan Orhan, an AI Automation Engineer based in Hannover, Germany. I work with Python, AI agents, APIs, RAG, MCP, and workflow automation.

My technical projects and case studies: ozan-orhan.com