Audit llms.txt and crawler rules without predicting AI citations
A website can publish llms.txt and still never appear in a particular AI answer. It can also lack the file while exposing useful public content. Treating one policy-file check as a visibility score collapses several separate questions: what the site publishes, what its root crawler rules say, what
A website can publish llms.txt and still never appear in a particular AI answer. It can also lack the file while exposing useful public content. Treating one policy-file check as a visibility score collapses several separate questions: what the site publishes, what its root crawler rules say, what a service retrieves, and what an answer engine later cites.
llms.txt Auditor build 0.3.2 addresses the first two questions. It checks llms.txt, llms-full.txt, and root robots.txt for a submitted public origin. It returns observed policy evidence and explicit gaps, rather than a promise of indexing, training exclusion, or answer-engine placement.
{ "items": [ "apify.com", "nytimes.com" ], "maxConcurrency": 2 } Enter fullscreen mode Exit fullscreen mode These values demonstrate input structure; the article does not assert their current policy states. The public schema requires one to 100 nonempty domain or website strings. Inputs normalize to public origins, so choose the origin whose policy you actually intend to review rather than expecting a deep page path to become a separate site policy.
The concurrency setting accepts one to fifty. A low value is a sensible starting point when the purpose is a carefully reviewed policy inventory. Increasing concurrency changes parallel collection, not the meaning or completeness of a particular origin's evidence.
A definitive HTTP 404 or 410 supports an observed missing-file state at the time of the check. Timeout, DNS failure, access denial, rate limiting, server failure, and truncated responses are different. They make evidence incomplete. Converting all of them to missing would create false remediation tasks and hide collection problems.
The installed fetch is bounded by a fifteen-second timeout and 300 KB response limit. Preserve those operational outcomes in the audit table. When a later attempt succeeds, keep the old record as an earlier observation rather than rewriting its uncertainty into a historical fact.
For root crawler policy, the curated bot set includes GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, Google-Extended, PerplexityBot, CCBot, Bytespider, and Amazonbot. The result distinguishes blocked, allowed, and unspecified root policy. An unspecified state should remain unspecified; it is not evidence that every crawler will retrieve every path.
A useful handoff includes the origin, checked path, observation time, response state, relevant policy evidence, and the narrow action being proposed. A missing llms.txt might become an editorial decision. An unreachable robots.txt may require an operational investigation. A root restriction should be reviewed in the context of the site's actual policy goals.
The audit does not prove what a crawler has already visited or retained. It also does not measure brand mentions, citation frequency, referral traffic, or conversions. Measure those outcomes through separate observations with their own time periods and source definitions. Keeping that separation prevents a configuration improvement from being reported as a marketing result.
For acceptance, use controlled fixtures covering a present file, a definitive 404, a forbidden response, and a rate-limited response. Verify that missing and incomplete remain different. Inspect a root policy with no matching bot directive and confirm that unspecified does not become an unconditional allow. These are proposed checks, not a report of a new runtime test.
Finally, preserve the run-level OUTPUT reconciliation when using the export in a dashboard. A partial audit should not look identical to a complete domain audit after a chart drops nulls. The product is useful when it gives an operator a concrete next investigation, with enough evidence to understand why the item was routed there.
The live FREE-tier event prices checked on 20 October 2026 are $0.01 per result-found event and $0.005 per automatic Actor-start event. The start event is memory-scaled: one event per GB, with a minimum of one event. A free result outcome therefore does not make an entire run free. Other account tiers have their own rates; check the live pricing panel before execution. These prices describe the installed contract, not measured customer revenue or a new runtime test.
Inspect the Actor, input, and current pricing.
AI-assisted drafting; technical statements reviewed against the installed Actor source and current API metadata. Examples are illustrative unless explicitly identified as saved observations.