Is Your Website Blocking AI Search Crawlers?
Your robots.txt file says an AI search crawler is welcome. Your firewall says absolutely not.
Both can be true.
A crawler may be allowed in robots.txt and rejected by a web application firewall. It may receive the page but miss the JavaScript-rendered content. It may reach a crawlable URL that is marked noindex. Or the team may have allowed a training crawler while blocking the separate bot used for search.
That is why an AI crawler audit should not begin with a copy-and-paste allow rule. It should begin with a business decision, then trace every control between the bot and a usable page.
First, decide what kind of access you actually want
“AI crawler” sounds like one job. It is not.
Platforms may use one bot to support search results, another to collect material that could contribute to model training, and another to retrieve a page after a person requests it. Those purposes carry different marketing, legal, security, and technical implications.
Google says eligibility for its generative Search features begins with normal Search requirements: a page must be crawlable, indexed, and eligible to display a snippet. There is no separate magic doorway labeled “AI Overviews.”
OpenAI
OpenAI documents OAI-SearchBot for search, GPTBot for potential model-training use, and ChatGPT-User for user-triggered visits. A site can support search discovery while signaling that its content should not be used for foundation-model training.
Anthropic
Anthropic documents separate controls for Claude-SearchBot, ClaudeBot, and Claude-User. Again: search, training, and user-directed retrieval are not one permission.
Perplexity
Perplexity distinguishes its indexing crawler from its user-requested fetcher. Its documentation also explains that the user-triggered fetcher may not behave like the indexing bot under robots.txt.
A blanket “block all AI bots” policy may remove visibility the business wanted. “Allow everything” may approve uses the business never discussed. Neither is a policy; both are shortcuts.
Robots.txt is one control, not the whole audit
Robots.txt tells compliant crawlers which URLs they may request. It is useful for crawler management, but it is not authentication, and it does not guarantee that a URL disappears from an index.
Google’s robots.txt documentation warns that a blocked URL may still be discovered through links and can appear without a description. Private content belongs behind real access controls. Robots.txt is a sign on the door, not the lock.
The file also has scope. A rule on the main hostname does not automatically govern every subdomain. Syntax, user-agent spelling, rule order, stale deployments, and conflicting files can all create surprises.
After any change, fetch the live file from every relevant hostname. The version in a ticket or code repository is not evidence that production got the memo.
The firewall may overrule the welcome mat
A content delivery network, web application firewall, bot-management product, hosting platform, or custom security rule can block a crawler that robots.txt allows.
Allowing a bot in robots.txt while challenging it at the edge is the technical equivalent of putting someone on the guest list and then locking the venue.
Perplexity recommends checking both its documented user-agent strings and published IP ranges when configuring Cloudflare or AWS WAF rules. OpenAI also publishes IP ranges for its bots. User-agent text alone is easy to imitate, so it should not serve as proof of identity when a security decision depends on who is requesting the page.
Cloudflare’s AI Crawl Control shows how operational this layer has become: it can report AI crawler activity, apply per-crawler policies, and monitor robots.txt compliance. Whether a site uses Cloudflare or another provider, the audit must inspect enforcement, not assume the public file describes reality.
Successful access does not mean the page is usable
A crawler can receive a 200 response and still miss the content that matters.
JavaScript may fail or load only after an interaction. A consent wall may cover the page. The server may return different content to an automated request. Canonical tags may point elsewhere. A noindex directive may prevent inclusion. Important facts may live in an image, interface, or application state that is absent from the initial HTML.
For Google, AI-search eligibility still depends on standard indexing and snippet rules, and JavaScript-heavy sites still need sound technical SEO. Google index status, however, cannot prove that OpenAI, Anthropic, or Perplexity can use the page. Different systems, different receipts.
This work belongs in ordinary SEO services as much as in an AI-search project. The difference is that the audit has to follow each platform’s own crawlers, not assume one allow rule covers them all.
How to run an AI crawler access audit
1. Set the business policy
Decide which search, training, and user-triggered uses the organization intends to allow for each platform. Marketing should not quietly make a legal decision, and security should not quietly erase a marketing decision.
2. Inventory current user agents
Use current first-party documentation. Bot names, purposes, versions, and IP ranges can change, and an old industry roundup is a poor source of truth for a live security rule.
3. Check every robots.txt file
Record the live response, status code, applicable rules, sitemap references, host scope, and conflicting directives for every relevant hostname.
4. Inspect CDN and WAF enforcement
Review managed-bot settings, custom rules, IP controls, rate limits, security events, and vendor-specific AI crawler features.
5. Test representative URLs
Include articles, product or service pages, JavaScript-heavy templates, PDFs, pagination, and pages affected by consent or geographic logic. Testing only the homepage is an excellent way to prove that the homepage exists.
6. Verify page-level controls and rendering
Review noindex directives, canonical tags, response headers, authentication, redirects, and the rendered content available to the requester.
7. Read server and edge logs
Confirm whether documented bots requested the site, which URLs they accessed, what status they received, and whether requests were challenged, rate-limited, or blocked.
8. Retest after changes
Allow for published platform-control delays, repeat the same requests, and document what changed. “We updated the rule” is a task completion. “The intended crawler can now use the page” is a verified outcome.
Build an AI Crawler Access Matrix
The audit should leave behind a decision record, not just a repaired text file.
For each platform and access type, record:
- The bot’s documented purpose and current user agent
- The organization’s intended policy
- The applicable robots.txt rule
- CDN or WAF treatment
- Verified IP source, where the platform publishes one
- Representative URL test results
- Renderability and indexability
- Observed log activity
- The owner and next review date
That matrix prevents a future security change from quietly reversing the marketing policy. It also keeps search visibility and model-training permission from collapsing into one vague yes-or-no decision.
Common findings, and the response they actually require
Search bot blocked; training bot allowed
Change the rules only if the approved policy is to support search discovery while opting out of training. Do not assume that configuration is wrong before confirming the intended policy.
Robots.txt allows access; the WAF challenges the bot
Adjust the verified-bot policy or IP rule while preserving protections against impersonators. “Turn off bot protection” is not a serious remediation plan.
The crawler reaches the URL; the main content is absent
Fix rendering, server responses, consent behavior, or the page template before commissioning more content.
The page is crawlable but noindex
Decide whether exclusion is intentional. An allow rule cannot overrule a page-level indexing decision.
No bot activity appears in the logs
Confirm the expected bot, hostname, time window, log coverage, and whether the platform has any reason to request the page. Absence of a visit is not automatically proof of a block.
What this audit cannot promise
Successful access creates eligibility. It does not guarantee retrieval, citation, traffic, or revenue.
A platform may crawl a page and never select it for an answer. It may retrieve information without displaying a citation. A user-triggered visit may follow a different path from automated indexing.
Report access separately from visibility, and visibility separately from business results. Combining them produces a simple dashboard and a misleading conclusion.
The next technical decision
Choose a small set of commercially important public pages and complete the access matrix before making a sitewide change. The test will show whether the problem sits in robots.txt, the security layer, rendering, indexing, or somewhere after access.
It will also give legal and security teams a precise decision about search, training, and user-directed retrieval instead of asking them to approve “AI crawlers” as one undifferentiated category.
Perfect Search Media can incorporate this work into a broader AI-search technical assessment. The immediate output is a verified access map. Content recommendations should follow only after the intended systems can reach and process the pages under review.
Unsure whether crawler access is failing at robots.txt, the security layer, rendering, or indexing? Explore Perfect Search Media’s AI Search services.
Frequently asked questions
If robots.txt allows an AI crawler, is the site open to that bot?
Not necessarily. A content delivery network, firewall, bot-management rule, or hosting policy can still block or challenge the same crawler. Check enforcement, not just the public file.
Are search, training, and user-triggered fetching the same permission?
No. OpenAI, Anthropic, and Perplexity document separate bots or fetchers for those jobs. A site can allow search discovery while restricting training use, or the reverse, if that is the approved policy.
Does successful crawler access guarantee AI citations or traffic?
No. Access creates eligibility. It does not guarantee retrieval, citation, clicks, or revenue. Report those outcomes separately.
Sources
- Google Search Central, “Succeeding in Google’s AI search experiences”: https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
- OpenAI, “Overview of OpenAI Crawlers”: https://developers.openai.com/api/docs/bots
- Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?”: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Perplexity, “Perplexity crawlers”: https://docs.perplexity.ai/docs/resources/perplexity-crawlers
- Google Search Central, “Introduction to robots.txt”: https://developers.google.com/search/docs/crawling-indexing/robots/intro
- Cloudflare, “AI Crawl Control”: https://developers.cloudflare.com/ai-crawl-control/