AI crawler access

What AI crawlers are

A crawler is a program that fetches web pages. Search engines have used them for decades; AI companies now run their own. Some collect pages for training models. Some fetch pages to build a search index that an assistant draws on when it answers. Some fetch a page at the moment a user asks a question that needs it. The companies publish the names their crawlers use (the user-agent string), so a website can decide which ones to admit.

For a local business that wants to be named in AI answers, the most important crawlers are the ones that feed live answers: search and user-request fetchers. Training crawlers matter too, over a longer horizon, because they shape what a model knows without searching.

Operators add and rename crawlers from time to time, so check each company's published crawler documentation once a year. The network this handbook belongs to admits all of them; each site's robots.txt lists twenty AI agents by name with an explicit allow rule.

How robots.txt controls access

The robots.txt file sits at the root of a website (for example example.com/robots.txt) and tells crawlers which parts of the site they may fetch. Well-behaved crawlers, including those in the table, follow it. A single line such as Disallow: / under a crawler's name shuts that crawler out of the whole site. A line under User-agent: * applies to every crawler not named separately.

Open the file in a browser and read it. Look for any Disallow rule under an AI crawler's name, and for a blanket Disallow: / under the wildcard. Both are common on sites built from templates, sites that once ran a staging copy, and sites where someone followed advice to block AI companies.

A permissive example

A robots.txt for a local business that wants AI visibility can be very simple:

User-agent: *
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

Sitemap: https://www.example-plumbing.com/sitemap.xml

The wildcard rule admits every crawler. The named rules are optional but make the intent explicit for the crawlers that matter most, which helps when someone later edits the file. The sitemap line tells all of them where to find the list of pages.

Training crawlers: a business decision

Some owners are comfortable with AI companies using their pages to train models; others are not. It is a legitimate choice. The practical point is to separate it from the search question. Blocking GPTBot or ClaudeBot (training) does not, by the operators' own descriptions, block OAI-SearchBot or Claude-SearchBot (search). A business that prefers not to contribute to training can block the training agents and still admit the search and user agents, so it can be found and quoted in live answers. Blocking everything removes the business from the pages assistants can read.

For most local businesses, the content in question is service descriptions, hours, prices and FAQs: facts the business wants repeated as widely as possible. That is why most GEO work, including Bay Area AEO's, aims for open access.

Firewalls and hosting defaults

Robots.txt is only one gate. Many sites sit behind a content delivery network or a security service that can block bots before robots.txt is ever read. Some of these services offer a one-click setting to block AI crawlers, and some have switched it on by default for new sites. Hosting companies may also rate-limit or challenge unfamiliar bots. The owner may never have seen these settings.

Check the security or bot-management section of the CDN or hosting dashboard. Look for settings named for AI bots, AI scrapers or crawler control, and for rules that challenge or block verified bots. Make sure the search and user-request agents are allowed. If a web developer manages these settings, ask them directly which AI crawlers are blocked.

Testing access

Three checks give a reasonable picture. First, read robots.txt as above. Second, check the server or CDN logs for visits from the AI user agents; if none appear over several weeks, something may be blocking them. Third, ask the assistants. Give Perplexity or ChatGPT a page address and ask it to summarize the page. If it cannot fetch it, or summarizes an old version, investigate access before anything else.

Some crawl-testing services and SEO tools can fetch a page as a named user agent and report the response. A blocked crawler typically receives a 403 error or a challenge page in place of the content.

About llms.txt

A proposed convention called llms.txt suggests placing a plain-text summary of a site, with links to its main pages, at /llms.txt. It is easy to add and does no harm. It is not a substitute for open access and readable pages, which are what the crawlers in this chapter actually use. Treat it as an optional extra after robots.txt, the firewall settings and the pages themselves are in order.

Access beyond the website

Crawler access matters for the business's own site, but assistants also read listings, directories and review platforms that the business does not control. Those platforms set their own rules. That is one more reason the listing and directory work in Part 2 matters: even an assistant that cannot read the website can read a correct Google, Bing or Yelp listing.

After access is restored

Unblocking a crawler does not bring it back the same afternoon. Crawlers revisit sites on their own schedules, and a site that has turned them away for months may be visited less often at first. Help them find the pages again: make sure the XML sitemap is current and listed in robots.txt, submit it in Google Search Console and Bing Webmaster Tools, and use IndexNow if the platform supports it. Then watch the logs for the AI user agents and run the monthly prompt test as usual. Changes in answers usually follow a few weeks after the crawlers return.

Crawl load and bandwidth

Some owners block bots because of server load. For a small business site, legitimate AI crawlers rarely cause real strain, but aggressive scrapers sometimes do. The better fix is targeted: rate-limit or block the specific agents misbehaving in the logs, and leave the named search and user agents from the major assistants open. A blanket block solves a bandwidth problem by removing the business from AI answers, which is an expensive trade.

Telling real crawlers from impostors

Anyone can send a request claiming to be a well-known crawler. The major operators publish ways to verify their bots, such as lists of the network addresses their crawlers use or reverse DNS lookups. Security services use these to separate genuine crawlers from impostors, and most handle it automatically. If a firewall rule is needed, base it on verified bot lists where the service offers them, so the real crawlers stay welcome while impostors are filtered out.

Making it part of the routine

Add a crawler check to the quarterly review: read robots.txt, glance at the CDN's bot settings, and look for AI user agents in the logs. Re-check after any website migration, redesign or change of hosting, which are the moments when access is most often lost. A site redesign that ships with the staging site's “block everything” robots.txt can silently undo months of GEO work.

Related chapters28. Pages machines can read · 5. Platform by platform · 25. LocalBusiness markup · 30. Keeping facts current