Home → Arctic Reservations → Guest-Facing Site → Limiting AI Agent Bot & Crawler Site Traffic
30.8. Limiting AI Agent Bot & Crawler Site Traffic
AI agent bots and crawlers can suck up a tremendous amount of bandwidth and site resources if left unchecked. The best way to control and limit some of these crawlers is with a robots.txt file. Additionally, for some, other access files may be needed depending on the agent/crawler in question since not all agents/crawlers respect a robots.txt file.
First, let's define a robots.txt file. A robots.txt file is a plain text file placed in a website's root directory that instructs search engine crawlers and automated bots on which pages or sections of the site they are allowed to crawl.
It is also important to know that not all automated traffic is from AI Agent Bots or Crawlers. There are some non-AI bots and crawlers out there as well and a robots.txt file will control access for both.
The purpose of this resource is to focus on AI Agent Bots and Crawlers as they now represent a lion's share of the traffic on the internet. These agents and crawlers represent an estimated 57% of site traffic currently.
Many AI agents and crawlers roaming the internet are what are referred to as Learning Models. This means that these agents and crawlers are simply out on the web scraping as much data as possible to train the AI. These AI don't provide any meaningful benefit to your site traffic but can and often do suck up a lot of data resources which can cause your site to respond slowly or even crash depending on the number of calls and pages they are scraping at once.
If you are using a subdomain guest-facing site implementation, Arctic provides a native robots.txt file as well as other access files for your guest-facing site.
However, the Arctic supplied access files do not extend to your main website. Nor do the files cover IFRAME guest-facing site implementations because the IFRAME lives on your main website.
If you are using an IFRAME implementation, you will need to have your web people implement a robots.txt file and/or other types of access files for your site. For the IFRAME implementation of the guest-facing site, the access files you implement for your main site will also cover your guest-facing site since the IFRAME lives on your main site. You do not need two sets of access files, one for your main site and one for the IFRAME, you just need one set implemented for your main site and that will also cover your IFRAME guest-facing site.
Most web teams will deploy access files automatically when creating a new site but it is good to verify that these files exist as they will help protect your website's performance. And in the case of malicious agents/bots they can add some security as well as they can prevent these bad actors from being able to access your site.
The purpose of this guide is to specifically address the growing AI agent/crawler traffic. To help you and your web person determine what access rules specifically related to AI agents and crawlers to add to your site, we recommend using the following GitHub resource:
https://github.com/ai-robots-txt/ai.robots.txt
This is a free public resource that provides extensive feedback on AI agents and crawlers and how they function, who owns them and whether or not they respect robots.txt files.
NOTE: Robots.txt files can also help with other non-AI bots and crawlers so you and your team should evaluate the non-AI bot traffic on your site as well to make sure you are covering not just AI based agents and crawlers that may be taking site resources but also the non-AI run bots and crawlers roaming the internet as well.
ALSO NOTE: Not all AI agents and crawlers or non-AI bots and crawlers are malicious or aggressive. Many provide valuable information for search engines as they map and index the sites available on the internet. There are some of these agents that are overly aggressive with the data they pull from your site which can impede site performance and there are some that are malicious but your web person and the resource above will be able to help you create access files that work best for your site.
As you explore the GitHub repository above, there are some agents/crawlers included in the ai.robots.txt file that Arctic does not block. These are the agents we chose not to block of the ones provided in the GitHub repository:
AI search indexers (crawl ahead of time so the site can show up
in AI search answers):
┌──────────────────┬───────────────────────────────────────────────┐
│ User agent │ Operator / product │
├──────────────────┼───────────────────────────────────────────────┤
│ OAI-SearchBot │ OpenAI (ChatGPT search) │
├──────────────────┼───────────────────────────────────────────────┤
│ Claude-SearchBot │ Anthropic (Claude search) │
├──────────────────┼───────────────────────────────────────────────┤
│ PerplexityBot │ Perplexity │
├──────────────────┼───────────────────────────────────────────────┤
│ Applebot │ Apple (Siri / Spotlight / Safari suggestions) │
├──────────────────┼───────────────────────────────────────────────┤
│ Amzn-SearchBot │ Amazon (Alexa / Rufus) │
├──────────────────┼───────────────────────────────────────────────┤
│ MistralAI-Index │ Mistral (Le Chat) │
├──────────────────┼───────────────────────────────────────────────┤
│ Bravebot │ Brave Search │
├──────────────────┼───────────────────────────────────────────────┤
│ meta-webindexer │ Meta AI │
└──────────────────┴───────────────────────────────────────────────┘
User-triggered fetchers (load a page when a user asks an assistant
about it or shares a link):
┌────────────────────────┬─────────────────────────────────────────┐
│ User agent │ Operator / product │
├────────────────────────┼─────────────────────────────────────────┤
│ ChatGPT-User │ OpenAI │
├────────────────────────┼─────────────────────────────────────────┤
│ Claude-User │ Anthropic │
├────────────────────────┼─────────────────────────────────────────┤
│ Perplexity-User │ Perplexity │
├────────────────────────┼─────────────────────────────────────────┤
│ MistralAI-User │ Mistral │
├────────────────────────┼─────────────────────────────────────────┤
│ Amzn-User │ Amazon (Alexa) │
├────────────────────────┼─────────────────────────────────────────┤
│ DuckAssistBot │ DuckDuckGo │
├────────────────────────┼─────────────────────────────────────────┤
│ GoogleAgent-URLContext │ Google (Gemini) │
├────────────────────────┼─────────────────────────────────────────┤
│ meta-externalfetcher │ Meta AI │
├────────────────────────┼─────────────────────────────────────────┤
│ facebookexternalhit │ Meta (Facebook/Instagram link previews) │
└────────────────────────┴─────────────────────────────────────────┘
NOTE: Of the AI Agents and Crawlers that Arctic does not block, only 6 were specifically chosen to allow. They are highlighted in the tables above and are OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User and facebookexternalhit. These are AI agents/crawlers that are specifically related to search indexing and user responsive agents, i.e. a user requests the AI agent to do a search or find out about X topic. The other allowed agents/crawlers in the tables above were just not chosen to be specifically included but are also not specifically blocked either.
ALSO NOTE: Regular search crawlers like Googlebot and Bingbot aren't in the GitHub ai.robots.txt list at all, and Arctic doesn't block them either.
IMPORTANT: This resource is meant to be a loose guide and should not be used as the final word for your specific website. You should work with your web person to determine what you would like to allow/disallow based on your site traffic and your goals for your site. Your web person will be able to provide valuable information specific to your site traffic to help you determine if your access files should be updated.
Our goal with our implementation was to limit the Learning Model based AI's that don't provide value but do take up a lot of data resources.