Anyone who looks at their website’s access logs these days will often find that a large proportion of the page views do not come from humans. In addition to the well-known search engines, crawlers have emerged that collect content for training AI models or for generating responses in AI search services. Some of these operate very aggressively: they access thousands of pages in a short space of time, including search, filter and shopping basket pages, which place a load on PHP and the database every time they are accessed.
It is your genuine visitors who feel the consequences most keenly: pages load more slowly, resource limits are reached and, in the worst-case scenario, the website may be temporarily unavailable. The good news is that you can get bot traffic under control with just a few simple measures.
Which bots are out there
- Search engines such as Googlebot and Bingbot. They are important for your visibility and should retain access.
- AI training crawlers such as GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Meta-ExternalAgent (Meta) or Bytespider (ByteDance). They collect content used to train language models.
- AI search and queries on behalf of users, such as OAI-SearchBot and ChatGPT-User (OpenAI), Claude-SearchBot and Claude-User (Anthropic), or PerplexityBot. If you block them, your site may appear less frequently - or not at all - in the results provided by these services.
- Control identifiers without their own crawlers, such as Google-Extended and Applebot-Extended. These allow you to specify whether content may be used for AI models without affecting normal search results.
- Unwanted programmes: scrapers that copy prices or text, vulnerability scanners, and bots that impersonate a browser or search engine using a fake identifier.
Step 1: Analyse who is generating the load
Before you block anything, it’s worth taking a look at your hosting provider’s access log. Pay attention to three things: the identifier (user-agent), the IP addresses with the most requests, and the URLs accessed. The load is often concentrated on a small number of bots and on pages with parameters, such as filter combinations in an online shop. Our article ‘Reading and understanding server logs’ shows you how to find and read the logs.
Step 2: Control bots via robots.txt
The file robots.txt in the root directory of your domain tells crawlers what they are permitted to access. Several User-agentlines can be grouped together. Here’s an example that prohibits AI training, allows search queries, and blocks expensive pages for all bots:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Disallow: /suche
Disallow: /warenkorb
Disallow: /*?filter=
Sitemap: https://www.muster.de/sitemap.xmlAdjust the paths to suit your system. Bear in mind two limitations of the robots.txt:
- This is a request, not a restriction. Reputable crawlers comply with it, whereas aggressive or disguised bots do not.
- It does not prevent content that has already been collected from being used. It only applies to future access.
Legally, you can object to your content being used for text and data mining by means of a machine-readable usage restriction under Section 44b of the German Copyright Act (UrhG). The robots.txt is the standard approach for this. Whether this alone is sufficient in every case has not yet been conclusively clarified in law. If in doubt, add a notice in the legal notice or in the terms and conditions of use.
Step 3: Technically restrict those who do not comply with the rules
- Rate limiting: Anyone making too many requests in a short space of time will be throttled and receive status code 429. Genuine visitors will not notice this.
- Firewall rules: Known malicious identifiers and suspicious patterns can be blocked by the web application firewall before they reach PHP and the database.
- Verify authenticity: A bot’s identifier can easily be forged. You can identify genuine search engine crawlers by performing a reverse lookup on the IP address; for Googlebot, for example, look for
googlebot.comorgoogle.com. A purported Googlebot originating from an unfamiliar address may be blocked.
Step 4: Reduce the load, regardless of who is making the request
- Page caching: Content retrieved from the cache requires hardly any processing time, even with a thousand bot requests.
- Tame filter and search pages: An infinite number of filter combinations are a maze for crawlers. For such pages, set
noindexand canonical URLs, and do not link to every combination. - Close unnecessary interfaces: In WordPress, for example
xmlrpc.phpif you don’t use them. - An upstream cache: A CDN or reverse proxy handles recurring requests before they reach your server.
And what about llms.txt?
The file llms.txt is a suggestion designed to provide AI systems with a curated overview of the most important content. It is not a binding standard, nor does it act as a restriction: it does not control who accesses your website, nor how often.
What we do to ensure this in our hosting service
In our hosting service, a web application firewall and ModSecurity filter out known attack patterns before they reach your application. ShieldCache adds an upstream cache with protective functions that intercepts recurring requests. If a bot is placing a noticeable strain on your website, please get in touch: we’ll prepare a report for you and adjust the rules accordingly.
Conclusion
Make an informed decision about which bots you want: search engines and AI search services bring visibility, whilst pure training crawlers usually just create a load. Manage the legitimate crawlers via the robots.txt, restrict the rest technically, and use caching to ensure that bot traffic doesn’t slow down your genuine visitors.
