AI Crawlers: Measured GPTBot and ClaudeBot Visits, and Blocking Them with robots.txt and nginx

An AI crawler is a bot that AI companies use to collect web pages for model training, AI search and answering user questions. To see how often these bots visit a server with an Nginx + Apache setup, I matched the UAs in 14 days of origin server access logs against the IP ranges each company publishes: requests from the official ranges totaled 18,605, and I summarized the settings for blocking or allowing them in robots.txt and nginx according to the official documentation.

The period is October 4–17, 2026; requests answered by the front cache that never reached the origin server, and requests whose real IP wasn't logged, are not included in the numbers.

1️⃣ Identifying AI Crawlers: UAs in the Access Log and Published IP Ranges

Not every request logged as GPTBot comes from OpenAI. The UA is text the requester writes and sends itself, so anyone can make requests under the GPTBot or ClaudeBot name.

So I first count requests per bot name by UA, then match them against the IP ranges each company publishes. Below are the AI bot names counted in one day of logs.

grep -oiE '(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Bytespider|CCBot|Amazonbot|meta-externalagent)/' access.log-20261018 | sort | uniq -c | sort -rn

   2968 GPTBot/
   2084 ClaudeBot/
    622 meta-externalagent/
    535 Amazonbot/
    167 PerplexityBot/
     99 OAI-SearchBot/
     30 ChatGPT-User/
     21 Claude-User/
     17 Claude-SearchBot/
     12 Meta-ExternalAgent/
     10 Perplexity-User/
      7 CCBot/

Counting with [/] after the name leaves out text that isn't a name, such as the contact address claudebot@ inside the Claude-SearchBot UA, and names logged with different letter case are counted separately. Next, I check from the first column which IPs the GPTBot requests came from.

grep 'GPTBot/' access.log-20261018 | awk '{print $1}' | sort | uniq -c | sort -rn | head -5

   2524 74.7.x.x
    428 185.152.x.x
     12 34.53.x.x
      1 74.7.x.x
      1 172.182.x.x

For requests that went through a CDN or proxy, the first column records the intermediate server's address, so match those lines using the address in the X-Forwarded-For column at the end of the log line. OpenAI publishes GPTBot's IP ranges as a JSON file.

curl -s https://openai.com/gptbot.json -o gptbot.json

Whether an IP is in the downloaded range file can be checked with ipaddress, a standard Python module.

# inrange.py — check whether an IP is in a JSON range file
import json, sys, ipaddress
nets = [ipaddress.ip_network(p.get('ipv4Prefix') or p.get('ipv6Prefix')) for p in json.load(open(sys.argv[1]))['prefixes']]
for a in sys.argv[2:]:
    ip = ipaddress.ip_address(a)
    print(a, 'IN' if any(ip in n for n in nets) else 'OUT')

Put the IPs to check after the range file name.

python3 inrange.py gptbot.json 74.7.x.x 34.53.x.x

74.7.x.x IN
34.53.x.x OUT

The IP that sent 2,524 requests was inside the published ranges, while the one that sent 12 was outside. Anthropic, Perplexity, Apple, Common Crawl, Google and Bing publish range files in the same format, and for Amazonbot I checked whether the reverse DNS ends in crawl.amazonbot.amazon. It's like airport passport control, which checks the passport rather than the name on the boarding pass.

Matching 14 days this way showed that 70 IPs took turns using 23 bot names, including GPTBot, ClaudeBot and Googlebot, to send 48,041 requests, 57% of them to URLs probing for vulnerabilities such as .php or .env. All 809 requests carrying the Google-Extended name also came from this group; according to Google's documentation, Google-Extended is a robots.txt-only token with no UA used in HTTP requests.

2️⃣ Visits by UA: 14 Days of Measurements and Weekly Trends

The table below shows the requests over the 14 days from October 4 to 17. For bots with published ranges I counted only requests from inside the ranges; for bots I didn't match against ranges I counted by UA, excluding the spoofing group.

Among AI crawlers verified from published ranges, ClaudeBot had the most at 10,305 requests, followed by PerplexityBot with 3,406 and GPTBot with 2,546. Claude-SearchBot and Perplexity-User had zero requests from published ranges; every request under those names came from the spoofing group.

Even after filtering out spoofing, bots that weren't matched against ranges remain a problem. Meta-ExternalAgent had the most by UA with 14,384 requests, but the Meta crawler documentation I checked has no list of IP ranges, so I counted it by UA only.

Week by week, GPTBot swung between 12 and 3,383 requests, while ClaudeBot stayed between 6,930 and 10,217 in all five weeks.

3️⃣ Which URLs They Fetch: URL Types and Response Codes

AI crawlers don't fetch only article text. Over the 14 days, 21% of Meta-ExternalAgent requests were for images, while GPTBot from published ranges requested the RSS feed 476 times and ClaudeBot requested the sitemap 314 times.

They also collect the URLs of other language versions: from published ranges, 1,111 ClaudeBot requests and 436 GPTBot requests were for posts in other languages. Response codes can be counted straight from one day of logs.

grep 'ClaudeBot/' access.log-20261018 | awk '{print $9}' | sort | uniq -c | sort -rn | head -4

   1357 200
    665 301
     31 304
     21 404

Even in a single day, close to a third are 301s, and over the 14 days 5,915 (57%) of the 10,305 ClaudeBot requests from published ranges were 301s. Of those, 4,736 were requests without a trailing [/], so WordPress answered with a 301 to the URL with [/] added, and the same post was requested once more.

4️⃣ robots.txt Settings: Training and Search Tokens

robots.txt suits training and search bots that collect automatically, but not bots that fetch a page when a user asks a question. OpenAI's and Perplexity's documentation note that robots.txt rules may not apply to ChatGPT-User and Perplexity-User requests.

Each company splits its tokens by purpose, and according to OpenAI's documentation, blocking one token doesn't affect the settings of the others. Below are the tokens confirmed in each company's official documentation.

Google-Extended and Applebot-Extended aren't separate crawling bots but tokens used only in robots.txt. The existing Google and Apple crawlers do the crawling, and these tokens only decide whether the collected content may be used to train AI models.

A robots.txt that refuses only training collection while keeping AI search visibility looks like this; OpenAI's, Perplexity's and Meta's documentation say changes can take up to 24 hours to take effect.

# refuse training collection
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

# allow AI search bots
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

The current robots.txt on the Nginx + Apache server has no lines naming AI bots.

curl -s https://example.com/robots.txt

# applies to all search engine bots
User-agent: *

# block the admin pages (security)
Disallow: /wp-admin/

# allow ajax calls among the admin functions (needed for normal operation)
Allow: /wp-admin/admin-ajax.php

# added: block spam search results (SEO protection)
Disallow: /search/
Disallow: /?s=

# point to the sitemap (faster indexing)
Sitemap: https://example.com/sitemap_index.xml

With no AI bot lines in place, the 2,546 GPTBot requests from published ranges over the 14 days got no 403s: 2,090 were answered with 200 and 432 with 301. In the same period, robots.txt requests left in the origin logs from published ranges were 36 from Googlebot, 8 from ClaudeBot and 6 from OAI-SearchBot, with none from GPTBot or PerplexityBot. robots.txt requests answered by the front cache don't show up in the origin logs either.

5️⃣ Blocking AI Crawlers in nginx and Its Limits

Even when blocked in robots.txt, requests that don't read the rules come straight through to the server. In nginx, [map] sets a variable to 1 when the UA contains a bot name.

# inside the http block — 1 if a training bot's name is in the UA
map $http_user_agent $ai_train_bot {
    default        0;
    ~*GPTBot/      1;
    ~*ClaudeBot/   1;
    ~*Bytespider   1;
    ~*CCBot/       1;
}

In the server block, return 403 when that variable is 1.

# inside the server block
location / {
    if ($fake_searchbot) {
        return 403;
    }
    if ($ai_train_bot) {
        return 403;
    }
    # existing settings below
}

Blocking by UA only checks the name, so spoofed requests are blocked along with the real bot when they use the same name and get through once they change it. Requests borrowing the name of a search bot you allow can only be filtered by matching against published IP ranges.

To allow OAI-SearchBot while rejecting only same-name requests from outside its ranges, turn the published ranges into a variable with [geo] and match it together with the UA.

# inside the http block — OAI-SearchBot published ranges (put all of searchbot.json in; only two lines shown)
geo $oai_search_net {
    default              0;
    104.210.140.128/28   1;
    135.234.64.0/24      1;
}

map $http_user_agent:$oai_search_net $fake_searchbot {
    default                  0;
    ~*OAI-SearchBot/.*:0$    1;
}

If there's a CDN or proxy in front, the address [geo] sees is the intermediate server's, so restore the real IP with real_ip before matching. Anthropic's documentation notes that blocking by IP address also stops the bot from reading robots.txt, so the refusal signal may not get through properly.

I didn't put this configuration on the live server; I only checked its syntax with a test configuration file.

nginx -t -p ./ -c nginx-test/nginx.conf

nginx: the configuration file ./nginx-test/nginx.conf syntax is ok
nginx: configuration file ./nginx-test/nginx.conf test is successful

Since only the syntax of the example was checked, I didn't verify the 403 responses; in the 14 days of logs, 2,347 of the 3,937 requests carrying the OAI-SearchBot name came from the spoofing group outside the published ranges.

6️⃣ Getting Paid by AI Crawlers: Pay per crawl and Web Bot Auth

Pay per crawl is a feature of Cloudflare AI Crawl Control that lets site owners set a price per zone and charge for each AI crawler request. According to the official documentation checked on October 18, 2026, it's in private beta, and Cloudflare handles payment as the Merchant of Record.

Crawlers that send their intent to pay in a request header get a 200 response, while the others get a [402 Payment Required] response containing the price. If the same crawler is blocked with WAF or Bot Management, the blocking rule takes priority over charging.

Crawlers that want to pay must prove who they are with Web Bot Auth. With Web Bot Auth, the bot signs its requests with an Ed25519 key and publishes its public key at [/.well-known/http-message-signatures-directory], so the receiving side can verify the signature.

The signature travels in three headers, [Signature-Agent], [Signature-Input] and [Signature], and identifies the bot by cryptographic signature instead of IP ranges. The origin server's default access log format doesn't record request headers, so I didn't check whether the AI crawler requests over the 14 days carried signatures.

The 18,605 AI crawler requests verified from published IP ranges over the 14 days equaled 77% of the 24,022 requests Googlebot and bingbot sent from their published ranges in the same period. Whether to add training tokens to robots.txt is a call to make on the numbers with spoofing removed, and adding range matching to a weekly log count lets you follow the trend.

🔢 FAQ & Recommended Content

🤖 Recommended content: AI crawlers 5 posts · 8 references

🗂️ WEB

📂 AI · Search3

📋 Zero-click search 📄 Google algorithm ranking factors 📄 JavaScript SEO

📂 IP·CDN2

📄 Check My IP 📄 Cloudflare CDN setup

🪐 References8

📡 OpenAI crawlers 📡 Anthropic crawlers 📡 Perplexity crawlers 📡 Google common crawlers 📡 Applebot 📡 Meta web crawlers 📡 Pay per crawl 📡 Web Bot Auth

No search results.

WEB · AI search

Zero-Click Search: The Big Marketing Shift a Click-Free World Brings in the 2025 AI Era

A post on the share of searches that end without a click on results pages and in AI answers, and what that shift means for marketing.

View Post →Current article

WEB · SEO

SEO: Google Algorithm Ranking Factors

A post summarizing the factors that count toward Google search rankings, with a weighting chart.

View Post →Current article

WEB · SEO

JavaScript SEO: How Google and Naver Process JS

A post on how search crawlers fetch, render and index pages built with JavaScript, covering Google and Naver separately.

View Post →Current article

WEB · IP lookup

What Is My IP? Check Your Public IP and Location, and the Limits of Accuracy

A tool post that looks up your public IP and location and explains how far IP location data can be trusted.

View Post →Current article

WEB · CDN

Cloudflare CDN Setup: Pro Plan

A post summarizing Cloudflare's plans and the CDN settings in the Pro plan.

View Post →Current article

Official docs · OpenAI

Overview of OpenAI Crawlers

Official documentation on the purposes of GPTBot, OAI-SearchBot and ChatGPT-User, their robots.txt tokens and the JSON URLs of their published IP ranges.

View Post →Current article

Official docs · Anthropic

Does Anthropic crawl data from the web, and how can site owners block the crawler?

Official documentation on the purposes of ClaudeBot, Claude-User and Claude-SearchBot, the effect of blocking them, and Crawl-delay support.

View Post →Current article

Official docs · Perplexity

Perplexity Crawlers

Official documentation on the purposes of PerplexityBot and Perplexity-User, their published IP ranges and firewall allow settings.

View Post →Current article

Official docs · Google

Google’s common crawlers

Official documentation on why the Google-Extended token is used only in robots.txt, without a UA in HTTP requests, and what it affects.

View Post →Current article

Official docs · Apple

About Applebot

Official documentation on the difference between Applebot and Applebot-Extended, reverse DNS verification and nosnippet handling.

View Post →Current article

Official docs · Meta

Meta Web Crawlers

Official documentation on the purposes of Meta-ExternalAgent, Meta-WebIndexer and Meta-ExternalFetcher and their robots.txt settings.

View Post →Current article

Official docs · Cloudflare

What is Pay Per Crawl?

Official documentation for the feature that sends AI crawler requests a 402 response with a price header and answers with 200 only those that intend to pay.

View Post →Current article

Official docs · Cloudflare

Web Bot Auth

Official documentation on how a bot proves itself by adding an Ed25519 signature to its request headers, and on registering a key directory.

View Post →Current article

‹ PreviousNext ›

ℹ️ Affiliate Disclosure
This site's content contains affiliate links. When a visitor buys a product or service through one of them, the site receives a commission from the seller. The amount the buyer pays^{(it goes down during event discounts ↓)}does not go up. Posted prices, discounts, and stock reflect the time of writing and may differ, so confirm with the seller before buying. Products are chosen and reviewed by our own standards, and commissions do not affect the order or content of reviews.