The Complete Guide to AI Crawlers and User Agents (February 2026)

A practical reference for every webmaster, SEO, and developer who wants to know which AI crawlers are reading their site, what they want, and how to control them.

The short version

AI crawlers identify themselves through a User-Agent string in HTTP requests. When you spot GPTBot, ClaudeBot, PerplexityBot — or any of the newer 2026 entries below — in your server logs, an AI model is indexing, scraping, or quoting your page.

Keep your robots.txt current and you'll guide how language models interact with your work. Ignore it and you have no idea who's reading you.

This guide reflects February 2026 reality. Three things changed recently that older lists (including our own past notes) didn't yet capture:

• Anthropic split their crawler into three earlier this month — ClaudeBot, Claude-User, Claude-SearchBot, plus a new Claude-Code. The old anthropic-ai and claude-web are deprecated but still appear in logs.
• Google-Extended and Applebot-Extended are not user agents. They're robots.txt opt-out tokens. The actual HTTP request still comes through as Googlebot or Applebot. Many guides got this wrong — including ours, before this update.
• Agentic browsers don't have user agents. ChatGPT Atlas, OpenAI Operator, Claude for Chrome — they all use standard Chrome strings. robots.txt cannot block them. This is the fastest-growing category of AI traffic in 2026, and it requires a completely different defense strategy.

Quick definitions

• AI crawler — A bot that fetches public web pages so a language model can train on them, retrieve them, or quote them in real time.
• User-Agent (UA) string — A short text identifier in the HTTP request header. Used in robots.txt rules.
• robots.txt — A plain-text file at the root of your site (example.com/robots.txt) that tells crawlers what they may fetch. One block per User-agent: you want to allow or block.
• robots.txt token — A string in robots.txt that controls a vendor's behavior but isn't actually a UA. Google-Extended is the canonical example. The vendor still crawls with Googlebot, but checks whether you've opted out via the token.

The four kinds of AI crawler

Before reading the vendor catalog below, it helps to know that AI crawlers fall into four functional categories — and the right robots.txt decision depends on the category, not the vendor. The Kind column in every table that follows maps each User-Agent to one of these.

1. Training — fetches your pages into a corpus that trains the next model. Gives nothing back. Block these if you don't want your content baked into the next generation of GPT, Claude, Gemini, or LLaMA. Examples: GPTBot, ClaudeBot, Meta-ExternalAgent, CCBot, cohere-ai.

2. Retrieval — indexes your pages so an AI product can cite them in real-time answers. Allow these and you appear in ChatGPT search, Perplexity, Claude search, Gemini, Apple Intelligence. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, YouBot.

3. User-fetch — fires when a real person asks an AI to look at a specific URL ("summarize this page," "click this citation"). Effectively a human-by-proxy; usually arrives with a referer and brings traffic with it. Examples: ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User.

4. Agentic — autonomous task-completers driving a real Chrome to research, shop, or transact across multiple sites on a user's behalf. Most have no distinguishing UA at all; robots.txt cannot block them and detection requires a different toolset (covered in The agentic browser problem below). Examples: ChatGPT Atlas, OpenAI Operator, Claude for Chrome, GoogleAgent-Mariner (the only one with a documented UA).

A fifth bucket, Multi-purpose, applies to a small number of crawlers whose vendor uses them for several of the above at once — FacebookBot, Amazonbot, Bytespider, Diffbot. These are flagged explicitly in the catalog.

The catalog below is organized by vendor for easy lookup, but each entry's Kind column maps it back to one of these five buckets — so you can scan for "all training crawlers" or "all retrieval crawlers" across vendors in a single pass.

Why you should care

Server logs show AI bots, which now account for a measurable and growing share of who reads your content. Understanding which agents the major AI products use lets you encourage or discourage that traffic deliberately.

• Allowing helpful crawlers gets you cited and linked in AI search products — Perplexity, ChatGPT search, Gemini, Claude.
• Blocking abusive scrapers stops your content from being trained into models you don't want to support.
• Knowing what's possible matters: some crawlers can't be blocked at all (agentic browsers), and others ignore robots.txt entirely. Strategy beats paranoia.

Most AI crawlers respect robots.txt by default. But the space changes monthly — vendor docs update, new vendors emerge, deprecated crawlers linger. A list older than three months is already stale.

Complete AI crawler list (February 2026)

This list reflects the state as of February 2026. Categories and behaviors based on each vendor's official documentation, cross-referenced against the ai-robots-txt GitHub repository.

OpenAI

Anthropic

Anthropic split their crawler architecture earlier this month, replacing the older anthropic-ai and claude-web crawlers with three specialized bots. Both old UAs are deprecated but still appear in logs from older clients.

Perplexity

Google

Google uses several specialized agents in addition to the main Googlebot. Note that Google-Extended is not a user agent — see the next section.

Apple

Meta

Other notable AI crawlers

What's NOT a User-Agent: robots.txt opt-out tokens

This is the most-misunderstood corner of the AI crawler space. Google-Extended and Applebot-Extended are not user agents. Many guides — including ours, before February 2026 — listed them in the UA table. They're not.

Here's what they actually are:

• Google-Extended — A robots.txt token that controls whether Googlebot's crawled content is used for Gemini training. The actual HTTP request still comes from Googlebot. Adding User-agent: Google-Extended / Disallow: / to your robots.txt tells Google "you can index me for search, but don't train Gemini on me."
• Applebot-Extended — Same pattern. Controls whether Applebot's crawled content is used for Apple AI training. The HTTP request still comes from Applebot.

You will never see Google-Extended or Applebot-Extended in your server logs as a User-Agent. If a guide tells you to "block Google-Extended," what they mean is "add this token to robots.txt to opt out of Gemini training."

robots.txt example for opting out of training while keeping search:

# Allow Google search indexing
User-agent: Googlebot
Allow: /

# Opt out of Gemini training
User-agent: Google-Extended
Disallow: /

# Allow Apple search
User-agent: Applebot
Allow: /

# Opt out of Apple AI training
User-agent: Applebot-Extended
Disallow: /

The agentic browser problem

This is the elephant in the room of 2026 AI traffic.

Agentic browsers don't have identifying user agents. ChatGPT Atlas uses a standard Chrome UA string. OpenAI Operator runs in a remote Chrome-like browser with no published UA. Claude for Chrome (the Anthropic browser extension for Max subscribers) uses real Chrome. xAI's Grok crawler reportedly switches to iPhone UAs to avoid blocks.

In your server logs, all of this looks like a regular Chrome user. robots.txt cannot block them. UA-based firewall rules don't apply.

This category is the fastest-growing AI traffic type in 2026. Users invoke these agents to research, shop, summarize, or transact on their behalf — and the agents arrive at your site looking like a normal browser visit. Google Analytics counts them as humans. Shopify counts them as Shopify-channel sessions if they convert. You have no idea what share of your "human" traffic is actually agentic.

If you need to detect or control this category, robots.txt is the wrong tool. You'll need:

• IP-range allowlists/blocklists — OpenAI, Anthropic, and xAI publish IP ranges for some of their agents. Cloudflare bot management can match against these.
• Behavioral fingerprinting — request timing patterns, JavaScript execution signatures, session correlation. This requires either a CDN with bot management features or a tracking pixel that captures these signals.
• Account-level access controls — for content behind login, use rate limits and abuse detection at the application layer.

Protal's Phase 2 product, Protal Analytics, is specifically designed to detect and report on this category — agents that don't show up via UA. It's in development; if this problem is on your radar, join the early-access list.

Robots.txt examples for common goals

Goal: Allow AI search, block AI training

This is the most common 2026 setup. You want to be cited by ChatGPT, Perplexity, and Claude — but you don't want your content baked into the next model.

# Block training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: CCBot
Disallow: /

# Opt out of Gemini and Apple AI training (these are tokens, not UAs)
User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Allow search and citation crawlers
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: YouBot
Allow: /

# Allow user-triggered fetches (real visitor on the other end)
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: MistralAI-User
Allow: /

# Default for everything else
User-agent: *
Allow: /

Goal: Allow everything (let AI read freely)

For sites that want maximum AI distribution — public docs, marketing content, brand-building blogs:

User-agent: *
Allow: /

That's it. The default for any unlisted crawler is allow.

Goal: Block everything

For private content, staging environments, or sites that don't want any AI access:

User-agent: *
Disallow: /

Note: this won't stop bots that ignore robots.txt (Bytespider has a track record of ignoring it). Use firewall rules for hard blocking.

Best practices

1. Specify every AI agent you care about explicitly. A wildcard User-agent: * catches most, but some bots only check specific blocks. Listing them by name is more reliable.
2. Always pair User-agent: with at least one Allow: or Disallow: line. A User-agent: declaration without a directive does nothing.
3. Use blank lines between blocks. Some parsers merge adjacent blocks if you don't.
4. Re-test after major LLM model releases. New model versions sometimes ship new crawler UAs or change behavior. The space moves monthly.
5. Don't rely on robots.txt for hard security. It's a politeness protocol, not access control. For real blocking, use firewall rules or auth.

How to check what's actually visiting you

The best way to know which AI crawlers reach your site is to read your server logs.

grep -Ei "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|claude-searchbot|perplexitybot|bytespider|google-agent|amazonbot|googleagent-mariner" access.log

You'll see hits like:

203.0.113.10 - - [15/Apr/2026:08:14:22 -0500] "GET /blog/ HTTP/1.1" 200 15843 "-" "GPTBot/1.2"

What this tells you:

• An expected bot is missing? Either you've blocked it (check robots.txt), it hasn't discovered your site yet, or it's not crawling your category.
• An unexpected bot is hitting you hard? Check the legitimacy — some crawlers misrepresent themselves. Verify against published IP ranges where available.
• Patterns of "Chrome" traffic with no JavaScript execution and no referer? Could be agentic browsers. UA alone won't tell you.

Want to know if your robots.txt is configured correctly?

We built Protal Audit specifically to check this — and dozens of other AI-readiness factors — in one pass.

Run a free Protal Audit →

Free. No signup required. Reports include your full robots.txt analysis, schema.org structured data validation, llms.txt presence, MCP discovery, and a battery of other rules across 9 categories. Every rule is documented in our public methodology.

FAQs

Do AI crawlers have to follow robots.txt?

No. robots.txt is a politeness protocol, not a legal mandate. Most reputable crawlers respect it. Some — Bytespider has the most documented track record — ignore it. For hard blocking, use firewall rules or password-protect the content.

What about IP-based blocking?

OpenAI, Anthropic, and a few others publish IP ranges for their crawlers. You can use these in Cloudflare firewall rules or your CDN's bot management features. This is more reliable than UA-based rules but more work to maintain. Note that IP ranges change — subscribe to vendor announcements or use a service that tracks them.

My logs show "Chrome" requests that don't look human. What are they?

Likely candidates: agentic browsers (ChatGPT Atlas, Operator, Claude for Chrome), headless browsers running on cloud IPs (Selenium / Playwright scrapers), or actual humans with privacy extensions. Distinguishing them requires JavaScript execution analysis, IP range correlation, or behavioral fingerprinting — none of which are visible in UA alone.

Should I block GPTBot to opt out of training?

Depends on your business. If your content is your product (a publication, a SaaS docs site, a research org), opting out preserves your competitive moat. If your content is your marketing (a blog promoting a service), being trained into ChatGPT might increase your discoverability. There's no universal right answer.

Why do you say Google-Extended isn't a user agent?

Because it isn't. Google's documentation is explicit: Google-Extended is a robots.txt token that controls how Googlebot's crawled content is used for Gemini training. The HTTP request still comes through as Googlebot. You will never see Google-Extended in a User-Agent header. Many guides — including ours, prior to February 2026 — listed it in their UA tables. We've corrected that.

How often does this list change?

Roughly once a quarter, sometimes faster. Anthropic's recent crawler split (this month) is the most recent major change. We update this guide whenever vendor documentation changes, new significant crawlers emerge, or community data (especially the ai-robots-txt repo) reflects shifts in the ecosystem.

Spotted an error or missing crawler? Reach us at hello@protal.ai — we monitor this list as part of our public methodology and update it regularly.

This guide is part of Protal's effort to provide neutral, accurate, technical references for the AI-era web. Protal builds tools that help websites prepare for and observe AI traffic. Learn more about Protal Audit and our forthcoming Protal Analytics.