A Quick Field Guide to AI Agent Traffic

Something changed about web traffic in the last year. Alongside the humans clicking through your pages and the search crawlers indexing them, a third kind of visitor has quietly become one of the largest categories in many sites' logs: the agent — software that reads, navigates, and acts on a person's behalf.

If you run a website, it's worth understanding what these agents are and, roughly, how anyone can tell them apart from a normal visitor. This isn't a deep technical manual — it's a tour of the landscape, with just enough detail to make the picture click. The field is also moving fast: the specifics here are a snapshot, and the names and tools will keep shifting. The useful thing to take away isn't a fixed list but a way of thinking about it.

First, a quick distinction: agents are not crawlers

People lump these together, but they're different animals, and the difference is the whole foundation.

A crawler fetches your content and leaves — that's bots like GPTBot or ClaudeBot pulling pages to train models or feed search results. An agent doesn't just read; it does things: navigates, clicks, fills forms, completes tasks.

The confusing case sits right on the line. When you ask an assistant like ChatGPT or Claude to "go read this page for me," it sends out a fetcher (ChatGPT-User, Claude-User) — and because you triggered it, it feels agent-like. But mechanically it's still just a one-shot fetch of the raw page; it doesn't browse. So the useful rule is to classify by how it behaves, not by who set it off. A live page-fetch is a crawler. Something that actually works through your site is an agent. (And the same request can go either way: "ask Claude to read a page" is a crawler-style fetch, while "ask Claude Code to read a page" runs locally and behaves like an agent. Same words, different thing under the hood.)

With that line drawn, the rest of this is about the agents.

The main kinds of agent

There's no official taxonomy, and the categories blur at the edges, but most agents you'll encounter interact with a site in one of a few ways. Grouping them by how they interact is handy, because it also happens to be what makes them recognizable.

Browser agents

These drive a real browser — rendering pages and clicking around much like a person would. This is the category most people picture: consumer products like ChatGPT Atlas and Perplexity Comet, or an AI assistant living in a Chrome extension.

The interesting wrinkle is that, from the outside, they look almost exactly like a human using Chrome — because under the hood, they are Chrome. That's what makes them the trickiest kind to spot.

Programmatic agents

These are driven by code rather than a consumer app — built on automation tools like Playwright or Puppeteer, sometimes wrapped in an AI reasoning layer, sometimes assembled by an individual and run from their own machine. They range from polished cloud services to a script someone hacked together over a weekend. As a group, they're defined by being scripted — software talking to your server directly, rather than a person (or an agent) browsing.

MCP agents

The newest and most different kind. Instead of loading and clicking your pages, these talk to a structured interface built for machines — protocols like MCP (the Model Context Protocol) and its web counterpart, or files like llms.txt. They skip the visual page entirely and go straight for the data or the actions. This is an early but fast-rising mode, and a good example of why the list of "kinds of agent" keeps growing: a year ago this barely existed.

Worth noting: these categories aren't walls. A capable agent often mixes modes — using a structured interface when one exists and falling back to driving a browser when it doesn't — so the same agent can show up looking like more than one kind.

So how can you tell?

The short version: each kind of agent leaves its traces in a different place, which is exactly why grouping them by behavior is useful. You look where that kind of agent actually shows up.

For programmatic agents, the clues arrive with the request itself, before any page even loads — things like an unusual "who am I" label (the User-Agent), missing the little headers a real browser always sends, or coming from a data-center rather than a home internet connection. A server can glance at all this the moment a request comes in. As a rough illustration of the idea — not anything elaborate — it's a bit like this:

// A simplified sketch of the idea, not a real implementation
function looksLikeAgent(request) {
  const id = request.headers["user-agent"] || "";

  if (id.includes("python-requests") || id.includes("Headless")) {
    return "looks programmatic";       // not a normal browser
  }
  if (!request.headers["sec-fetch-mode"]) {
    return "missing the usual browser headers";
  }
  return "looks like an ordinary visitor — need a closer look";
}

The point of the snippet isn't the code — it's the intuition: a lot can be guessed just from how a request introduces itself.

For browser agents, that early glance isn't enough, because they introduce themselves exactly like a real browser would. Here you have to watch how the visit behaves once it's underway — the rhythm of clicks and typing, whether a form gets filled out impossibly fast or evenly, telltale leftovers from the automation tools steering the browser. None of these is a smoking gun on its own; it's the combination that paints a picture. This is the genuinely hard case, and it's an ongoing back-and-forth: as agents get better at acting human, the tells get subtler.

For MCP agents, there's nothing to watch on the page at all — they never load it. They only show up in the logs of the structured interface they're talking to. If you don't offer such an interface, you simply won't see them; if you do, that's the place to look.

There's also a cleaner path emerging that sidesteps all the guesswork: some agents are starting to announce themselves honestly using cryptographic signatures (an effort called Web Bot Auth, now moving through standards bodies). When an agent signs its requests this way, a site can verify exactly who it is — no detective work required. It's early, and only cooperative agents participate, but it points at where things may be heading: a web where well-behaved agents simply identify themselves, and the harder detective work is reserved for the ones that don't.

Why any of this matters: the agent economy is arriving fast

It's tempting to file all this under "interesting but niche." The numbers say otherwise — this is one of the fastest-moving shifts the web has seen.

In its 2026 benchmark report, the security firm HUMAN analyzed more than a quadrillion interactions and found that traffic from AI agents and agentic browsers grew 7,851% year over year — while automated traffic overall is now growing roughly eight times faster than human traffic.¹ Cloudflare, which handles traffic for about a fifth of all websites, expects bots to outnumber humans on the internet by 2027; its CEO frames the mechanism vividly — a person buying a camera might visit five sites, while an agent doing the same errand might visit five thousand.² The multiplier is structural, and it isn't slowing down.

Behind those numbers is an emerging agent economy that the industry's leaders talk about in increasingly literal terms. OpenAI's Sam Altman describes companies treating autonomous agents like a team of "junior employees" — with human roles shifting toward assigning tasks and reviewing output.³ Anthropic's Dario Amodei has gone further, warning that agents could reshape a large share of white-collar entry-level work within a few years (a forecast he and Altman have both since tempered, a useful reminder of how uncertain the timeline really is).⁴ The framing of agents as a kind of labor force rather than a mere tool is now mainstream among the people building them.

And that framing is sprouting real infrastructure. Job marketplaces have appeared where agents are "hired" like freelancers — you describe a task, an agent quotes and delivers, and increasingly nobody on the other side is human.⁵ App-store-style directories (Anthropic's Claude Skills, OpenAI's GPT Store, MCP hubs) have become the primary way agents get discovered and distributed.⁶ There's even a budding training-and-tooling layer — a wave of Silicon Valley startups building the "environments" where agents are drilled on multi-step tasks, much like a training ground for new hires.⁷ And the human job market is reshaping around all this too, with postings mentioning agentic-AI skills jumping nearly 1,000% in a single year (2023–2024); the World Economic Forum has called trust the foundation of this whole emerging economy.⁸ In other words: agents aren't just visiting websites; they're becoming economic actors with their own marketplaces, credentials, and hiring dynamics.

Which brings it back to your own site. You don't need to act on every agent that visits — most are benign, and a growing number are doing useful things for real people, like researching a product or completing a purchase. But being able to see them matters, starting with a mundane but real problem: standard analytics tools quietly count browser agents as human visitors, because they can't tell the difference. If a slice of your "traffic" is actually agents browsing and leaving, your numbers — conversion rates, bounce rates, where visitors come from — are subtly off, and you may be drawing the wrong conclusions about real people.

More broadly, the agent web is still taking shape. The categories will shift, new interaction modes will appear, and today's clever detection trick will be tomorrow's footnote. The lasting takeaway is simpler: a real and fast-growing slice of your visitors is neither a person nor a crawler — and the sites that can see that clearly are the ones in a position to decide what to do about it.

References