{"registry":{"version":"2026-09-24","count":247,"operators":146,"metrics":[{"metric":"Total agents in registry","count":241},{"metric":"Source: ai.robots.txt + Cloudflare Radar","count":55},{"metric":"Source: ai.robots.txt only","count":116},{"metric":"Source: Cloudflare Radar only","count":63},{"metric":"Source: OSINT additions","count":7},{"metric":"Purpose: Autonomous / browser agent","count":58},{"metric":"Purpose: Coding agent","count":6},{"metric":"Purpose: Other / undocumented","count":12},{"metric":"Purpose: Search / retrieval index","count":39},{"metric":"Purpose: Training / bulk crawl","count":77},{"metric":"Purpose: User-triggered fetcher","count":49},{"metric":"Verification: Web Bot Auth signature","count":57},{"metric":"Verification: User-agent string only (spoofable)","count":179},{"metric":"Verification: Unattributable / spoofed","count":5},{"metric":"Share cryptographically verifiable","count":0.236514522821577},{"metric":"Agents with evasion/compliance flag","count":12},{"metric":"Agents with a published traffic share","count":3}],"purpose_classes":{"Search / retrieval index":40,"Training / bulk crawl":77,"Autonomous / browser agent":63,"User-triggered fetcher":49,"Other / undocumented":12,"Coding agent":6},"verification":{"User-agent string only (spoofable)":179,"Web Bot Auth signature":57,"Unattributable / spoofed":10,"Published IP ranges":1}},"traffic_shares":[{"agent":"Perplexity Comet","operator":"Perplexity","metric":"Share of agentic traffic","value":0.4713,"period":"Jul 2026","source":"HUMAN State of Agentic Traffic"},{"agent":"Claude for Chrome","operator":"Anthropic","metric":"Share of agentic traffic","value":0.24,"period":"Jul 2026","source":"HUMAN"},{"agent":"ChatGPT Atlas","operator":"OpenAI","metric":"Share of agentic traffic","value":0.155,"period":"Jul 2026","source":"HUMAN"},{"agent":"ChatGPT Agent","operator":"OpenAI","metric":"Share of agentic traffic","value":0.061,"period":"Jul 2026","source":"HUMAN"},{"agent":"Browserbase","operator":"Browserbase","metric":"Share of agentic traffic","value":0.026,"period":"Jul 2026","source":"HUMAN"},{"agent":"Genspark","operator":"Genspark","metric":"Share of agentic traffic","value":0.024,"period":"Jul 2026","source":"HUMAN"},{"agent":"Browser Use Cloud","operator":"Browser Use","metric":"Share of agentic traffic","value":0.01,"period":"Jul 2026","source":"HUMAN"},{"agent":"Googlebot","operator":"Google","metric":"Share of AI-bot traffic","value":0.275,"period":"Q2 2026","source":"Cloudflare Radar via TechnologyChecker"},{"agent":"ClaudeBot","operator":"Anthropic","metric":"Share of AI-bot traffic","value":0.139,"period":"Q2 2026","source":"Cloudflare Radar via TechnologyChecker"},{"agent":"Grok Bot (GrokAgent)","operator":"xAI","metric":"Share of agentic traffic","value":"Not published","period":"—","source":"Gap: launched Aug 11 2026, after latest HUMAN report"}],"report_summary":"# Agent Consumers, 2025–2026: Who AI Agents Are, Where They Go, What They Do, and What They Spend\n\nThe agent consumer of September 2026 is overwhelmingly a reader, not a buyer. Bots now make up the majority of web traffic (57.5% of HTML requests on Cloudflare in June 2026), but most AI-agent activity is training crawls and research. Verifiable agent-initiated spending is tiny: about $15 million in adjusted, cumulative x402 stablecoin volume and a few hundred documented card-network agentic transactions. Compare that with hundreds of billions of dollars of \"AI-influenced\" purchases that humans still complete themselves.\n\n## TL;DR\n\n- **Traffic is huge and mostly takes without giving back.** On Cloudflare in Q2 2026, training and mixed-purpose crawling made up about 88% of AI crawler requests, and user-triggered agent actions about 2.5%. AI platforms send back far fewer visits than they take: Anthropic crawled about 1,900–4,600 pages per referral in mid-2026, against about 5 for Google. Browser agents are the fastest-growing slice (HUMAN: +7,851% YoY in 2025), but 76% of their activity is product and search browsing, and only 2.6% touches checkout or payment.\n- **Fiat dominates actual agent-linked spending, but humans complete almost all of it.** Salesforce attributes $262B of 2025 holiday online sales to \"AI and agent influence,\" and Adobe shows AI-referred shoppers now convert 42–54% better than other traffic. Yet OpenAI retired in-chat Instant Checkout in March 2026. The only disclosed count of card-network agent purchases is Visa's Dec 18, 2025 statement that its pilots with Skyfire, Nekuda, PayOS and Ramp had \"produced hundreds of controlled, real-world agent-initiated transactions\"; Mastercard has disclosed no count. Forecasts ($190B–$5T by 2030) are projections, not measurements.\n- **Crypto rails carry the most autonomous agent payments but the least money, and most of the headline volume is noise.** x402 claims 165M+ transactions, but a Visa/Artemis analysis adjusts that to about $15.0M of real volume across 109.6M transactions, with about 89% of raw volume classified as wash or test activity. Olas shows 14.7M agent-to-agent transactions that generated only about $110K in lifetime turnover. Stablecoins are winning machine-to-machine micropayments; cards are winning consumer checkout.\n\n## Key Findings\n\n1. **\"Agent traffic\" is four different things, and they behave differently.**\n   - Training crawlers (GPTBot, ClaudeBot, Bytespider, CCBot, Meta-ExternalAgent) generate most of the volume and almost no referrals.\n   - Search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) generate some referrals.\n   - User-triggered fetchers (ChatGPT-User, Perplexity-User, Claude-User) follow human daily cycles.\n   - Browser and computer-use agents (Perplexity Comet, Claude for Chrome, ChatGPT Agent/Atlas, Browserbase, Browser Use) render pages, log in, and occasionally check out.\n   \n   Aggregate \"bot share\" numbers blend all four and can't be compared across vendors.\n2. **The crawl-to-referral imbalance is shrinking at the margin but remains extreme.** Cloudflare's Anthropic ratio went from 70,900:1 (June 2025), with 2025 peaks near 500,000:1, to 23,951:1 (Q1 2026), 10,300:1 (May 31, 2026), about 4,580:1 (June 2026) and 1,917:1 (July 2026). Depending on the window, OpenAI sits around 217–850:1, Perplexity about 110–190:1 and Google about 5:1. Each ratio depends on its time window, so they should never be averaged.\n3. **Publishers have moved from advisory robots.txt rules to enforced blocking.** In July 2026, 35.8% of crawler requests on Cloudflare got 4xx errors, up from 14.0% a year earlier. On September 15, 2026, Cloudflare began blocking \"mixed-use\" crawlers on ad-supported pages by default. It is also turning pay-per-crawl into \"Pay Per Use.\"\n4. **The biggest money numbers measure \"influence,\" not agent spending.** Salesforce's $67B (Cyber Week) and $262B (holiday season) figures count orders touched by AI recommendations and conversational tools, including retailers' own chatbots. They are not purchases executed by agents.\n5. **Consumer agentic checkout has stalled; merchants keeping checkout on their own sites is winning.** The Information first reported the Instant Checkout shutdown on March 6, 2026. OpenAI then said \"the initial version of Instant Checkout did not offer the level of flexibility that we aspire to provide, so we're allowing merchants to use their own checkout experiences while we focus our efforts on product discovery.\" Walmart EVP Daniel Danker said the roughly 200,000 Walmart items offered through Instant Checkout from November \"converted at one-third the rate of click-out transactions,\" and called the experience \"unsatisfying.\" Google's UCP checkout is still limited to select merchants, and Visa and Mastercard are still in pilot or early-live phases.\n6. **Crypto agent metrics need heavy discounting.** Every independent audit of x402 (Artemis, CoinDesk, Visa/Artemis, Helios, Token Terminal) finds that most dollar volume was wash, test or meme-mint activity. The public x402.org counter also appears not to have changed since March 2026.\n7. **Identity is becoming the gate for money.** Web Bot Auth (HTTP Message Signatures, RFC 9421) underpins Cloudflare's signed agents, Visa's Trusted Agent Protocol, Mastercard Agent Pay and pay-per-crawl. Cryptographic agent identity is becoming the price of entry to any paid agent lane.","sector_scans":[{"id":"ecommerce-2026-09","sector":"E-commerce","date":"2026-09-29","aggregate":{"sites_scanned":16,"sites_with_ai_robots_rule":3,"sites_with_content_signal":0,"sites_challenging_automation":8,"sites_with_402_or_x402":0,"sites_with_llms_txt":6,"sites_with_agent_commerce_manifest":1,"repos_scanned":12,"repos_dormant":0},"headline_findings":[{"finding":"The first surface in four sectors and 55 sites where an agent can find a machine-readable way to buy. Allbirds, a Shopify storefront, serves a Universal Commerce Protocol manifest at /.well-known/ucp. It declares cart, checkout, order and catalog capabilities, a live endpoint, and payment handlers. Asked without any credential what it offers, the endpoint listed 13 tools, among them create_checkout and complete_checkout. No purchase was attempted.","confidence":"high","measured_or_estimate":"measured"},{"finding":"The manifest looks like a platform default, not one merchant's project. Of three further Shopify storefronts checked for the same file, Skims and Brooklinen serve it at nearly the same size and version, and Gymshark does not. If so, the agent-ready part of e-commerce is whatever runs on Shopify, and merchants got it without building it.","confidence":"medium","measured_or_estimate":"inferred"},{"finding":"Agent checkout is arriving on card rails. The manifest's payment handlers are Google Pay, Shop Pay and card. No site in this scan returned HTTP 402 or an x402 challenge. Across four sectors the only 402 seen named no price, and no x402 challenge has been seen at all. The way in for agents is a protocol wrapped around existing payments, not a new way to pay.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Retail is split. Walmart, Target, Costco, Instacart, Allbirds, Shopify, Stripe, PayPal served all four user agents the page. Amazon answered every one with a script-only interstitial, Wayfair and DoorDash with a challenge, and The Home Depot with a refusal. Best Buy returned no HTTP response to any request, robots.txt included.","confidence":"high","measured_or_estimate":"measured"},{"finding":"eBay states its position on agents in words. Its robots.txt carries a Robots and Agent Policy in comments: checkouts are strictly for human users, and buy-for-me agents, LLM-driven bots and any flow that places orders without human review are strictly prohibited. Approved integrations are sent to its API. It is the only site in this scan to address buying agents by name, and it does so in a place no crawler is obliged to read.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Amazon's robots.txt blocks 22 of the 27 tracked AI crawler tokens outright, the most in the sector. Most retailers name none: 11 of 14 readable robots.txt files have no AI rule. Retail relies on bot defences at the edge, where news publishers wrote policy.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Etsy publishes an llms.txt titled as its official reference for AI assistants, and answers GPTBot, ClaudeBot and PerplexityBot with a challenge at the edge while serving a Chrome user agent. 6 of 14 sites publish llms.txt, against none in news. E-commerce wants to be described to agents and has not decided to let their crawlers in.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Commerce platforms are being written with agents at more than twice the rate measured in developer tooling (4.8%) and news publishing (3.3%). Across eight platforms and 6,255 commits in 90 days, 807 (12.9%) carry an AI-agent co-author trailer. Saleor leads at 51 of 131 (39%), then WooCommerce at 331 of 1,435 (23%) and Spree at 331 of 1,748 (19%). Magento has none in 948 commits.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Of the four agent-payment protocols, x402 is the most active and the only one with agent co-authored commits: 31 of 287 (11%). The Universal Commerce Protocol has 88 commits and none co-authored. The Agentic Commerce Protocol has one commit in the window and AP2 has none. The protocol with the most deployment found in this scan is not the one with the most development.","confidence":"high","measured_or_estimate":"measured"}]},{"id":"news-publishing-2026-09","sector":"News and publishing","date":"2026-09-28","aggregate":{"sites_scanned":16,"sites_with_ai_robots_rule":15,"sites_with_content_signal":1,"sites_challenging_automation":13,"sites_with_402_or_x402":1,"sites_with_llms_txt":0,"repos_scanned":10,"repos_dormant":0},"headline_findings":[{"finding":"News publishers write the most detailed AI policies of any sector scanned. 15 of 16 robots.txt files name AI crawlers, and the average site blocks 14 of the 27 tracked crawler tokens outright. CNN blocks 26, The Verge 24, The New York Times 23. In developer tooling the figure was one site in eleven.","confidence":"high","measured_or_estimate":"measured"},{"finding":"The policy is enforced at the edge, not left to robots.txt. 13 of 16 sites challenged, refused or cut off at least one of four scripted requests. Only BBC, Wired, Substack served all four user agents the page, and two of those three disallow AI crawlers in robots.txt, so their block relies on the crawler's good behaviour.","confidence":"high","measured_or_estimate":"measured"},{"finding":"8 of 16 sites gave at least one AI user agent a different answer than the Chrome user agent: The New York Times, The Washington Post, The Guardian, Financial Times, CNN, The Atlantic, The Verge, NPR. CNN answers all three AI user agents with HTTP 451, Unavailable For Legal Reasons. NPR closes the connection without an HTTP response.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Two sites serve GPTBot and refuse ClaudeBot and PerplexityBot: The Guardian and The Verge. The Verge's robots.txt also allows GPTBot while blocking 24 other tracked tokens. Which crawler gets in is decided per operator, not per category.","confidence":"high","measured_or_estimate":"measured"},{"finding":"The per-operator pattern matches publicly announced content licensing agreements between OpenAI and these publishers' owners. The scan cannot see a contract, so the link is an inference from who is served and who is refused.","confidence":"medium","measured_or_estimate":"inferred"},{"finding":"One HTTP 402 in the sector, and it cannot be paid. The Atlantic answers ClaudeBot with 402 and the message \"Please contact the site owner for access\": no price, no payment header, no x402 challenge. GPTBot and PerplexityBot get a 403 from the same site. No site in three sectors and 39 sites has offered an agent a price it could pay.","confidence":"high","measured_or_estimate":"measured"},{"finding":"The Atlantic is the only site declaring Content-Signal. It sets search=yes, ai-input=no, ai-train=no for four named crawlers: Googlebot, BingBot, DuckAssistBot and FacebookBot. They may index the page and may not use it in AI answers or training.","confidence":"high","measured_or_estimate":"measured"},{"finding":"No site publishes /llms.txt: 0 of the 8 where it could be requested. Publishers are telling agents what they may not do and are not describing themselves to them.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Publishing software is being written with agents while publishers block them. Across nine repositories and 5,526 commits in 90 days, 182 (3.3%) carry an AI-agent co-author trailer. Hugo leads at 39 of 177 (22%), every one signed by Claude. Gutenberg, the WordPress editor, has 118 of 2,074 (5.7%). The Guardian's own two repositories carry 13 agent co-authored commits, mostly Copilot, while its site refuses ClaudeBot.","confidence":"high","measured_or_estimate":"measured"},{"finding":"WordPress core could not be measured. Its repository mirrors Subversion, credits contributors with a Props line, and carries no co-author trailers of any kind, so the method has nothing to read.","confidence":"high","measured_or_estimate":"measured"}]},{"id":"devtools-registries-2026-09","sector":"Developer tooling and package registries","date":"2026-09-28","aggregate":{"sites_scanned":12,"sites_with_ai_robots_rule":1,"sites_with_content_signal":1,"sites_challenging_automation":2,"sites_with_402_or_x402":0,"sites_with_llms_txt":2,"repos_scanned":10,"repos_dormant":0},"headline_findings":[{"finding":"Agent-written code is concentrated, not spread. Across ten repositories and 8,850 commits in 90 days, 421 (4.8%) carry an AI-agent co-author trailer, and three repositories account for 371 of them: pnpm (311 of 2,035, 15%), rubygems.org (44 of 279, 16%) and the npm CLI (16 of 39, 41%). crates.io has none in 1,049 commits.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Which agent a project uses is a project-level choice. Claude 324, Copilot 43, Cursor 25, Amp 24, Codex/ChatGPT 4, other 1 commits across the sector. Claude accounts for 282 of pnpm's 311; the npm CLI's are all Copilot; rubygems.org's are mostly Amp (23) and Copilot (19).","confidence":"high","measured_or_estimate":"measured"},{"finding":"1,824 of Bun's 2,351 commits on main (78%) are authored by one account, robobun, which belongs to the project's own organization. Only 5 Bun commits carry an agent trailer, so trailer counts alone would report Bun as 0.2% AI-written.","confidence":"high","measured_or_estimate":"measured"},{"finding":"robobun is probably an AI coding agent: its pull requests carry a machine-written evidence block with an iteration counter and a fixed Problem / Fix structure. The account does not say so itself, so this is an inference. If it holds, co-author trailers understate agent-written code wherever a project runs its agent under its own account.","confidence":"medium","measured_or_estimate":"inferred"},{"finding":"One of eleven sites whose robots.txt could be requested names an AI crawler. GitHub lists GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai and PerplexityBot with a crawl delay and an allow-list of marketing pages, and blocks Bytespider entirely. No package registry has an AI rule. Maven Central serves no robots.txt at all.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Stack Overflow is the only site that treats AI user agents differently. A Chrome user agent is offered a JavaScript challenge; GPTBot, ClaudeBot and PerplexityBot get a 25-byte refusal. Its robots.txt arrives with HTTP 418, disallows everything, and declares Content-signal: search=no, ai-train=no. The other sites that answered served all four user agents the same page.","confidence":"high","measured_or_estimate":"measured"},{"finding":"npm challenges every scripted client, including one sending a Chrome user agent, and the challenge covers robots.txt. A crawler that respects robots.txt cannot read npm's policy.","confidence":"high","measured_or_estimate":"measured"},{"finding":"No agent-payable surface on any of twelve sites: no HTTP 402, no x402 and no pay-per-crawl header. Two sites, Docker Hub and GitHub, publish /llms.txt. The sector is starting to describe itself to agents and has not started charging them.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Funding is fiat and sponsorship: GitHub Sponsors, Open Collective, and foundation donation pages. pnpm is the exception, listing a crypto donations page and Drips. Five of ten repositories declare no funding file.","confidence":"high","measured_or_estimate":"measured"}]},{"id":"osint-cti-2026-09","sector":"OSINT and cyber threat intelligence","date":"2026-09-25","aggregate":{"sites_scanned":11,"sites_with_ai_robots_rule":0,"sites_with_content_signal":1,"sites_challenging_automation":1,"sites_with_402_or_x402":0,"repos_scanned":11,"repos_dormant":5},"headline_findings":[{"finding":"The agent story is inside the code, not the traffic. MISP: 360 commits in the window, 163 (45%) carrying AI-agent co-author trailers (Claude Fable 5.1, Opus 5, Opus 4.8). OpenCTI: 275 commits, about 73% bot- or agent-touched (119 Renovate, 74 AI co-authored, plus Copilot Autofix). These are the CTI platforms CERTs run on, and none of it is disclosed publicly.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Five of eleven repositories are dormant: SpiderFoot's last commit was in 2023, recon-ng and holehe in 2024.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Zero of eleven sites carry an AI-crawler rule in robots.txt. The one policy signal is Have I Been Pwned's Content-Signal header: ai-train=no, ai-input=no, search=yes. OSINT Framework's robots.txt returns HTTP 500, so it has no policy at all.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Only Shodan challenges automated clients, and it blocks every non-JavaScript client equally rather than targeting AI. The other ten served GPTBot, ClaudeBot and PerplexityBot exactly what they served Chrome.","confidence":"high","measured_or_estimate":"measured"},{"finding":"No agent-payable surface anywhere: zero HTTP 402, pay-per-crawl or x402 headers across all eleven targets. Funding pages show Patreon, Donorbox, PayPal and GitHub Sponsors only; no wallet addresses in any repository funding config.","confidence":"high","measured_or_estimate":"measured"},{"finding":"The money that is moving is undisclosed fiat spend on paid agent tooling (Claude, Copilot, CodeRabbit), inferred from commit trailers. No invoice or usage figure was found, so no dollar amount is attached.","confidence":"medium","measured_or_estimate":"inferred"}]}]}