{"registry":{"version":"2026-10-02","count":350,"operators":236,"metrics":[{"metric":"Total agents in registry","count":241},{"metric":"Source: ai.robots.txt + Cloudflare Radar","count":55},{"metric":"Source: ai.robots.txt only","count":116},{"metric":"Source: Cloudflare Radar only","count":63},{"metric":"Source: OSINT additions","count":7},{"metric":"Purpose: Autonomous / browser agent","count":58},{"metric":"Purpose: Coding agent","count":6},{"metric":"Purpose: Other / undocumented","count":12},{"metric":"Purpose: Search / retrieval index","count":39},{"metric":"Purpose: Training / bulk crawl","count":77},{"metric":"Purpose: User-triggered fetcher","count":49},{"metric":"Verification: Web Bot Auth signature","count":57},{"metric":"Verification: User-agent string only (spoofable)","count":179},{"metric":"Verification: Unattributable / spoofed","count":5},{"metric":"Share cryptographically verifiable","count":0.236514522821577},{"metric":"Agents with evasion/compliance flag","count":12},{"metric":"Agents with a published traffic share","count":3}],"purpose_classes":{"Search / retrieval index":71,"Other / undocumented":75,"Training / bulk crawl":78,"Autonomous / browser agent":63,"User-triggered fetcher":57,"Coding agent":6},"verification":{"User-agent string only (spoofable)":265,"Web Bot Auth signature":68,"Published IP ranges":7,"Unattributable / spoofed":10},"key_revocation":{"removal_only":47,"keys_expire":18,"directory_unreachable":3}},"traffic_shares":[{"agent":"Perplexity Comet","operator":"Perplexity","metric":"Share of agentic traffic","value":0.4713,"period":"Jul 2026","source":"HUMAN State of Agentic Traffic"},{"agent":"Claude for Chrome","operator":"Anthropic","metric":"Share of agentic traffic","value":0.24,"period":"Jul 2026","source":"HUMAN"},{"agent":"ChatGPT Atlas","operator":"OpenAI","metric":"Share of agentic traffic","value":0.155,"period":"Jul 2026","source":"HUMAN"},{"agent":"ChatGPT Agent","operator":"OpenAI","metric":"Share of agentic traffic","value":0.061,"period":"Jul 2026","source":"HUMAN"},{"agent":"Browserbase","operator":"Browserbase","metric":"Share of agentic traffic","value":0.026,"period":"Jul 2026","source":"HUMAN"},{"agent":"Genspark","operator":"Genspark","metric":"Share of agentic traffic","value":0.024,"period":"Jul 2026","source":"HUMAN"},{"agent":"Browser Use Cloud","operator":"Browser Use","metric":"Share of agentic traffic","value":0.01,"period":"Jul 2026","source":"HUMAN"},{"agent":"Googlebot","operator":"Google","metric":"Share of AI-bot traffic","value":0.275,"period":"Q2 2026","source":"Cloudflare Radar via TechnologyChecker"},{"agent":"ClaudeBot","operator":"Anthropic","metric":"Share of AI-bot traffic","value":0.139,"period":"Q2 2026","source":"Cloudflare Radar via TechnologyChecker"},{"agent":"Grok Bot (GrokAgent)","operator":"xAI","metric":"Share of agentic traffic","value":"Not published","period":"—","source":"Gap: launched Aug 11 2026, after latest HUMAN report"}],"report_summary":"# Agent Consumers, 2025–2026: Who AI Agents Are, Where They Go, What They Do, and What They Spend\n\nThe agent consumer of September 2026 is overwhelmingly a reader, not a buyer. Bots now make up the majority of web traffic (57.5% of HTML requests on Cloudflare in June 2026), but most AI-agent activity is training crawls and research. Verifiable agent-initiated spending is tiny: about $15 million in adjusted, cumulative x402 stablecoin volume and a few hundred documented card-network agentic transactions. Compare that with hundreds of billions of dollars of \"AI-influenced\" purchases that humans still complete themselves.\n\n## TL;DR\n\n- **Traffic is huge and mostly takes without giving back.** On Cloudflare in Q2 2026, training and mixed-purpose crawling made up about 88% of AI crawler requests, and user-triggered agent actions about 2.5%. AI platforms send back far fewer visits than they take: Anthropic crawled about 1,900–4,600 pages per referral in mid-2026, against about 5 for Google. Browser agents are the fastest-growing slice (HUMAN: +7,851% YoY in 2025), but 76% of their activity is product and search browsing, and only 2.6% touches checkout or payment.\n- **Fiat dominates actual agent-linked spending, but humans complete almost all of it.** Salesforce attributes $262B of 2025 holiday online sales to \"AI and agent influence,\" and Adobe shows AI-referred shoppers now convert 42–54% better than other traffic. Yet OpenAI retired in-chat Instant Checkout in March 2026. The only disclosed count of card-network agent purchases is Visa's Dec 18, 2025 statement that its pilots with Skyfire, Nekuda, PayOS and Ramp had \"produced hundreds of controlled, real-world agent-initiated transactions\"; Mastercard has disclosed no count. Forecasts ($190B–$5T by 2030) are projections, not measurements.\n- **Crypto rails carry the most autonomous agent payments but the least money, and most of the headline volume is noise.** x402 claims 165M+ transactions, but a Visa/Artemis analysis adjusts that to about $15.0M of real volume across 109.6M transactions, with about 89% of raw volume classified as wash or test activity. Olas shows 14.7M agent-to-agent transactions that generated only about $110K in lifetime turnover. Stablecoins are winning machine-to-machine micropayments; cards are winning consumer checkout.\n\n## Key Findings\n\n1. **\"Agent traffic\" is four different things, and they behave differently.**\n   - Training crawlers (GPTBot, ClaudeBot, Bytespider, CCBot, Meta-ExternalAgent) generate most of the volume and almost no referrals.\n   - Search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) generate some referrals.\n   - User-triggered fetchers (ChatGPT-User, Perplexity-User, Claude-User) follow human daily cycles.\n   - Browser and computer-use agents (Perplexity Comet, Claude for Chrome, ChatGPT Agent/Atlas, Browserbase, Browser Use) render pages, log in, and occasionally check out.\n   \n   Aggregate \"bot share\" numbers blend all four and can't be compared across vendors.\n2. **The crawl-to-referral imbalance is shrinking at the margin but remains extreme.** Cloudflare's Anthropic ratio went from 70,900:1 (June 2025), with 2025 peaks near 500,000:1, to 23,951:1 (Q1 2026), 10,300:1 (May 31, 2026), about 4,580:1 (June 2026) and 1,917:1 (July 2026). Depending on the window, OpenAI sits around 217–850:1, Perplexity about 110–190:1 and Google about 5:1. Each ratio depends on its time window, so they should never be averaged.\n3. **Publishers have moved from advisory robots.txt rules to enforced blocking.** In July 2026, 35.8% of crawler requests on Cloudflare got 4xx errors, up from 14.0% a year earlier. On September 15, 2026, Cloudflare began blocking \"mixed-use\" crawlers on ad-supported pages by default. It is also turning pay-per-crawl into \"Pay Per Use.\"\n4. **The biggest money numbers measure \"influence,\" not agent spending.** Salesforce's $67B (Cyber Week) and $262B (holiday season) figures count orders touched by AI recommendations and conversational tools, including retailers' own chatbots. They are not purchases executed by agents.\n5. **Consumer agentic checkout has stalled; merchants keeping checkout on their own sites is winning.** The Information first reported the Instant Checkout shutdown on March 6, 2026. OpenAI then said \"the initial version of Instant Checkout did not offer the level of flexibility that we aspire to provide, so we're allowing merchants to use their own checkout experiences while we focus our efforts on product discovery.\" Walmart EVP Daniel Danker said the roughly 200,000 Walmart items offered through Instant Checkout from November \"converted at one-third the rate of click-out transactions,\" and called the experience \"unsatisfying.\" Google's UCP checkout is still limited to select merchants, and Visa and Mastercard are still in pilot or early-live phases.\n6. **Crypto agent metrics need heavy discounting.** Every independent audit of x402 (Artemis, CoinDesk, Visa/Artemis, Helios, Token Terminal) finds that most dollar volume was wash, test or meme-mint activity. The public x402.org counter also appears not to have changed since March 2026.\n7. **Identity is becoming the gate for money.** Web Bot Auth (HTTP Message Signatures, RFC 9421) underpins Cloudflare's signed agents, Visa's Trusted Agent Protocol, Mastercard Agent Pay and pay-per-crawl. Cryptographic agent identity is becoming the price of entry to any paid agent lane.","sector_scans":[{"id":"news-publishing-2026-10","sector":"News and publishing","date":"2026-10-19","aggregate":{"sites_scanned":16,"sites_with_ai_robots_rule":15,"sites_with_content_signal":1,"sites_challenging_automation":13,"sites_with_402_or_x402":1,"sites_with_llms_txt":0,"repos_scanned":10,"repos_dormant":0},"headline_findings":[{"finding":"The sector's robots.txt policies are unchanged in detail since September: 15 of 16 files name AI crawlers (Reuters, unreadable in September, names three and blocks one), and the average site still disallows 14 of the 27 tracked tokens outright. CNN blocks 25 (one fewer than in September), The Verge 24, The New York Times 23. Substack and The Wall Street Journal block none.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Enforcement is at the edge, as in September: 13 of 16 sites challenged, refused or cut off at least one of four scripted requests, and only BBC, Substack and Wired served all four user agents the page. The same eight sites give at least one AI user agent a different answer than the Chrome user agent: CNN, Financial Times, NPR, The Atlantic, The Guardian, The New York Times, The Verge and The Washington Post. NPR drops the connection for all three AI agents while serving Chrome.","confidence":"high","measured_or_estimate":"measured"},{"finding":"The Guardian and The Verge still serve GPTBot and refuse ClaudeBot and PerplexityBot, and The Verge's robots.txt still allows GPTBot while blocking 24 other tracked tokens. Which crawler gets in is decided per operator, and the pattern has held for three weeks.","confidence":"high","measured_or_estimate":"measured"},{"finding":"The Atlantic still answers ClaudeBot alone with HTTP 402 and no price, payment header or x402 challenge, and remains the only site in the sector declaring Content-Signal (search=yes, ai-input=no, ai-train=no). No site publishes /llms.txt: 0 of the 13 where it could be requested. Publishers tell agents what they may not do and do not describe themselves to them.","confidence":"high","measured_or_estimate":"measured"},{"finding":"The Guardian states its position in words in robots.txt comments: content is available under its terms, and any other use, including for large language models, machine learning or AI-related purposes, is not permitted; licensing@theguardian.com is given for assistance. Comments bind nobody, and the file's groups block 14 tracked tokens outright.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Publishing software is being written with agents while publishers block them, a little more than in September: 296 of 7,363 commits (4.0%) across ten repositories in the 90 days to 2026-10-19 carry an agent trailer or come from an agent account, against 182 of 5,526 (3.3%) in September's window. Hugo leads at 45 of 186 (24.2%, all Claude). Astro's 93 of 535 (17.4%) are new and come almost entirely from factory[bot], Factory's agent account (85 commits), with no trailer on the commits themselves. Gutenberg has 125 of 2,296 (5.4%); Ghost 17 of 2,265.","confidence":"high","measured_or_estimate":"measured"},{"finding":"WordPress core still cannot be measured by this method: 1,083 commits on wordpress-develop in the window, none with a co-author trailer or an agent account, because the repository mirrors Subversion and credits contributors with a Props line.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Across 10 repositories and 7,363 commits in the 90 days to 2026-10-19, 296 (4.0%) carry an AI-agent co-author trailer or come from an agent account. Highest shares: Hugo 45 of 186 (24.2%), Astro 93 of 535 (17.4%), Gutenberg 125 of 2296 (5.4%).","confidence":"high","measured_or_estimate":"measured"},{"finding":"By agent family across the sector: Claude 176, other 85, Copilot 23, Cursor 12 commits.","confidence":"high","measured_or_estimate":"measured"},{"finding":"15 of 16 sites whose robots.txt could be read name a tracked AI crawler; 13 of 16 challenged, refused or cut off at least one of four scripted requests; 1 answered with HTTP 402 or a payment header; 0 of 13 where it could be requested publish /llms.txt.","confidence":"high","measured_or_estimate":"measured"}]},{"id":"osint-cti-2026-10","sector":"OSINT and cyber threat intelligence","date":"2026-10-19","aggregate":{"sites_scanned":11,"sites_with_ai_robots_rule":0,"sites_with_content_signal":1,"sites_challenging_automation":0,"sites_with_402_or_x402":0,"sites_with_llms_txt":2,"repos_scanned":11,"repos_dormant":4},"headline_findings":[{"finding":"MISP is being written with an agent at a rate higher than any repository in the three other sectors scanned except npm's command-line client: 295 of 766 commits (38.5%) in the 90 days to 2026-10-19 carry a Claude co-author trailer, every one of them Claude (Claude Fable 5.1 106, Claude Opus 5 with 1M context 101, Opus 5.5 29, Opus 4.8 24). In September's window the figure was 163 of 360 (45%); the share fell as the commit volume doubled.","confidence":"high","measured_or_estimate":"measured"},{"finding":"OpenCTI: 138 of 851 commits (16.2%) are agent co-authored (Claude 87, Copilot 48, Cursor 3) and 378 more come from automation accounts, so 516 of 851 (60.6%) were touched by a bot or an agent, down from about 73% in September's window. The two platforms CERTs and SOCs run on are both agent-written in part, and neither says so anywhere but in its commit trailers.","confidence":"high","measured_or_estimate":"measured"},{"finding":"The rest of the sector's open tooling is quiet. theHarvester (413 commits) and subfinder (67) carry no agent trailer, and four of eleven repositories had no commit on their default branch in 90 days: Amass (last commit April 2026), SpiderFoot (November 2023), holehe and Aleph (December 2025). The recon tools a generation of analysts learned on are not being maintained, by people or by agents. Caldera moved from mitre to apache and made 10 commits.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Every one of the 11 sites served all four user agents the page: the sector neither challenges scripted clients nor singles out AI crawlers, the only sector of four where that is true. No robots.txt names an AI crawler, unchanged since September. Have I Been Pwned remains the only site with a machine-readable AI policy (Content-Signal: ai-train=no, ai-input=no, search=yes). Shodan and Censys publish /llms.txt, so the two commercial scanners now describe themselves to agents while nobody in the sector restricts them. osintframework.com's robots.txt returned a Cloudflare worker error. No site answered with HTTP 402 or a payment header.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Sector check figures: 11 sites, 0 with an AI robots rule, 1 with Content-Signal, 2 with /llms.txt, 0 challenging automation, 0 with 402 or x402; 11 repositories, 4 dormant.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Across 11 repositories and 2,120 commits in the 90 days to 2026-10-19, 433 (20.4%) carry an AI-agent co-author trailer or come from an agent account. Highest shares: MISP 295 of 766 (38.5%), OpenCTI 138 of 851 (16.2%), theHarvester 0 of 413 (0.0%).","confidence":"high","measured_or_estimate":"measured"},{"finding":"By agent family across the sector: Claude 382, Copilot 48, Cursor 3 commits.","confidence":"high","measured_or_estimate":"measured"}]},{"id":"ecommerce-2026-10","sector":"E-commerce and payments","date":"2026-10-19","aggregate":{"sites_scanned":16,"sites_with_ai_robots_rule":5,"sites_with_content_signal":0,"sites_challenging_automation":10,"sites_with_402_or_x402":0,"sites_with_llms_txt":6,"sites_with_agent_commerce_manifest":1,"repos_scanned":12,"repos_dormant":2},"headline_findings":[{"finding":"Allbirds is still the only site in the sector, and in four sectors and 55 sites, that serves a machine-readable way to buy: a Universal Commerce Protocol manifest at /.well-known/ucp, alongside /llms.txt. Since September the same storefront has started rate-limiting the crawlers that would read it: GPTBot and ClaudeBot now receive HTTP 429 while Chrome and PerplexityBot are served the page. The manifest is published and two of the three AI agents tested cannot reach the site that publishes it.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Retail has tightened at the edge in three weeks. Walmart and The Home Depot served all four user agents in September and now serve Chrome while challenging GPTBot, ClaudeBot and PerplexityBot; Best Buy and eBay drop the connection for the three AI agents and serve or refuse Chrome; Etsy, which served Chrome and challenged the agents in September, now challenges all four alike; Amazon still answers every scripted client with a script-only interstitial. 10 of 16 sites challenged, refused or cut off at least one request, up from 9. Costco, Instacart, PayPal, Shopify, Stripe and Target served all four.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Target serves AI user agents a different page than a browser: GPTBot, ClaudeBot and PerplexityBot received a 130 KB page with more text (3,920 characters) than the 459 KB page sent to the Chrome user agent (3,185 characters). With Docker Hub in developer tooling, that makes two sites in four sectors giving agents a lighter, fuller page rather than a wall.","confidence":"high","measured_or_estimate":"measured"},{"finding":"eBay's Robots & Agent Policy is unchanged: checkouts are strictly for human users, and automated scraping, buy-for-me agents, LLM-driven bots and any end-to-end flow that places orders without human review are strictly prohibited, in comments no crawler is required to read; the file names 13 tracked tokens and disallows 9 entirely. AliExpress has added AI rules since September: its robots.txt now names six tracked tokens and blocks all six. Amazon still blocks 22 of 27. 5 of 15 readable files name an AI crawler; no site declares Content-Signal, and no site answered with HTTP 402 or a payment header. Six publish /llms.txt: Allbirds, Etsy, PayPal, Shopify, Stripe and Target.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Commerce platforms remain the most agent-written code of the four sectors: 959 of 7,599 commits (12.6%) across twelve repositories in the 90 days to 2026-10-19 carry an agent trailer or come from an agent account, against 807 of 6,255 (12.9%) in September's window, and against 6.8% in developer tooling and 4.0% in publishing. WooCommerce 463 of 1,691 (27.4%, 454 of them Claude), Saleor 48 of 129 (37.2%, all Claude), Spree 247 of 1,892 (13.1%, Claude 211 and Cursor 35), Vendure 86 of 866, Shopify Hydrogen 35 of 219 (Codex 22, Claude 12). Magento and PrestaShop, the older platforms, are under 3%.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Of the four agent-payment protocols, two have gone quiet. The Agent Payments Protocol (AP2) last committed in April 2026 and the Agentic Commerce Protocol on 2026-07-18, three days before the window opens, so both are dormant for this scan. The Universal Commerce Protocol made 104 commits, none agent co-authored; x402 made 267, 32 of them (12%) agent co-authored (Claude 16, Cursor 16), about the same share as in September (31 of 287).","confidence":"high","measured_or_estimate":"measured"},{"finding":"Across 12 repositories and 7,599 commits in the 90 days to 2026-10-19, 959 (12.6%) carry an AI-agent co-author trailer or come from an agent account. Highest shares: Saleor 48 of 129 (37.2%), WooCommerce 463 of 1691 (27.4%), Shopify Hydrogen 35 of 219 (16.0%).","confidence":"high","measured_or_estimate":"measured"},{"finding":"By agent family across the sector: Claude 857, Cursor 61, Codex/ChatGPT 22, Copilot 18, Amp 1 commits.","confidence":"high","measured_or_estimate":"measured"},{"finding":"5 of 15 sites whose robots.txt could be read name a tracked AI crawler; 10 of 16 challenged, refused or cut off at least one of four scripted requests; 0 answered with HTTP 402 or a payment header; 6 of 14 where it could be requested publish /llms.txt.","confidence":"high","measured_or_estimate":"measured"}]},{"id":"devtools-registries-2026-10","sector":"Developer tooling and package registries","date":"2026-10-19","aggregate":{"sites_scanned":12,"sites_with_ai_robots_rule":1,"sites_with_content_signal":0,"sites_challenging_automation":3,"sites_with_402_or_x402":0,"sites_with_llms_txt":2,"repos_scanned":10,"repos_dormant":0},"headline_findings":[{"finding":"Agent-written code in the sector rose by almost half in three weeks of overlap: 611 of 8,922 commits (6.8%) in the 90 days to 2026-10-19 carry an AI-agent co-author trailer or come from an agent account, against 421 of 8,850 (4.8%) in the window to 2026-09-28. It is still concentrated: pnpm 478 of 2,312 (20.7%, up from 15.3%), rubygems.org 74 of 295 (25.1%, up from 15.8%) and the npm CLI 17 of 38 (44.7%) account for 569 of the 611.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Which agent a project uses is still a project-level choice. Claude 482, Amp 57, Copilot 41, Cursor 24, Codex/ChatGPT 7 commits across the sector; pnpm's are 444 Claude, rubygems.org's are mostly Amp (53 of 74), and every one of the npm CLI's is Copilot. crates.io and Deno carry no agent trailer at all in the window.","confidence":"high","measured_or_estimate":"measured"},{"finding":"1,461 of Bun's 2,079 commits on main (70%) are authored by robobun, the project's own account, down from 78% in September; only 4 carry an agent trailer. Automation accounts are the larger share of machine-made commits in uv (576 of 1,336), crates.io (417 of 1,104) and Bun; agent trailers are the larger share in pnpm, rubygems.org and the npm CLI.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Maven Central now treats AI user agents differently from a browser: a Chrome user agent is served the page, and GPTBot, ClaudeBot and PerplexityBot each get a 25-byte HTTP 403. In September all four were served. Stack Overflow is unchanged (a JavaScript challenge for Chrome, a flat refusal for the three AI agents) and npm still challenges every scripted client, robots.txt included. Three of twelve sites now challenge or refuse, up from two.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Docker Hub is the first site in four sectors seen giving an AI user agent more than a browser gets: GPTBot, ClaudeBot and PerplexityBot received a 480 KB page with 6,871 characters of text, while the Chrome user agent received a 285 KB page with 2,345 characters. The site publishes /llms.txt and its robots.txt disallows nothing.","confidence":"high","measured_or_estimate":"measured"},{"finding":"GitHub's robots.txt is the only one in the sector that names AI crawlers, as in September: GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai and PerplexityBot share a group with a crawl delay of 1 and an allow-list of marketing pages, and Bytespider is listed separately. Two sites, Docker Hub and GitHub, publish /llms.txt; none answers with HTTP 402 or a payment header. pnpm has removed its crypto-donations page and Drips from FUNDING.yml since September, so the sector's one crypto funding rail is gone: funding is GitHub Sponsors, Open Collective and foundation pages, and five of ten repositories declare none.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Across 10 repositories and 8,922 commits in the 90 days to 2026-10-19, 611 (6.8%) carry an AI-agent co-author trailer or come from an agent account. Highest shares: cli 17 of 38 (44.7%), rubygems.org 74 of 295 (25.1%), pnpm 478 of 2312 (20.7%).","confidence":"high","measured_or_estimate":"measured"},{"finding":"By agent family across the sector: Claude 482, Amp 57, Copilot 41, Cursor 24, Codex/ChatGPT 7 commits.","confidence":"high","measured_or_estimate":"measured"},{"finding":"1 of 10 sites whose robots.txt could be read name a tracked AI crawler; 3 of 12 challenged, refused or cut off at least one of four scripted requests; 0 answered with HTTP 402 or a payment header; 2 of 9 where it could be requested publish /llms.txt.","confidence":"high","measured_or_estimate":"measured"}]},{"id":"ecommerce-2026-09","sector":"E-commerce","date":"2026-09-29","aggregate":{"sites_scanned":16,"sites_with_ai_robots_rule":3,"sites_with_content_signal":0,"sites_challenging_automation":8,"sites_with_402_or_x402":0,"sites_with_llms_txt":6,"sites_with_agent_commerce_manifest":1,"repos_scanned":12,"repos_dormant":0},"headline_findings":[{"finding":"The first surface in four sectors and 55 sites where an agent can find a machine-readable way to buy. Allbirds, a Shopify storefront, serves a Universal Commerce Protocol manifest at /.well-known/ucp. It declares cart, checkout, order and catalog capabilities, a live endpoint, and payment handlers. Asked without any credential what it offers, the endpoint listed 13 tools, among them create_checkout and complete_checkout. No purchase was attempted.","confidence":"high","measured_or_estimate":"measured"},{"finding":"The manifest looks like a platform default, not one merchant's project. Of three further Shopify storefronts checked for the same file, Skims and Brooklinen serve it at nearly the same size and version, and Gymshark does not. If so, the agent-ready part of e-commerce is whatever runs on Shopify, and merchants got it without building it.","confidence":"medium","measured_or_estimate":"inferred"},{"finding":"Agent checkout is arriving on card rails. The manifest's payment handlers are Google Pay, Shop Pay and card. No site in this scan returned HTTP 402 or an x402 challenge. Across four sectors the only 402 seen named no price, and no x402 challenge has been seen at all. The way in for agents is a protocol wrapped around existing payments, not a new way to pay.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Retail is split. Walmart, Target, Costco, Instacart, Allbirds, Shopify, Stripe, PayPal served all four user agents the page. Amazon answered every one with a script-only interstitial, Wayfair and DoorDash with a challenge, and The Home Depot with a refusal. Best Buy returned no HTTP response to any request, robots.txt included.","confidence":"high","measured_or_estimate":"measured"},{"finding":"eBay states its position on agents in words. Its robots.txt carries a Robots and Agent Policy in comments: checkouts are strictly for human users, and buy-for-me agents, LLM-driven bots and any flow that places orders without human review are strictly prohibited. Approved integrations are sent to its API. It is the only site in this scan to address buying agents by name, and it does so in a place no crawler is obliged to read.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Amazon's robots.txt blocks 22 of the 27 tracked AI crawler tokens outright, the most in the sector. Most retailers name none: 11 of 14 readable robots.txt files have no AI rule. Retail relies on bot defences at the edge, where news publishers wrote policy.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Etsy publishes an llms.txt titled as its official reference for AI assistants, and answers GPTBot, ClaudeBot and PerplexityBot with a challenge at the edge while serving a Chrome user agent. 6 of 14 sites publish llms.txt, against none in news. E-commerce wants to be described to agents and has not decided to let their crawlers in.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Commerce platforms are being written with agents at more than twice the rate measured in developer tooling (4.8%) and news publishing (3.3%). Across eight platforms and 6,255 commits in 90 days, 807 (12.9%) carry an AI-agent co-author trailer. Saleor leads at 51 of 131 (39%), then WooCommerce at 331 of 1,435 (23%) and Spree at 331 of 1,748 (19%). Magento has none in 948 commits.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Of the four agent-payment protocols, x402 is the most active and the only one with agent co-authored commits: 31 of 287 (11%). The Universal Commerce Protocol has 88 commits and none co-authored. The Agentic Commerce Protocol has one commit in the window and AP2 has none. The protocol with the most deployment found in this scan is not the one with the most development.","confidence":"high","measured_or_estimate":"measured"}]},{"id":"news-publishing-2026-09","sector":"News and publishing","date":"2026-09-28","aggregate":{"sites_scanned":16,"sites_with_ai_robots_rule":15,"sites_with_content_signal":1,"sites_challenging_automation":13,"sites_with_402_or_x402":1,"sites_with_llms_txt":0,"repos_scanned":10,"repos_dormant":0},"headline_findings":[{"finding":"News publishers write the most detailed AI policies of any sector scanned. 15 of 16 robots.txt files name AI crawlers, and the average site blocks 14 of the 27 tracked crawler tokens outright. CNN blocks 26, The Verge 24, The New York Times 23. In developer tooling the figure was one site in eleven.","confidence":"high","measured_or_estimate":"measured"},{"finding":"The policy is enforced at the edge, not left to robots.txt. 13 of 16 sites challenged, refused or cut off at least one of four scripted requests. Only BBC, Wired, Substack served all four user agents the page, and two of those three disallow AI crawlers in robots.txt, so their block relies on the crawler's good behaviour.","confidence":"high","measured_or_estimate":"measured"},{"finding":"8 of 16 sites gave at least one AI user agent a different answer than the Chrome user agent: The New York Times, The Washington Post, The Guardian, Financial Times, CNN, The Atlantic, The Verge, NPR. CNN answers all three AI user agents with HTTP 451, Unavailable For Legal Reasons. NPR closes the connection without an HTTP response.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Two sites serve GPTBot and refuse ClaudeBot and PerplexityBot: The Guardian and The Verge. The Verge's robots.txt also allows GPTBot while blocking 24 other tracked tokens. Which crawler gets in is decided per operator, not per category.","confidence":"high","measured_or_estimate":"measured"},{"finding":"The per-operator pattern matches publicly announced content licensing agreements between OpenAI and these publishers' owners. The scan cannot see a contract, so the link is an inference from who is served and who is refused.","confidence":"medium","measured_or_estimate":"inferred"},{"finding":"One HTTP 402 in the sector, and it cannot be paid. The Atlantic answers ClaudeBot with 402 and the message \"Please contact the site owner for access\": no price, no payment header, no x402 challenge. GPTBot and PerplexityBot get a 403 from the same site. No site in three sectors and 39 sites has offered an agent a price it could pay.","confidence":"high","measured_or_estimate":"measured"},{"finding":"The Atlantic is the only site declaring Content-Signal. It sets search=yes, ai-input=no, ai-train=no for four named crawlers: Googlebot, BingBot, DuckAssistBot and FacebookBot. They may index the page and may not use it in AI answers or training.","confidence":"high","measured_or_estimate":"measured"},{"finding":"No site publishes /llms.txt: 0 of the 8 where it could be requested. Publishers are telling agents what they may not do and are not describing themselves to them.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Publishing software is being written with agents while publishers block them. Across nine repositories and 5,526 commits in 90 days, 182 (3.3%) carry an AI-agent co-author trailer. Hugo leads at 39 of 177 (22%), every one signed by Claude. Gutenberg, the WordPress editor, has 118 of 2,074 (5.7%). The Guardian's own two repositories carry 13 agent co-authored commits, mostly Copilot, while its site refuses ClaudeBot.","confidence":"high","measured_or_estimate":"measured"},{"finding":"WordPress core could not be measured. Its repository mirrors Subversion, credits contributors with a Props line, and carries no co-author trailers of any kind, so the method has nothing to read.","confidence":"high","measured_or_estimate":"measured"}]},{"id":"devtools-registries-2026-09","sector":"Developer tooling and package registries","date":"2026-09-28","aggregate":{"sites_scanned":12,"sites_with_ai_robots_rule":1,"sites_with_content_signal":1,"sites_challenging_automation":2,"sites_with_402_or_x402":0,"sites_with_llms_txt":2,"repos_scanned":10,"repos_dormant":0},"headline_findings":[{"finding":"Agent-written code is concentrated, not spread. Across ten repositories and 8,850 commits in 90 days, 421 (4.8%) carry an AI-agent co-author trailer, and three repositories account for 371 of them: pnpm (311 of 2,035, 15%), rubygems.org (44 of 279, 16%) and the npm CLI (16 of 39, 41%). crates.io has none in 1,049 commits.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Which agent a project uses is a project-level choice. Claude 324, Copilot 43, Cursor 25, Amp 24, Codex/ChatGPT 4, other 1 commits across the sector. Claude accounts for 282 of pnpm's 311; the npm CLI's are all Copilot; rubygems.org's are mostly Amp (23) and Copilot (19).","confidence":"high","measured_or_estimate":"measured"},{"finding":"1,824 of Bun's 2,351 commits on main (78%) are authored by one account, robobun, which belongs to the project's own organization. Only 5 Bun commits carry an agent trailer, so trailer counts alone would report Bun as 0.2% AI-written.","confidence":"high","measured_or_estimate":"measured"},{"finding":"robobun is probably an AI coding agent: its pull requests carry a machine-written evidence block with an iteration counter and a fixed Problem / Fix structure. The account does not say so itself, so this is an inference. If it holds, co-author trailers understate agent-written code wherever a project runs its agent under its own account.","confidence":"medium","measured_or_estimate":"inferred"},{"finding":"One of eleven sites whose robots.txt could be requested names an AI crawler. GitHub lists GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai and PerplexityBot with a crawl delay and an allow-list of marketing pages, and blocks Bytespider entirely. No package registry has an AI rule. Maven Central serves no robots.txt at all.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Stack Overflow is the only site that treats AI user agents differently. A Chrome user agent is offered a JavaScript challenge; GPTBot, ClaudeBot and PerplexityBot get a 25-byte refusal. Its robots.txt arrives with HTTP 418, disallows everything, and declares Content-signal: search=no, ai-train=no. The other sites that answered served all four user agents the same page.","confidence":"high","measured_or_estimate":"measured"},{"finding":"npm challenges every scripted client, including one sending a Chrome user agent, and the challenge covers robots.txt. A crawler that respects robots.txt cannot read npm's policy.","confidence":"high","measured_or_estimate":"measured"},{"finding":"No agent-payable surface on any of twelve sites: no HTTP 402, no x402 and no pay-per-crawl header. Two sites, Docker Hub and GitHub, publish /llms.txt. The sector is starting to describe itself to agents and has not started charging them.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Funding is fiat and sponsorship: GitHub Sponsors, Open Collective, and foundation donation pages. pnpm is the exception, listing a crypto donations page and Drips. Five of ten repositories declare no funding file.","confidence":"high","measured_or_estimate":"measured"}]},{"id":"osint-cti-2026-09","sector":"OSINT and cyber threat intelligence","date":"2026-09-25","aggregate":{"sites_scanned":11,"sites_with_ai_robots_rule":0,"sites_with_content_signal":1,"sites_challenging_automation":1,"sites_with_402_or_x402":0,"repos_scanned":11,"repos_dormant":5},"headline_findings":[{"finding":"The agent story is inside the code, not the traffic. MISP: 360 commits in the window, 163 (45%) carrying AI-agent co-author trailers (Claude Fable 5.1, Opus 5, Opus 4.8). OpenCTI: 275 commits, about 73% bot- or agent-touched (119 Renovate, 74 AI co-authored, plus Copilot Autofix). These are the CTI platforms CERTs run on, and none of it is disclosed publicly.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Five of eleven repositories are dormant: SpiderFoot's last commit was in 2023, recon-ng and holehe in 2024.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Zero of eleven sites carry an AI-crawler rule in robots.txt. The one policy signal is Have I Been Pwned's Content-Signal header: ai-train=no, ai-input=no, search=yes. OSINT Framework's robots.txt returns HTTP 500, so it has no policy at all.","confidence":"high","measured_or_estimate":"measured"},{"finding":"Only Shodan challenges automated clients, and it blocks every non-JavaScript client equally rather than targeting AI. The other ten served GPTBot, ClaudeBot and PerplexityBot exactly what they served Chrome.","confidence":"high","measured_or_estimate":"measured"},{"finding":"No agent-payable surface anywhere: zero HTTP 402, pay-per-crawl or x402 headers across all eleven targets. Funding pages show Patreon, Donorbox, PayPal and GitHub Sponsors only; no wallet addresses in any repository funding config.","confidence":"high","measured_or_estimate":"measured"},{"finding":"The money that is moving is undisclosed fiat spend on paid agent tooling (Claude, Copilot, CodeRabbit), inferred from commit trailers. No invoice or usage figure was found, so no dollar amount is attached.","confidence":"medium","measured_or_estimate":"inferred"}]}]}