OverwingGet a key

Overwing Atlas

Open data on AI agents.

Who they are, where they go, what they spend. A registry of 350 crawlers, fetchers, and browser agents from 236 operators, a lookup API that says what a User-Agent string claims to be and whether the claim can be trusted, and research on agent traffic and agent spending across fiat and crypto rails.

Get a free key JSON overview

Agents can call the lookup with no key, 10 times a day. A free key gives 100. Registry version 2026-10-02.

Try the lookup

Paste a User-Agent. Get what it claims to be, and whether the claim holds.

Who

350 agents, each with its user-agent tokens, operator, purpose, verification method, and Web Bot Auth key directory where one exists. For every signing agent, how a withdrawn key would stop working: 18 directories give keys a validity period, 47 publish bare keys that die only when a verifier refetches. Overwing’s own fetchers, OverwingBeacon and OverwingAtlas, sign the same way.

Where

Published traffic shares for browser agents, crawl purpose mix, crawl-to-referral ratios, and which sectors agents actually visit.

What they spend

Measured, adjusted agent spending on card rails and stablecoin rails, separated from the much larger “AI-influenced” numbers vendors publish.

Registry

The registry

Two open datasets merged and reconciled, then extended with our own classification, verification analysis, and operator research. Browse it, query it, or download it.

By purpose
  • Training / bulk crawl78
  • Other / undocumented75
  • Search / retrieval index71
  • Autonomous / browser agent63
  • User-triggered fetcher57
  • Coding agent6
By verification
  • User-agent string only (spoofable)265
  • Web Bot Auth signature68
  • Unattributable / spoofed10
  • Published IP ranges7

A user-agent string is not identity. Only signed requests can be trusted, and the registry says which operators sign.

Browser-agent traffic share
  • Perplexity Comet47%
  • Claude for Chrome24%
  • ChatGPT Atlas16%
  • ChatGPT Agent6.1%
  • Browserbase2.6%
  • Genspark2.4%
  • Browser Use Cloud1.0%
  • Googlebot28%

Jul 2026, HUMAN State of Agentic Traffic.

# What does this User-Agent claim to be, and can I trust it?
curl "https://overwing.ai/api/v1/atlas/lookup?user_agent=GPTBot/1.2" \
  -H "Authorization: Bearer $OVERWING_API_KEY"

# Browse the registry (no key needed for the public fields)
curl "https://overwing.ai/api/v1/atlas/agents?purpose=browser&limit=20"

# Download the whole registry as CSV (Atlas Pro)
curl -OJ "https://overwing.ai/api/v1/atlas/datasets/agents?format=csv" \
  -H "Authorization: Bearer $OVERWING_API_KEY"

# No account: pay $0.001 USDC per lookup over x402
curl "https://overwing.ai/api/x402/atlas/lookup?user_agent=ClaudeBot"

For agent builders

Run an agent? Put it in the registry.

A site that looks up your User-Agent should learn who operates it. If your agent is not in Atlas, the answer is “no registered agent matches”, and the site has nothing to go on.

Registration is free. You name the agent, the token its User-Agent carries, and the operator’s domain. You prove you control that domain with a DNS record or a file, and the entry is published.

What it proves, exactly. That whoever registered the agent controls the operator’s domain. A User-Agent is still a string anyone can send, so the entry says so, unless you also publish a Web Bot Auth key directory on that domain. Then it is listed as signed.

Register from the dashboard, or from code with an Overwing key.

POST /api/v1/atlas/registrations        # Authorization: Bearer ow_live_...
{ "name": "AcmeBot", "operator": "Acme, Inc.", "domain": "acme.com",
  "tokens": ["AcmeBot"], "purpose": "user_fetch",
  "user_agent": "Mozilla/5.0 (compatible; AcmeBot/1.0; +https://acme.com/bot)" }

201 { "id": "areg_...", "status": "pending_verification",
      "verification": {
        "dns":  { "type": "TXT", "name": "_overwing-atlas.acme.com",
                  "value": "overwing-atlas-verification=..." },
        "http": { "url": "https://acme.com/.well-known/overwing-atlas.txt",
                  "body": "overwing-atlas-verification=..." } } }

  …publish either one…

POST /api/v1/atlas/registrations/areg_.../verify

200 { "status": "published",
      "agent": "https://overwing.ai/api/v1/atlas/agents/acmebot" }

Field scans

Whole sectors, measured first-hand.

8 sectors so far. Latest: News and publishing, 2026-10-19.

News and publishing2026-10-19 · 16 sites · 10 repositories

15 of 16
sites with an AI-crawler rule in robots.txt
1 of 16
sites that answered with a payment demand (HTTP 402, x402, pay per crawl). A demand is not always payable: see the findings
25%
of Automattic/newspack-plugin’s recent commits carry AI-agent co-author trailers, the highest share in this scan
0 of 10
repositories dormant since 2024 or earlier
  • measuredThe sector's robots.txt policies are unchanged in detail since September: 15 of 16 files name AI crawlers (Reuters, unreadable in September, names three and blocks one), and the average site still disallows 14 of the 27 tracked tokens outright. CNN blocks 25 (one fewer than in September), The Verge 24, The New York Times 23. Substack and The Wall Street Journal block none.
  • measuredEnforcement is at the edge, as in September: 13 of 16 sites challenged, refused or cut off at least one of four scripted requests, and only BBC, Substack and Wired served all four user agents the page. The same eight sites give at least one AI user agent a different answer than the Chrome user agent: CNN, Financial Times, NPR, The Atlantic, The Guardian, The New York Times, The Verge and The Washington Post. NPR drops the connection for all three AI agents while serving Chrome.
  • measuredThe Guardian and The Verge still serve GPTBot and refuse ClaudeBot and PerplexityBot, and The Verge's robots.txt still allows GPTBot while blocking 24 other tracked tokens. Which crawler gets in is decided per operator, and the pattern has held for three weeks.
  • measuredThe Atlantic still answers ClaudeBot alone with HTTP 402 and no price, payment header or x402 challenge, and remains the only site in the sector declaring Content-Signal (search=yes, ai-input=no, ai-train=no). No site publishes /llms.txt: 0 of the 13 where it could be requested. Publishers tell agents what they may not do and do not describe themselves to them.
  • measuredThe Guardian states its position in words in robots.txt comments: content is available under its terms, and any other use, including for large language models, machine learning or AI-related purposes, is not permitted; licensing@theguardian.com is given for assistance. Comments bind nobody, and the file's groups block 14 tracked tokens outright.
  • measuredPublishing software is being written with agents while publishers block them, a little more than in September: 296 of 7,363 commits (4.0%) across ten repositories in the 90 days to 2026-10-19 carry an agent trailer or come from an agent account, against 182 of 5,526 (3.3%) in September's window. Hugo leads at 45 of 186 (24.2%, all Claude). Astro's 93 of 535 (17.4%) are new and come almost entirely from factory[bot], Factory's agent account (85 commits), with no trailer on the commits themselves. Gutenberg has 125 of 2,296 (5.4%); Ghost 17 of 2,265.
  • measuredWordPress core still cannot be measured by this method: 1,083 commits on wordpress-develop in the window, none with a co-author trailer or an agent account, because the repository mirrors Subversion and credits contributors with a Props line.
  • measuredAcross 10 repositories and 7,363 commits in the 90 days to 2026-10-19, 296 (4.0%) carry an AI-agent co-author trailer or come from an agent account. Highest shares: Hugo 45 of 186 (24.2%), Astro 93 of 535 (17.4%), Gutenberg 125 of 2296 (5.4%).
  • measuredBy agent family across the sector: Claude 176, other 85, Copilot 23, Cursor 12 commits.
  • measured15 of 16 sites whose robots.txt could be read name a tracked AI crawler; 13 of 16 challenged, refused or cut off at least one of four scripted requests; 1 answered with HTTP 402 or a payment header; 0 of 13 where it could be requested publish /llms.txt.

Full scan: method, limits, and how to cite

OSINT and cyber threat intelligence2026-10-19 · 11 sites · 11 repositories

0 of 11
sites with an AI-crawler rule in robots.txt
0 of 11
sites that answered with a payment demand (HTTP 402, x402, pay per crawl). A demand is not always payable: see the findings
39%
of MISP/MISP’s recent commits carry AI-agent co-author trailers, the highest share in this scan
4 of 11
repositories dormant since 2024 or earlier
  • measuredMISP is being written with an agent at a rate higher than any repository in the three other sectors scanned except npm's command-line client: 295 of 766 commits (38.5%) in the 90 days to 2026-10-19 carry a Claude co-author trailer, every one of them Claude (Claude Fable 5.1 106, Claude Opus 5 with 1M context 101, Opus 5.5 29, Opus 4.8 24). In September's window the figure was 163 of 360 (45%); the share fell as the commit volume doubled.
  • measuredOpenCTI: 138 of 851 commits (16.2%) are agent co-authored (Claude 87, Copilot 48, Cursor 3) and 378 more come from automation accounts, so 516 of 851 (60.6%) were touched by a bot or an agent, down from about 73% in September's window. The two platforms CERTs and SOCs run on are both agent-written in part, and neither says so anywhere but in its commit trailers.
  • measuredThe rest of the sector's open tooling is quiet. theHarvester (413 commits) and subfinder (67) carry no agent trailer, and four of eleven repositories had no commit on their default branch in 90 days: Amass (last commit April 2026), SpiderFoot (November 2023), holehe and Aleph (December 2025). The recon tools a generation of analysts learned on are not being maintained, by people or by agents. Caldera moved from mitre to apache and made 10 commits.
  • measuredEvery one of the 11 sites served all four user agents the page: the sector neither challenges scripted clients nor singles out AI crawlers, the only sector of four where that is true. No robots.txt names an AI crawler, unchanged since September. Have I Been Pwned remains the only site with a machine-readable AI policy (Content-Signal: ai-train=no, ai-input=no, search=yes). Shodan and Censys publish /llms.txt, so the two commercial scanners now describe themselves to agents while nobody in the sector restricts them. osintframework.com's robots.txt returned a Cloudflare worker error. No site answered with HTTP 402 or a payment header.
  • measuredSector check figures: 11 sites, 0 with an AI robots rule, 1 with Content-Signal, 2 with /llms.txt, 0 challenging automation, 0 with 402 or x402; 11 repositories, 4 dormant.
  • measuredAcross 11 repositories and 2,120 commits in the 90 days to 2026-10-19, 433 (20.4%) carry an AI-agent co-author trailer or come from an agent account. Highest shares: MISP 295 of 766 (38.5%), OpenCTI 138 of 851 (16.2%), theHarvester 0 of 413 (0.0%).
  • measuredBy agent family across the sector: Claude 382, Copilot 48, Cursor 3 commits.

Full scan: method, limits, and how to cite

E-commerce and payments2026-10-19 · 16 sites · 12 repositories

5 of 16
sites with an AI-crawler rule in robots.txt
0 of 16
sites that answered with a payment demand (HTTP 402, x402, pay per crawl). A demand is not always payable: see the findings
37%
of saleor/saleor’s recent commits carry AI-agent co-author trailers, the highest share in this scan
2 of 12
repositories dormant since 2024 or earlier
  • measuredAllbirds is still the only site in the sector, and in four sectors and 55 sites, that serves a machine-readable way to buy: a Universal Commerce Protocol manifest at /.well-known/ucp, alongside /llms.txt. Since September the same storefront has started rate-limiting the crawlers that would read it: GPTBot and ClaudeBot now receive HTTP 429 while Chrome and PerplexityBot are served the page. The manifest is published and two of the three AI agents tested cannot reach the site that publishes it.
  • measuredRetail has tightened at the edge in three weeks. Walmart and The Home Depot served all four user agents in September and now serve Chrome while challenging GPTBot, ClaudeBot and PerplexityBot; Best Buy and eBay drop the connection for the three AI agents and serve or refuse Chrome; Etsy, which served Chrome and challenged the agents in September, now challenges all four alike; Amazon still answers every scripted client with a script-only interstitial. 10 of 16 sites challenged, refused or cut off at least one request, up from 9. Costco, Instacart, PayPal, Shopify, Stripe and Target served all four.
  • measuredTarget serves AI user agents a different page than a browser: GPTBot, ClaudeBot and PerplexityBot received a 130 KB page with more text (3,920 characters) than the 459 KB page sent to the Chrome user agent (3,185 characters). With Docker Hub in developer tooling, that makes two sites in four sectors giving agents a lighter, fuller page rather than a wall.
  • measuredeBay's Robots & Agent Policy is unchanged: checkouts are strictly for human users, and automated scraping, buy-for-me agents, LLM-driven bots and any end-to-end flow that places orders without human review are strictly prohibited, in comments no crawler is required to read; the file names 13 tracked tokens and disallows 9 entirely. AliExpress has added AI rules since September: its robots.txt now names six tracked tokens and blocks all six. Amazon still blocks 22 of 27. 5 of 15 readable files name an AI crawler; no site declares Content-Signal, and no site answered with HTTP 402 or a payment header. Six publish /llms.txt: Allbirds, Etsy, PayPal, Shopify, Stripe and Target.
  • measuredCommerce platforms remain the most agent-written code of the four sectors: 959 of 7,599 commits (12.6%) across twelve repositories in the 90 days to 2026-10-19 carry an agent trailer or come from an agent account, against 807 of 6,255 (12.9%) in September's window, and against 6.8% in developer tooling and 4.0% in publishing. WooCommerce 463 of 1,691 (27.4%, 454 of them Claude), Saleor 48 of 129 (37.2%, all Claude), Spree 247 of 1,892 (13.1%, Claude 211 and Cursor 35), Vendure 86 of 866, Shopify Hydrogen 35 of 219 (Codex 22, Claude 12). Magento and PrestaShop, the older platforms, are under 3%.
  • measuredOf the four agent-payment protocols, two have gone quiet. The Agent Payments Protocol (AP2) last committed in April 2026 and the Agentic Commerce Protocol on 2026-07-18, three days before the window opens, so both are dormant for this scan. The Universal Commerce Protocol made 104 commits, none agent co-authored; x402 made 267, 32 of them (12%) agent co-authored (Claude 16, Cursor 16), about the same share as in September (31 of 287).
  • measuredAcross 12 repositories and 7,599 commits in the 90 days to 2026-10-19, 959 (12.6%) carry an AI-agent co-author trailer or come from an agent account. Highest shares: Saleor 48 of 129 (37.2%), WooCommerce 463 of 1691 (27.4%), Shopify Hydrogen 35 of 219 (16.0%).
  • measuredBy agent family across the sector: Claude 857, Cursor 61, Codex/ChatGPT 22, Copilot 18, Amp 1 commits.
  • measured5 of 15 sites whose robots.txt could be read name a tracked AI crawler; 10 of 16 challenged, refused or cut off at least one of four scripted requests; 0 answered with HTTP 402 or a payment header; 6 of 14 where it could be requested publish /llms.txt.

Full scan: method, limits, and how to cite

Developer tooling and package registries2026-10-19 · 12 sites · 10 repositories

1 of 12
sites with an AI-crawler rule in robots.txt
0 of 12
sites that answered with a payment demand (HTTP 402, x402, pay per crawl). A demand is not always payable: see the findings
45%
of npm/cli’s recent commits carry AI-agent co-author trailers, the highest share in this scan
0 of 10
repositories dormant since 2024 or earlier
  • measuredAgent-written code in the sector rose by almost half in three weeks of overlap: 611 of 8,922 commits (6.8%) in the 90 days to 2026-10-19 carry an AI-agent co-author trailer or come from an agent account, against 421 of 8,850 (4.8%) in the window to 2026-09-28. It is still concentrated: pnpm 478 of 2,312 (20.7%, up from 15.3%), rubygems.org 74 of 295 (25.1%, up from 15.8%) and the npm CLI 17 of 38 (44.7%) account for 569 of the 611.
  • measuredWhich agent a project uses is still a project-level choice. Claude 482, Amp 57, Copilot 41, Cursor 24, Codex/ChatGPT 7 commits across the sector; pnpm's are 444 Claude, rubygems.org's are mostly Amp (53 of 74), and every one of the npm CLI's is Copilot. crates.io and Deno carry no agent trailer at all in the window.
  • measured1,461 of Bun's 2,079 commits on main (70%) are authored by robobun, the project's own account, down from 78% in September; only 4 carry an agent trailer. Automation accounts are the larger share of machine-made commits in uv (576 of 1,336), crates.io (417 of 1,104) and Bun; agent trailers are the larger share in pnpm, rubygems.org and the npm CLI.
  • measuredMaven Central now treats AI user agents differently from a browser: a Chrome user agent is served the page, and GPTBot, ClaudeBot and PerplexityBot each get a 25-byte HTTP 403. In September all four were served. Stack Overflow is unchanged (a JavaScript challenge for Chrome, a flat refusal for the three AI agents) and npm still challenges every scripted client, robots.txt included. Three of twelve sites now challenge or refuse, up from two.
  • measuredDocker Hub is the first site in four sectors seen giving an AI user agent more than a browser gets: GPTBot, ClaudeBot and PerplexityBot received a 480 KB page with 6,871 characters of text, while the Chrome user agent received a 285 KB page with 2,345 characters. The site publishes /llms.txt and its robots.txt disallows nothing.
  • measuredGitHub's robots.txt is the only one in the sector that names AI crawlers, as in September: GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai and PerplexityBot share a group with a crawl delay of 1 and an allow-list of marketing pages, and Bytespider is listed separately. Two sites, Docker Hub and GitHub, publish /llms.txt; none answers with HTTP 402 or a payment header. pnpm has removed its crypto-donations page and Drips from FUNDING.yml since September, so the sector's one crypto funding rail is gone: funding is GitHub Sponsors, Open Collective and foundation pages, and five of ten repositories declare none.
  • measuredAcross 10 repositories and 8,922 commits in the 90 days to 2026-10-19, 611 (6.8%) carry an AI-agent co-author trailer or come from an agent account. Highest shares: cli 17 of 38 (44.7%), rubygems.org 74 of 295 (25.1%), pnpm 478 of 2312 (20.7%).
  • measuredBy agent family across the sector: Claude 482, Amp 57, Copilot 41, Cursor 24, Codex/ChatGPT 7 commits.
  • measured1 of 10 sites whose robots.txt could be read name a tracked AI crawler; 3 of 12 challenged, refused or cut off at least one of four scripted requests; 0 answered with HTTP 402 or a payment header; 2 of 9 where it could be requested publish /llms.txt.

Full scan: method, limits, and how to cite

E-commerce2026-09-29 · 16 sites · 12 repositories

3 of 16
sites with an AI-crawler rule in robots.txt
0 of 16
sites that answered with a payment demand (HTTP 402, x402, pay per crawl). A demand is not always payable: see the findings
39%
of saleor/saleor’s recent commits carry AI-agent co-author trailers, the highest share in this scan
0 of 12
repositories dormant since 2024 or earlier
  • measuredThe first surface in four sectors and 55 sites where an agent can find a machine-readable way to buy. Allbirds, a Shopify storefront, serves a Universal Commerce Protocol manifest at /.well-known/ucp. It declares cart, checkout, order and catalog capabilities, a live endpoint, and payment handlers. Asked without any credential what it offers, the endpoint listed 13 tools, among them create_checkout and complete_checkout. No purchase was attempted.
  • inferredThe manifest looks like a platform default, not one merchant's project. Of three further Shopify storefronts checked for the same file, Skims and Brooklinen serve it at nearly the same size and version, and Gymshark does not. If so, the agent-ready part of e-commerce is whatever runs on Shopify, and merchants got it without building it.
  • measuredAgent checkout is arriving on card rails. The manifest's payment handlers are Google Pay, Shop Pay and card. No site in this scan returned HTTP 402 or an x402 challenge. Across four sectors the only 402 seen named no price, and no x402 challenge has been seen at all. The way in for agents is a protocol wrapped around existing payments, not a new way to pay.
  • measuredRetail is split. Walmart, Target, Costco, Instacart, Allbirds, Shopify, Stripe, PayPal served all four user agents the page. Amazon answered every one with a script-only interstitial, Wayfair and DoorDash with a challenge, and The Home Depot with a refusal. Best Buy returned no HTTP response to any request, robots.txt included.
  • measuredeBay states its position on agents in words. Its robots.txt carries a Robots and Agent Policy in comments: checkouts are strictly for human users, and buy-for-me agents, LLM-driven bots and any flow that places orders without human review are strictly prohibited. Approved integrations are sent to its API. It is the only site in this scan to address buying agents by name, and it does so in a place no crawler is obliged to read.
  • measuredAmazon's robots.txt blocks 22 of the 27 tracked AI crawler tokens outright, the most in the sector. Most retailers name none: 11 of 14 readable robots.txt files have no AI rule. Retail relies on bot defences at the edge, where news publishers wrote policy.
  • measuredEtsy publishes an llms.txt titled as its official reference for AI assistants, and answers GPTBot, ClaudeBot and PerplexityBot with a challenge at the edge while serving a Chrome user agent. 6 of 14 sites publish llms.txt, against none in news. E-commerce wants to be described to agents and has not decided to let their crawlers in.
  • measuredCommerce platforms are being written with agents at more than twice the rate measured in developer tooling (4.8%) and news publishing (3.3%). Across eight platforms and 6,255 commits in 90 days, 807 (12.9%) carry an AI-agent co-author trailer. Saleor leads at 51 of 131 (39%), then WooCommerce at 331 of 1,435 (23%) and Spree at 331 of 1,748 (19%). Magento has none in 948 commits.
  • measuredOf the four agent-payment protocols, x402 is the most active and the only one with agent co-authored commits: 31 of 287 (11%). The Universal Commerce Protocol has 88 commits and none co-authored. The Agentic Commerce Protocol has one commit in the window and AP2 has none. The protocol with the most deployment found in this scan is not the one with the most development.

Full scan: method, limits, and how to cite

News and publishing2026-09-28 · 16 sites · 10 repositories

15 of 16
sites with an AI-crawler rule in robots.txt
1 of 16
sites that answered with a payment demand (HTTP 402, x402, pay per crawl). A demand is not always payable: see the findings
22%
of gohugoio/hugo’s recent commits carry AI-agent co-author trailers, the highest share in this scan
0 of 10
repositories dormant since 2024 or earlier
  • measuredNews publishers write the most detailed AI policies of any sector scanned. 15 of 16 robots.txt files name AI crawlers, and the average site blocks 14 of the 27 tracked crawler tokens outright. CNN blocks 26, The Verge 24, The New York Times 23. In developer tooling the figure was one site in eleven.
  • measuredThe policy is enforced at the edge, not left to robots.txt. 13 of 16 sites challenged, refused or cut off at least one of four scripted requests. Only BBC, Wired, Substack served all four user agents the page, and two of those three disallow AI crawlers in robots.txt, so their block relies on the crawler's good behaviour.
  • measured8 of 16 sites gave at least one AI user agent a different answer than the Chrome user agent: The New York Times, The Washington Post, The Guardian, Financial Times, CNN, The Atlantic, The Verge, NPR. CNN answers all three AI user agents with HTTP 451, Unavailable For Legal Reasons. NPR closes the connection without an HTTP response.
  • measuredTwo sites serve GPTBot and refuse ClaudeBot and PerplexityBot: The Guardian and The Verge. The Verge's robots.txt also allows GPTBot while blocking 24 other tracked tokens. Which crawler gets in is decided per operator, not per category.
  • inferredThe per-operator pattern matches publicly announced content licensing agreements between OpenAI and these publishers' owners. The scan cannot see a contract, so the link is an inference from who is served and who is refused.
  • measuredOne HTTP 402 in the sector, and it cannot be paid. The Atlantic answers ClaudeBot with 402 and the message "Please contact the site owner for access": no price, no payment header, no x402 challenge. GPTBot and PerplexityBot get a 403 from the same site. No site in three sectors and 39 sites has offered an agent a price it could pay.
  • measuredThe Atlantic is the only site declaring Content-Signal. It sets search=yes, ai-input=no, ai-train=no for four named crawlers: Googlebot, BingBot, DuckAssistBot and FacebookBot. They may index the page and may not use it in AI answers or training.
  • measuredNo site publishes /llms.txt: 0 of the 8 where it could be requested. Publishers are telling agents what they may not do and are not describing themselves to them.
  • measuredPublishing software is being written with agents while publishers block them. Across nine repositories and 5,526 commits in 90 days, 182 (3.3%) carry an AI-agent co-author trailer. Hugo leads at 39 of 177 (22%), every one signed by Claude. Gutenberg, the WordPress editor, has 118 of 2,074 (5.7%). The Guardian's own two repositories carry 13 agent co-authored commits, mostly Copilot, while its site refuses ClaudeBot.
  • measuredWordPress core could not be measured. Its repository mirrors Subversion, credits contributors with a Props line, and carries no co-author trailers of any kind, so the method has nothing to read.

Full scan: method, limits, and how to cite

Developer tooling and package registries2026-09-28 · 12 sites · 10 repositories

1 of 12
sites with an AI-crawler rule in robots.txt
0 of 12
sites that answered with a payment demand (HTTP 402, x402, pay per crawl). A demand is not always payable: see the findings
41%
of npm/cli’s recent commits carry AI-agent co-author trailers, the highest share in this scan
0 of 10
repositories dormant since 2024 or earlier
  • measuredAgent-written code is concentrated, not spread. Across ten repositories and 8,850 commits in 90 days, 421 (4.8%) carry an AI-agent co-author trailer, and three repositories account for 371 of them: pnpm (311 of 2,035, 15%), rubygems.org (44 of 279, 16%) and the npm CLI (16 of 39, 41%). crates.io has none in 1,049 commits.
  • measuredWhich agent a project uses is a project-level choice. Claude 324, Copilot 43, Cursor 25, Amp 24, Codex/ChatGPT 4, other 1 commits across the sector. Claude accounts for 282 of pnpm's 311; the npm CLI's are all Copilot; rubygems.org's are mostly Amp (23) and Copilot (19).
  • measured1,824 of Bun's 2,351 commits on main (78%) are authored by one account, robobun, which belongs to the project's own organization. Only 5 Bun commits carry an agent trailer, so trailer counts alone would report Bun as 0.2% AI-written.
  • inferredrobobun is probably an AI coding agent: its pull requests carry a machine-written evidence block with an iteration counter and a fixed Problem / Fix structure. The account does not say so itself, so this is an inference. If it holds, co-author trailers understate agent-written code wherever a project runs its agent under its own account.
  • measuredOne of eleven sites whose robots.txt could be requested names an AI crawler. GitHub lists GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai and PerplexityBot with a crawl delay and an allow-list of marketing pages, and blocks Bytespider entirely. No package registry has an AI rule. Maven Central serves no robots.txt at all.
  • measuredStack Overflow is the only site that treats AI user agents differently. A Chrome user agent is offered a JavaScript challenge; GPTBot, ClaudeBot and PerplexityBot get a 25-byte refusal. Its robots.txt arrives with HTTP 418, disallows everything, and declares Content-signal: search=no, ai-train=no. The other sites that answered served all four user agents the same page.
  • measurednpm challenges every scripted client, including one sending a Chrome user agent, and the challenge covers robots.txt. A crawler that respects robots.txt cannot read npm's policy.
  • measuredNo agent-payable surface on any of twelve sites: no HTTP 402, no x402 and no pay-per-crawl header. Two sites, Docker Hub and GitHub, publish /llms.txt. The sector is starting to describe itself to agents and has not started charging them.
  • measuredFunding is fiat and sponsorship: GitHub Sponsors, Open Collective, and foundation donation pages. pnpm is the exception, listing a crypto donations page and Drips. Five of ten repositories declare no funding file.

Full scan: method, limits, and how to cite

OSINT and cyber threat intelligence2026-09-25 · 11 sites · 11 repositories

0 of 11
sites with an AI-crawler rule in robots.txt
0 of 11
sites that answered with a payment demand (HTTP 402, x402, pay per crawl). A demand is not always payable: see the findings
45%
of MISP’s recent commits carry AI-agent co-author trailers, the highest share in this scan
5 of 11
repositories dormant since 2024 or earlier
  • measuredThe agent story is inside the code, not the traffic. MISP: 360 commits in the window, 163 (45%) carrying AI-agent co-author trailers (Claude Fable 5.1, Opus 5, Opus 4.8). OpenCTI: 275 commits, about 73% bot- or agent-touched (119 Renovate, 74 AI co-authored, plus Copilot Autofix). These are the CTI platforms CERTs run on, and none of it is disclosed publicly.
  • measuredFive of eleven repositories are dormant: SpiderFoot's last commit was in 2023, recon-ng and holehe in 2024.
  • measuredZero of eleven sites carry an AI-crawler rule in robots.txt. The one policy signal is Have I Been Pwned's Content-Signal header: ai-train=no, ai-input=no, search=yes. OSINT Framework's robots.txt returns HTTP 500, so it has no policy at all.
  • measuredOnly Shodan challenges automated clients, and it blocks every non-JavaScript client equally rather than targeting AI. The other ten served GPTBot, ClaudeBot and PerplexityBot exactly what they served Chrome.
  • measuredNo agent-payable surface anywhere: zero HTTP 402, pay-per-crawl or x402 headers across all eleven targets. Funding pages show Patreon, Donorbox, PayPal and GitHub Sponsors only; no wallet addresses in any repository funding config.
  • inferredThe money that is moving is undisclosed fiat spend on paid agent tooling (Claude, Copilot, CodeRabbit), inferred from commit trailers. No invoice or usage figure was found, so no dollar amount is attached.

Full scan: method, limits, and how to cite

Per-target rows are in the sector-scans dataset; headlines are free at /api/v1/atlas/summary.

Research

Datasets and research

Every number carries its source, period, and a confidence grade. Where vendors disagree, we say so. Where nobody has measured, we say that too.

Agent registry
Pro

350 AI crawlers, fetchers, and browser agents from 236 operators: user-agent tokens, verification method, Web Bot Auth key directories, robots.txt behaviour, purpose class, evasion flags, and published traffic shares.

json · csv · 350 rows · /api/v1/atlas/datasets/agents

Agent sightings on overwing.ai
free

Which registered agents visited overwing.ai, by day: requests, distinct clients, what they asked for (robots.txt, llms.txt, .well-known, MCP, A2A, x402, API, pages), whether their claimed identity was confirmed by the operator's published evidence, and how they were answered. First-hand, aggregated, no addresses. Since 2026-09-28.

json · csv · 2904 rows · /api/v1/atlas/datasets/sightings

Browser-agent traffic shares
free

Published share of agentic traffic per browser agent, with period and source.

json · csv · 10 rows · /api/v1/atlas/datasets/traffic-shares

Operator OSINT findings
Pro

Dated findings about crawler behaviour, stealth crawling, and compliance disputes, with the implication for attribution.

json · csv · 13 rows · /api/v1/atlas/datasets/osint

Sector field scans
Pro

First-hand scans of whole sectors: which sites carry AI-crawler rules, Content-Signal, bot challenges, or any agent-payable surface, and how much of each sector's code is now AI co-authored. 8 scans so far: OSINT and cyber threat intelligence (11 sites, 11 repositories, 2026-09-25); Developer tooling and package registries (12 sites, 10 repositories, 2026-09-28); News and publishing (16 sites, 10 repositories, 2026-09-28); E-commerce (16 sites, 12 repositories, 2026-09-29); Developer tooling and package registries (12 sites, 10 repositories, 2026-10-19); E-commerce and payments (16 sites, 12 repositories, 2026-10-19); OSINT and cyber threat intelligence (11 sites, 11 repositories, 2026-10-19); News and publishing (16 sites, 10 repositories, 2026-10-19).

json · csv · 182 rows · /api/v1/atlas/datasets/sector-scans

Agent consumer matrix
Pro

Agent type, operators, destinations, activity, payment method, measured volume or share, source, and confidence, one row per consumer segment.

csv · 17 rows · /api/v1/atlas/datasets/agent-consumer-matrix

Fiat vs crypto agent spending
Pro

Best measured agent-initiated volume per rail, with period, source, and confidence.

csv · 7 rows · /api/v1/atlas/datasets/fiat-vs-crypto

Report: Agent Consumers, 2025 to 2026
Pro

Who AI agents are, where they go, what they do, and what they spend. Traffic, fiat rails, crypto rails, open-data inventory, and analysis, with sources and caveats.

md · /api/v1/atlas/datasets/agent-consumers-2026-09

The report summary is free at /api/v1/atlas/report?section=summary. Wallet-holding agents can buy the full report for $9 in USDC at /api/x402/atlas/report.

Pricing

Atlas pricing

Separate from the guardrails plans. Same organization, same keys, its own subscription.

Atlas Free
$0/mo
100 lookups / day
  • Public registry fields
  • Report summaries and free datasets
  • Personal and single-product use

Browse the public registry fields, read report summaries, and identify up to 100 user agents a day with any Overwing key.

Start free
Atlas Pro
$49/mo
10,000 lookups / day
  • Full registry with curated fields
  • All datasets as JSON and CSV, full reports
  • Personal and single-product use

The whole registry with our curated fields, every dataset as JSON or CSV, the full reports, and 10,000 lookups a day. Updated monthly.

Choose Pro
Atlas Team
$199/mo
100,000 lookups / day
  • Full registry with curated fields
  • All datasets as JSON and CSV, full reports
  • Internal redistribution license

Everything in Pro, 100,000 lookups a day, and a license to load the data into your own internal products and dashboards.

Choose Team
Pay per use in USDC, no account

$0.001 per lookup at /api/x402/atlas/lookup, $9 for the full report at /api/x402/atlas/report, over x402 on Base.

Licensing, in one breath

Columns sourced from Cloudflare Radar are CC BY-NC 4.0 and free to everyone with attribution. The ai.robots.txt column is MIT. Our classification, verification analysis, OSINT, and reports are licensed under the Atlas data license.

Machine-readable at /api/v1/atlas. Agents subscribe with POST /api/v1/atlas/checkout.