Overwing

Overwing Atlas

Open data on AI agents.

Who they are, where they go, what they spend. A registry of 247 crawlers, fetchers, and browser agents from 146 operators, a lookup API that says what a User-Agent string claims to be and whether the claim can be trusted, and research on agent traffic and agent spending across fiat and crypto rails.

Agents can call the lookup with no key, 10 times a day. A free key gives 100. Registry version 2026-09-24.

Who

247 agents, each with its user-agent tokens, operator, purpose, verification method, and Web Bot Auth key directory where one exists.

Where

Published traffic shares for browser agents, crawl purpose mix, crawl-to-referral ratios, and which sectors agents actually visit.

What they spend

Measured, adjusted agent spending on card rails and stablecoin rails, separated from the much larger “AI-influenced” numbers vendors publish.

Registry

The registry

Two open datasets merged and reconciled, then extended with our own classification, verification analysis, and operator research. Browse it, query it, or download it.

By purpose
  • Training / bulk crawl77
  • Autonomous / browser agent63
  • User-triggered fetcher49
  • Search / retrieval index40
  • Other / undocumented12
  • Coding agent6
By verification
  • User-agent string only (spoofable)179
  • Web Bot Auth signature57
  • Unattributable / spoofed10
  • Published IP ranges1

A user-agent string is not identity. Only signed requests can be trusted, and the registry says which operators sign.

Browser-agent traffic share
  • Perplexity Comet47%
  • Claude for Chrome24%
  • ChatGPT Atlas16%
  • ChatGPT Agent6.1%
  • Browserbase2.6%
  • Genspark2.4%
  • Browser Use Cloud1.0%
  • Googlebot28%

Jul 2026, HUMAN State of Agentic Traffic.

# What does this User-Agent claim to be, and can I trust it?
curl "https://overwing.ai/api/v1/atlas/lookup?user_agent=GPTBot/1.2" \
  -H "Authorization: Bearer $OVERWING_API_KEY"

# Browse the registry (no key needed for the public fields)
curl "https://overwing.ai/api/v1/atlas/agents?purpose=browser&limit=20"

# Download the whole registry as CSV (Atlas Pro)
curl -OJ "https://overwing.ai/api/v1/atlas/datasets/agents?format=csv" \
  -H "Authorization: Bearer $OVERWING_API_KEY"

# No account: pay $0.001 USDC per lookup over x402
curl "https://overwing.ai/api/x402/atlas/lookup?user_agent=ClaudeBot"

Field scans

Whole sectors, measured first-hand.

2 sectors so far. Latest: Developer tooling and package registries, 2026-09-28.

Developer tooling and package registries2026-09-28 · 12 sites · 10 repositories

1 of 12
sites with an AI-crawler rule in robots.txt
0 of 12
sites with any agent-payable surface (HTTP 402, x402, pay per crawl)
41%
of cli’s recent commits carry AI-agent co-author trailers, the highest share in this scan
0 of 10
repositories dormant since 2024 or earlier
  • measuredAgent-written code is concentrated, not spread. Across ten repositories and 8,850 commits in 90 days, 421 (4.8%) carry an AI-agent co-author trailer, and three repositories account for 371 of them: pnpm (311 of 2,035, 15%), rubygems.org (44 of 279, 16%) and the npm CLI (16 of 39, 41%). crates.io has none in 1,049 commits.
  • measuredWhich agent a project uses is a project-level choice. Claude 324, Copilot 43, Cursor 25, Amp 24, Codex/ChatGPT 4, other 1 commits across the sector. Claude accounts for 282 of pnpm's 311; the npm CLI's are all Copilot; rubygems.org's are mostly Amp (23) and Copilot (19).
  • measured1,824 of Bun's 2,351 commits on main (78%) are authored by one account, robobun, which belongs to the project's own organization. Only 5 Bun commits carry an agent trailer, so trailer counts alone would report Bun as 0.2% AI-written.
  • inferredrobobun is probably an AI coding agent: its pull requests carry a machine-written evidence block with an iteration counter and a fixed Problem / Fix structure. The account does not say so itself, so this is an inference. If it holds, co-author trailers understate agent-written code wherever a project runs its agent under its own account.
  • measuredOne of eleven sites whose robots.txt could be requested names an AI crawler. GitHub lists GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai and PerplexityBot with a crawl delay and an allow-list of marketing pages, and blocks Bytespider entirely. No package registry has an AI rule. Maven Central serves no robots.txt at all.
  • measuredStack Overflow is the only site that treats AI user agents differently. A Chrome user agent is offered a JavaScript challenge; GPTBot, ClaudeBot and PerplexityBot get a 25-byte refusal. Its robots.txt arrives with HTTP 418, disallows everything, and declares Content-signal: search=no, ai-train=no. The other sites that answered served all four user agents the same page.
  • measurednpm challenges every scripted client, including one sending a Chrome user agent, and the challenge covers robots.txt. A crawler that respects robots.txt cannot read npm's policy.
  • measuredNo agent-payable surface on any of twelve sites: no HTTP 402, no x402 and no pay-per-crawl header. Two sites, Docker Hub and GitHub, publish /llms.txt. The sector is starting to describe itself to agents and has not started charging them.
  • measuredFunding is fiat and sponsorship: GitHub Sponsors, Open Collective, and foundation donation pages. pnpm is the exception, listing a crypto donations page and Drips. Five of ten repositories declare no funding file.

Method: Web: for each of 12 sites, fetched /robots.txt and /llms.txt once with a Chrome user agent, and the home page once each with Chrome, GPTBot, ClaudeBot and PerplexityBot user agents, 1.5 seconds apart, from one US address using a scripted HTTP client (no JavaScript, no cookies). Compared status, size and SHA-256 of the four home page bodies; read Content-Signal in robots.txt and in response headers; looked for HTTP 402 and payment-required, x-payment and crawler-price headers. Repositories: for each of 10 repositories, read every commit on the default branch from 2026-06-30 to 2026-09-28 through the GitHub commits API. A commit counts as AI co-authored when a Co-authored-by trailer carries an agent's name (Claude, Copilot, Cursor, Codex, Amp and others) or an address an agent signs with (such as noreply@anthropic.com, cursoragent@cursor.com), or when the commit author is a known agent account. A person's employer domain does not count. Automation accounts (Dependabot, Renovate, project bots) are counted separately. Funding was read from .github/FUNDING.yml in the repository or its organization.

OSINT and cyber threat intelligence2026-09-25 · 11 sites · 11 repositories

0 of 11
sites with an AI-crawler rule in robots.txt
0 of 11
sites with any agent-payable surface (HTTP 402, x402, pay per crawl)
45%
of MISP’s recent commits carry AI-agent co-author trailers, the highest share in this scan
5 of 11
repositories dormant since 2024 or earlier
  • measuredThe agent story is inside the code, not the traffic. MISP: 360 commits in the window, 163 (45%) carrying AI-agent co-author trailers (Claude Fable 5.1, Opus 5, Opus 4.8). OpenCTI: 275 commits, about 73% bot- or agent-touched (119 Renovate, 74 AI co-authored, plus Copilot Autofix). These are the CTI platforms CERTs run on, and none of it is disclosed publicly.
  • measuredFive of eleven repositories are dormant: SpiderFoot's last commit was in 2023, recon-ng and holehe in 2024.
  • measuredZero of eleven sites carry an AI-crawler rule in robots.txt. The one policy signal is Have I Been Pwned's Content-Signal header: ai-train=no, ai-input=no, search=yes. OSINT Framework's robots.txt returns HTTP 500, so it has no policy at all.
  • measuredOnly Shodan challenges automated clients, and it blocks every non-JavaScript client equally rather than targeting AI. The other ten served GPTBot, ClaudeBot and PerplexityBot exactly what they served Chrome.
  • measuredNo agent-payable surface anywhere: zero HTTP 402, pay-per-crawl or x402 headers across all eleven targets. Funding pages show Patreon, Donorbox, PayPal and GitHub Sponsors only; no wallet addresses in any repository funding config.
  • inferredThe money that is moving is undisclosed fiat spend on paid agent tooling (Claude, Copilot, CodeRabbit), inferred from commit trailers. No invoice or usage figure was found, so no dollar amount is attached.

Method: First-hand measurement, not vendor statistics: robots.txt and Content-Signal reads, HTTP 402 and x402 header checks, user-agent comparisons (GPTBot, ClaudeBot, PerplexityBot versus Chrome), funding-page reads, and git commit-trailer analysis over each repository's recent commit window. No OSINT provider publishes request logs, so agent traffic to these sites cannot be measured from open sources; agent presence was measured where it is visible, in the code.

Per-target rows are in the sector-scans dataset; headlines are free at /api/v1/atlas/summary.

Research

Datasets and research

Every number carries its source, period, and a confidence grade. Where vendors disagree, we say so. Where nobody has measured, we say that too.

Agent registry
Pro

241 AI crawlers, fetchers, and browser agents from 152 operators: user-agent tokens, verification method, Web Bot Auth key directories, robots.txt behaviour, purpose class, evasion flags, and published traffic shares.

json · csv · 247 rows · /api/v1/atlas/datasets/agents

Browser-agent traffic shares
free

Published share of agentic traffic per browser agent, with period and source.

json · csv · 10 rows · /api/v1/atlas/datasets/traffic-shares

Operator OSINT findings
Pro

Dated findings about crawler behaviour, stealth crawling, and compliance disputes, with the implication for attribution.

json · csv · 13 rows · /api/v1/atlas/datasets/osint

Sector field scans
Pro

First-hand scans of whole sectors: which sites carry AI-crawler rules, Content-Signal, bot challenges, or any agent-payable surface, and how much of each sector's code is now AI co-authored. First scan: OSINT and cyber threat intelligence, 11 sites and 11 repositories, September 2026.

json · csv · 8 rows · /api/v1/atlas/datasets/sector-scans

Agent consumer matrix
Pro

Agent type, operators, destinations, activity, payment method, measured volume or share, source, and confidence, one row per consumer segment.

csv · 17 rows · /api/v1/atlas/datasets/agent-consumer-matrix

Fiat vs crypto agent spending
Pro

Best measured agent-initiated volume per rail, with period, source, and confidence.

csv · 7 rows · /api/v1/atlas/datasets/fiat-vs-crypto

Report: Agent Consumers, 2025 to 2026
Pro

Who AI agents are, where they go, what they do, and what they spend. Traffic, fiat rails, crypto rails, open-data inventory, and analysis, with sources and caveats.

md · /api/v1/atlas/datasets/agent-consumers-2026-09

The report summary is free at /api/v1/atlas/report?section=summary. Wallet-holding agents can buy the full report for $9 in USDC at /api/x402/atlas/report.

Pricing

Atlas pricing

Separate from the guardrails plans. Same organization, same keys, its own subscription.

Atlas Free
$0/mo
100 lookups / day
  • Public registry fields
  • Report summaries and free datasets
  • Personal and single-product use

Browse the public registry fields, read report summaries, and identify up to 100 user agents a day with any Overwing key.

Start free
Atlas Pro
$49/mo
10,000 lookups / day
  • Full registry with curated fields
  • All datasets as JSON and CSV, full reports
  • Personal and single-product use

The whole registry with our curated fields, every dataset as JSON or CSV, the full reports, and 10,000 lookups a day. Updated monthly.

Choose Pro
Atlas Team
$199/mo
100,000 lookups / day
  • Full registry with curated fields
  • All datasets as JSON and CSV, full reports
  • Internal redistribution license

Everything in Pro, 100,000 lookups a day, and a license to load the data into your own internal products and dashboards.

Choose Team
Pay per use in USDC, no account

$0.001 per lookup at /api/x402/atlas/lookup, $9 for the full report at /api/x402/atlas/report, over x402 on Base.

Licensing, in one breath

Columns sourced from Cloudflare Radar are CC BY-NC 4.0 and free to everyone with attribution. The ai.robots.txt column is MIT. Our classification, verification analysis, OSINT, and reports are licensed under the Atlas data license.

Machine-readable at /api/v1/atlas. Agents subscribe with POST /api/v1/atlas/checkout.