# Developer tooling and package registries: how the sector treats AI agents

> A first-hand scan of 12 sites and 10 repositories in developer tooling and package registries, measured on 2026-10-19. 1 of 12 sites name an AI crawler in robots.txt, 3 challenged or refused a scripted request, and 0 answered with a payment demand.

- Sector: Developer tooling and package registries
- Measured: 2026-10-19
- Scan id: devtools-registries-2026-10
- Publisher: Overwing Atlas (https://overwing.ai/atlas)
- Canonical page: https://overwing.ai/atlas/scans/devtools-registries-2026-10

## Figures

| Figure | Value |
|---|---|
| Sites naming an AI crawler in robots.txt | 1 of 12 |
| Sites answering with a payment demand (HTTP 402, x402, pay per crawl) | 0 of 12 |
| Highest share of AI co-authored commits in one repository (npm/cli) | 45% |
| Repositories dormant since 2024 or earlier | 0 of 10 |
| Sites that challenged, refused or cut off a scripted request | 3 of 12 |
| Sites publishing /llms.txt | 2 of 12 |
| Sites declaring Content-Signal | 0 of 12 |

Counts are out of every site scanned. A site that could not be read counts as not having the feature, so these are floors. The limits of the scan say which sites could not be read.

## Findings

Each finding is marked measured (read first-hand, with evidence) or inferred (a conclusion drawn from what was measured).

1. **Measured, high confidence.** Agent-written code in the sector rose by almost half in three weeks of overlap: 611 of 8,922 commits (6.8%) in the 90 days to 2026-10-19 carry an AI-agent co-author trailer or come from an agent account, against 421 of 8,850 (4.8%) in the window to 2026-09-28. It is still concentrated: pnpm 478 of 2,312 (20.7%, up from 15.3%), rubygems.org 74 of 295 (25.1%, up from 15.8%) and the npm CLI 17 of 38 (44.7%) account for 569 of the 611.
2. **Measured, high confidence.** Which agent a project uses is still a project-level choice. Claude 482, Amp 57, Copilot 41, Cursor 24, Codex/ChatGPT 7 commits across the sector; pnpm's are 444 Claude, rubygems.org's are mostly Amp (53 of 74), and every one of the npm CLI's is Copilot. crates.io and Deno carry no agent trailer at all in the window.
3. **Measured, high confidence.** 1,461 of Bun's 2,079 commits on main (70%) are authored by robobun, the project's own account, down from 78% in September; only 4 carry an agent trailer. Automation accounts are the larger share of machine-made commits in uv (576 of 1,336), crates.io (417 of 1,104) and Bun; agent trailers are the larger share in pnpm, rubygems.org and the npm CLI.
4. **Measured, high confidence.** Maven Central now treats AI user agents differently from a browser: a Chrome user agent is served the page, and GPTBot, ClaudeBot and PerplexityBot each get a 25-byte HTTP 403. In September all four were served. Stack Overflow is unchanged (a JavaScript challenge for Chrome, a flat refusal for the three AI agents) and npm still challenges every scripted client, robots.txt included. Three of twelve sites now challenge or refuse, up from two.
5. **Measured, high confidence.** Docker Hub is the first site in four sectors seen giving an AI user agent more than a browser gets: GPTBot, ClaudeBot and PerplexityBot received a 480 KB page with 6,871 characters of text, while the Chrome user agent received a 285 KB page with 2,345 characters. The site publishes /llms.txt and its robots.txt disallows nothing.
6. **Measured, high confidence.** GitHub's robots.txt is the only one in the sector that names AI crawlers, as in September: GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai and PerplexityBot share a group with a crawl delay of 1 and an allow-list of marketing pages, and Bytespider is listed separately. Two sites, Docker Hub and GitHub, publish /llms.txt; none answers with HTTP 402 or a payment header. pnpm has removed its crypto-donations page and Drips from FUNDING.yml since September, so the sector's one crypto funding rail is gone: funding is GitHub Sponsors, Open Collective and foundation pages, and five of ten repositories declare none.
7. **Measured, high confidence.** Across 10 repositories and 8,922 commits in the 90 days to 2026-10-19, 611 (6.8%) carry an AI-agent co-author trailer or come from an agent account. Highest shares: cli 17 of 38 (44.7%), rubygems.org 74 of 295 (25.1%), pnpm 478 of 2312 (20.7%).
8. **Measured, high confidence.** By agent family across the sector: Claude 482, Amp 57, Copilot 41, Cursor 24, Codex/ChatGPT 7 commits.
9. **Measured, high confidence.** 1 of 10 sites whose robots.txt could be read name a tracked AI crawler; 3 of 12 challenged, refused or cut off at least one of four scripted requests; 0 answered with HTTP 402 or a payment header; 2 of 9 where it could be requested publish /llms.txt.

## What was scanned

Sites (12): crates.io, Docker Hub, GitHub, GitLab, Go packages, Maven Central, MDN Web Docs, npm, NuGet, PyPI, RubyGems, Stack Overflow.

Repositories (10): bun, cli, content, crates.io, deno, pnpm, rubygems.org, uv, vite, warehouse.

## Method

Web: for each of 12 sites, fetched /robots.txt and /llms.txt once with a Chrome user agent, and the home page once each with Chrome, GPTBot, ClaudeBot and PerplexityBot user agents, 1.5 seconds apart, from one US address using a scripted HTTP client (no JavaScript, no cookies). Each response was classed as served (200 with readable text), challenged (a challenge page, a rate limit, or a 200 with no text), refused (a 4xx with no challenge), payment required (402), or reset (no HTTP response). Compared class and size of the four home page bodies; read Content-Signal in robots.txt and in response headers; looked for HTTP 402 and payment-required, x-payment and crawler-price headers. robots.txt was parsed into user-agent groups and checked against 27 tracked AI crawler tokens; a token counts as blocked when its group contains Disallow: /. Repositories: for each of 10 repositories, read every commit on the default branch from 2026-07-21 to 2026-10-19 through the GitHub commits API. A commit counts as AI co-authored when a Co-authored-by trailer carries an agent's name (Claude, Copilot, Cursor, Codex, Amp, Gemini, Devin and others) or an address an agent signs with, or when the commit author is a known agent account. A person's employer domain does not count. Automation accounts (Dependabot, Renovate, project bots) are counted separately. Funding was read from .github/FUNDING.yml in the repository or its organization. Scanner Overwing Atlas Scanner 3.0, source in the Overwing repository (scripts/atlas-scan.mts, rules in src/lib/atlas/scan-rules.ts).

## Limits of this scan

Read these before quoting a figure.

- The web client sent a Chrome user agent string but is not a browser: it runs no JavaScript and has a different TLS fingerprint. A site that challenged it may serve a real browser normally. 'Challenged' describes what a script met, not what a reader meets.
- AI user agents were sent from an ordinary address, not from the operators' published ranges, and unsigned. A site that verifies crawlers by address or signature may admit the real crawler where it refused this one, or refuse it where it admitted this one.
- Only the home page was requested. One reading per site on one day; policies served by a CDN can vary by region and over time.
- The 90-day commit window overlaps the previous scan's window where the two scans are less than 90 days apart; shares are of the window, not of the period between scans.
- Co-author trailers are voluntary. They undercount agent-written code where contributors strip them or where a project runs its agent under its own account. Only default-branch commits are counted; squash merges collapse a pull request into one commit.
- npmjs.com: robots.txt could not be read (HTTP 403, 5806 bytes, 16 chars of text, title "Just a moment...": challenged); its robots policy is unmeasured, not absent.
- stackoverflow.com: robots.txt could not be read (HTTP 403, 5790 bytes, 16 chars of text, title "Just a moment...": challenged); its robots policy is unmeasured, not absent.
- gitlab.com: /llms.txt is unmeasured (unmeasured: HTTP 403, 5748 bytes, 16 chars of text, title "Just a moment...": challenged).
- npmjs.com: /llms.txt is unmeasured (unmeasured: HTTP 403, 5800 bytes, 16 chars of text, title "Just a moment...": challenged).
- stackoverflow.com: /llms.txt is unmeasured (unmeasured: HTTP 403, 5784 bytes, 16 chars of text, title "Just a moment...": challenged).
- Compared with devtools-registries-2026-09: 25 site or repository values changed; the differences are listed in the scan's working file and summarised in the findings.

## How to cite

Overwing Atlas. "Developer tooling and package registries: field scan of 12 sites and 10 repositories." Measured 2026-10-19. https://overwing.ai/atlas/scans/devtools-registries-2026-10

Quote a figure with its date and its denominator. These are single readings on one day, and policies change.

## Data

- Headline findings, free: https://overwing.ai/api/v1/atlas/summary
- Per-target rows (Atlas Pro or Team): https://overwing.ai/api/v1/atlas/datasets/sector-scans
- License: https://overwing.ai/atlas/license