# Developer tooling and package registries: how the sector treats AI agents

> A first-hand scan of 12 sites and 10 repositories in developer tooling and package registries, measured on 2026-09-28. 1 of 12 sites name an AI crawler in robots.txt, 2 challenged or refused a scripted request, and 0 answered with a payment demand.

- Sector: Developer tooling and package registries
- Measured: 2026-09-28
- Scan id: devtools-registries-2026-09
- Publisher: Overwing Atlas (https://overwing.ai/atlas)
- Canonical page: https://overwing.ai/atlas/scans/devtools-registries-2026-09

## Figures

| Figure | Value |
|---|---|
| Sites naming an AI crawler in robots.txt | 1 of 12 |
| Sites answering with a payment demand (HTTP 402, x402, pay per crawl) | 0 of 12 |
| Highest share of AI co-authored commits in one repository (npm/cli) | 41% |
| Repositories dormant since 2024 or earlier | 0 of 10 |
| Sites that challenged, refused or cut off a scripted request | 2 of 12 |
| Sites publishing /llms.txt | 2 of 12 |
| Sites declaring Content-Signal | 1 of 12 |

Counts are out of every site scanned. A site that could not be read counts as not having the feature, so these are floors. The limits of the scan say which sites could not be read.

## Findings

Each finding is marked measured (read first-hand, with evidence) or inferred (a conclusion drawn from what was measured).

1. **Measured, high confidence.** Agent-written code is concentrated, not spread. Across ten repositories and 8,850 commits in 90 days, 421 (4.8%) carry an AI-agent co-author trailer, and three repositories account for 371 of them: pnpm (311 of 2,035, 15%), rubygems.org (44 of 279, 16%) and the npm CLI (16 of 39, 41%). crates.io has none in 1,049 commits.
2. **Measured, high confidence.** Which agent a project uses is a project-level choice. Claude 324, Copilot 43, Cursor 25, Amp 24, Codex/ChatGPT 4, other 1 commits across the sector. Claude accounts for 282 of pnpm's 311; the npm CLI's are all Copilot; rubygems.org's are mostly Amp (23) and Copilot (19).
3. **Measured, high confidence.** 1,824 of Bun's 2,351 commits on main (78%) are authored by one account, robobun, which belongs to the project's own organization. Only 5 Bun commits carry an agent trailer, so trailer counts alone would report Bun as 0.2% AI-written.
4. **Inferred, medium confidence.** robobun is probably an AI coding agent: its pull requests carry a machine-written evidence block with an iteration counter and a fixed Problem / Fix structure. The account does not say so itself, so this is an inference. If it holds, co-author trailers understate agent-written code wherever a project runs its agent under its own account.
5. **Measured, high confidence.** One of eleven sites whose robots.txt could be requested names an AI crawler. GitHub lists GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai and PerplexityBot with a crawl delay and an allow-list of marketing pages, and blocks Bytespider entirely. No package registry has an AI rule. Maven Central serves no robots.txt at all.
6. **Measured, high confidence.** Stack Overflow is the only site that treats AI user agents differently. A Chrome user agent is offered a JavaScript challenge; GPTBot, ClaudeBot and PerplexityBot get a 25-byte refusal. Its robots.txt arrives with HTTP 418, disallows everything, and declares Content-signal: search=no, ai-train=no. The other sites that answered served all four user agents the same page.
7. **Measured, high confidence.** npm challenges every scripted client, including one sending a Chrome user agent, and the challenge covers robots.txt. A crawler that respects robots.txt cannot read npm's policy.
8. **Measured, high confidence.** No agent-payable surface on any of twelve sites: no HTTP 402, no x402 and no pay-per-crawl header. Two sites, Docker Hub and GitHub, publish /llms.txt. The sector is starting to describe itself to agents and has not started charging them.
9. **Measured, high confidence.** Funding is fiat and sponsorship: GitHub Sponsors, Open Collective, and foundation donation pages. pnpm is the exception, listing a crypto donations page and Drips. Five of ten repositories declare no funding file.

## What was scanned

Sites (12): npm, PyPI, crates.io, RubyGems, Go packages, Docker Hub, Maven Central, NuGet, GitHub, GitLab, Stack Overflow, MDN Web Docs.

Repositories (10): cli, warehouse, crates.io, rubygems.org, uv, pnpm, bun, deno, vite, content.

## Method

Web: for each of 12 sites, fetched /robots.txt and /llms.txt once with a Chrome user agent, and the home page once each with Chrome, GPTBot, ClaudeBot and PerplexityBot user agents, 1.5 seconds apart, from one US address using a scripted HTTP client (no JavaScript, no cookies). Compared status, size and SHA-256 of the four home page bodies; read Content-Signal in robots.txt and in response headers; looked for HTTP 402 and payment-required, x-payment and crawler-price headers. Repositories: for each of 10 repositories, read every commit on the default branch from 2026-06-30 to 2026-09-28 through the GitHub commits API. A commit counts as AI co-authored when a Co-authored-by trailer carries an agent's name (Claude, Copilot, Cursor, Codex, Amp and others) or an address an agent signs with (such as noreply@anthropic.com, cursoragent@cursor.com), or when the commit author is a known agent account. A person's employer domain does not count. Automation accounts (Dependabot, Renovate, project bots) are counted separately. Funding was read from .github/FUNDING.yml in the repository or its organization.

## Limits of this scan

Read these before quoting a figure.

- The web client sent a Chrome user agent string but is not a browser: it runs no JavaScript and has a different TLS fingerprint. Sites that challenge it may still serve real browsers normally.
- AI user agents were sent from an ordinary address, not from the operators' published ranges. A site that verifies crawlers by address or signature may treat the real crawler differently from what was observed.
- npmjs.com answered every request, including robots.txt, with a challenge. Its robots.txt policy and llms.txt are unmeasured, not absent. gitlab.com's llms.txt is unmeasured for the same reason.
- Co-author trailers are voluntary. They undercount agent-written code where contributors strip them or where a project runs its agent under its own account, as Bun appears to.
- An earlier pass of this scan counted people with openai.com addresses as agents. The rule was corrected before any figure was recorded, and the matcher is tested against that case.
- Only default-branch commits are counted. Squash merges collapse a pull request into one commit, so shares are of commits, not of lines or of pull requests.
- One reading per site on one day. Policies served by a CDN can vary by region and over time.

## How to cite

Overwing Atlas. "Developer tooling and package registries: field scan of 12 sites and 10 repositories." Measured 2026-09-28. https://overwing.ai/atlas/scans/devtools-registries-2026-09

Quote a figure with its date and its denominator. These are single readings on one day, and policies change.

## Data

- Headline findings, free: https://overwing.ai/api/v1/atlas/summary
- Per-target rows (Atlas Pro or Team): https://overwing.ai/api/v1/atlas/datasets/sector-scans
- License: https://overwing.ai/atlas/license