Field scan · 2026-09-28
Developer tooling and package registries
A first-hand scan of 12 sites and 10 repositories in developer tooling and package registries, measured on 2026-09-28. 1 of 12 sites name an AI crawler in robots.txt, 2 challenged or refused a scripted request, and 0 answered with a payment demand.
12 sites · 10 repositories · 9 findings · Markdown
Figures
Counts are out of every site scanned. A site that could not be read counts as not having the feature, so these are floors. The limits of the scan say which sites could not be read.
Findings
Measured means read first-hand, with the evidence kept. Inferred means a conclusion drawn from what was measured, and never above medium confidence.
- Measured · highAgent-written code is concentrated, not spread. Across ten repositories and 8,850 commits in 90 days, 421 (4.8%) carry an AI-agent co-author trailer, and three repositories account for 371 of them: pnpm (311 of 2,035, 15%), rubygems.org (44 of 279, 16%) and the npm CLI (16 of 39, 41%). crates.io has none in 1,049 commits.
- Measured · highWhich agent a project uses is a project-level choice. Claude 324, Copilot 43, Cursor 25, Amp 24, Codex/ChatGPT 4, other 1 commits across the sector. Claude accounts for 282 of pnpm's 311; the npm CLI's are all Copilot; rubygems.org's are mostly Amp (23) and Copilot (19).
- Measured · high1,824 of Bun's 2,351 commits on main (78%) are authored by one account, robobun, which belongs to the project's own organization. Only 5 Bun commits carry an agent trailer, so trailer counts alone would report Bun as 0.2% AI-written.
- Inferred · mediumrobobun is probably an AI coding agent: its pull requests carry a machine-written evidence block with an iteration counter and a fixed Problem / Fix structure. The account does not say so itself, so this is an inference. If it holds, co-author trailers understate agent-written code wherever a project runs its agent under its own account.
- Measured · highOne of eleven sites whose robots.txt could be requested names an AI crawler. GitHub lists GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai and PerplexityBot with a crawl delay and an allow-list of marketing pages, and blocks Bytespider entirely. No package registry has an AI rule. Maven Central serves no robots.txt at all.
- Measured · highStack Overflow is the only site that treats AI user agents differently. A Chrome user agent is offered a JavaScript challenge; GPTBot, ClaudeBot and PerplexityBot get a 25-byte refusal. Its robots.txt arrives with HTTP 418, disallows everything, and declares Content-signal: search=no, ai-train=no. The other sites that answered served all four user agents the same page.
- Measured · highnpm challenges every scripted client, including one sending a Chrome user agent, and the challenge covers robots.txt. A crawler that respects robots.txt cannot read npm's policy.
- Measured · highNo agent-payable surface on any of twelve sites: no HTTP 402, no x402 and no pay-per-crawl header. Two sites, Docker Hub and GitHub, publish /llms.txt. The sector is starting to describe itself to agents and has not started charging them.
- Measured · highFunding is fiat and sponsorship: GitHub Sponsors, Open Collective, and foundation donation pages. pnpm is the exception, listing a crypto donations page and Drips. Five of ten repositories declare no funding file.
Sites scanned
npm, PyPI, crates.io, RubyGems, Go packages, Docker Hub, Maven Central, NuGet, GitHub, GitLab, Stack Overflow, MDN Web Docs.
Repositories scanned
cli, warehouse, crates.io, rubygems.org, uv, pnpm, bun, deno, vite, content.
Method
Web: for each of 12 sites, fetched /robots.txt and /llms.txt once with a Chrome user agent, and the home page once each with Chrome, GPTBot, ClaudeBot and PerplexityBot user agents, 1.5 seconds apart, from one US address using a scripted HTTP client (no JavaScript, no cookies). Compared status, size and SHA-256 of the four home page bodies; read Content-Signal in robots.txt and in response headers; looked for HTTP 402 and payment-required, x-payment and crawler-price headers. Repositories: for each of 10 repositories, read every commit on the default branch from 2026-06-30 to 2026-09-28 through the GitHub commits API. A commit counts as AI co-authored when a Co-authored-by trailer carries an agent's name (Claude, Copilot, Cursor, Codex, Amp and others) or an address an agent signs with (such as noreply@anthropic.com, cursoragent@cursor.com), or when the commit author is a known agent account. A person's employer domain does not count. Automation accounts (Dependabot, Renovate, project bots) are counted separately. Funding was read from .github/FUNDING.yml in the repository or its organization.
Limits of this scan
Read these before quoting a figure.
- The web client sent a Chrome user agent string but is not a browser: it runs no JavaScript and has a different TLS fingerprint. Sites that challenge it may still serve real browsers normally.
- AI user agents were sent from an ordinary address, not from the operators' published ranges. A site that verifies crawlers by address or signature may treat the real crawler differently from what was observed.
- npmjs.com answered every request, including robots.txt, with a challenge. Its robots.txt policy and llms.txt are unmeasured, not absent. gitlab.com's llms.txt is unmeasured for the same reason.
- Co-author trailers are voluntary. They undercount agent-written code where contributors strip them or where a project runs its agent under its own account, as Bun appears to.
- An earlier pass of this scan counted people with openai.com addresses as agents. The rule was corrected before any figure was recorded, and the matcher is tested against that case.
- Only default-branch commits are counted. Squash merges collapse a pull request into one commit, so shares are of commits, not of lines or of pull requests.
- One reading per site on one day. Policies served by a CDN can vary by region and over time.
How to cite
Overwing Atlas. "Developer tooling and package registries: field scan of 12 sites and 10 repositories." Measured 2026-09-28. https://overwing.ai/atlas/scans/devtools-registries-2026-09
Quote a figure with its date and its denominator. These are single readings on one day, and policies change. Headline findings are free to quote with attribution. Per-target rows are in the sector-scans dataset, under the Atlas data license.