Field scans
Whole sectors, measured first-hand.
Each scan reads the same things on every site in a sector: what robots.txt says to AI crawlers, what the site actually serves them, whether it asks for payment, and how much of the sector’s software is written with agents. 8 sectors so far.
- 2026-10-19 · 16 sites · 10 repositoriesNews and publishingThe sector's robots.txt policies are unchanged in detail since September: 15 of 16 files name AI crawlers (Reuters, unreadable in September, names three and blocks one), and the average site still disallows 14 of the 27 tracked tokens outright. CNN blocks 25 (one fewer than in September), The Verge 24, The New York Times 23. Substack and The Wall Street Journal block none.
- 2026-10-19 · 11 sites · 11 repositoriesOSINT and cyber threat intelligenceMISP is being written with an agent at a rate higher than any repository in the three other sectors scanned except npm's command-line client: 295 of 766 commits (38.5%) in the 90 days to 2026-10-19 carry a Claude co-author trailer, every one of them Claude (Claude Fable 5.1 106, Claude Opus 5 with 1M context 101, Opus 5.5 29, Opus 4.8 24). In September's window the figure was 163 of 360 (45%); the share fell as the commit volume doubled.
- 2026-10-19 · 16 sites · 12 repositoriesE-commerce and paymentsAllbirds is still the only site in the sector, and in four sectors and 55 sites, that serves a machine-readable way to buy: a Universal Commerce Protocol manifest at /.well-known/ucp, alongside /llms.txt. Since September the same storefront has started rate-limiting the crawlers that would read it: GPTBot and ClaudeBot now receive HTTP 429 while Chrome and PerplexityBot are served the page. The manifest is published and two of the three AI agents tested cannot reach the site that publishes it.
- 2026-10-19 · 12 sites · 10 repositoriesDeveloper tooling and package registriesAgent-written code in the sector rose by almost half in three weeks of overlap: 611 of 8,922 commits (6.8%) in the 90 days to 2026-10-19 carry an AI-agent co-author trailer or come from an agent account, against 421 of 8,850 (4.8%) in the window to 2026-09-28. It is still concentrated: pnpm 478 of 2,312 (20.7%, up from 15.3%), rubygems.org 74 of 295 (25.1%, up from 15.8%) and the npm CLI 17 of 38 (44.7%) account for 569 of the 611.
- 2026-09-29 · 16 sites · 12 repositoriesE-commerceThe first surface in four sectors and 55 sites where an agent can find a machine-readable way to buy. Allbirds, a Shopify storefront, serves a Universal Commerce Protocol manifest at /.well-known/ucp. It declares cart, checkout, order and catalog capabilities, a live endpoint, and payment handlers. Asked without any credential what it offers, the endpoint listed 13 tools, among them create_checkout and complete_checkout. No purchase was attempted.
- 2026-09-28 · 16 sites · 10 repositoriesNews and publishingNews publishers write the most detailed AI policies of any sector scanned. 15 of 16 robots.txt files name AI crawlers, and the average site blocks 14 of the 27 tracked crawler tokens outright. CNN blocks 26, The Verge 24, The New York Times 23. In developer tooling the figure was one site in eleven.
- 2026-09-28 · 12 sites · 10 repositoriesDeveloper tooling and package registriesAgent-written code is concentrated, not spread. Across ten repositories and 8,850 commits in 90 days, 421 (4.8%) carry an AI-agent co-author trailer, and three repositories account for 371 of them: pnpm (311 of 2,035, 15%), rubygems.org (44 of 279, 16%) and the npm CLI (16 of 39, 41%). crates.io has none in 1,049 commits.
- 2026-09-25 · 11 sites · 11 repositoriesOSINT and cyber threat intelligenceThe agent story is inside the code, not the traffic. MISP: 360 commits in the window, 163 (45%) carrying AI-agent co-author trailers (Claude Fable 5.1, Opus 5, Opus 4.8). OpenCTI: 275 commits, about 73% bot- or agent-touched (119 Renovate, 74 AI co-authored, plus Copilot Autofix). These are the CTI platforms CERTs run on, and none of it is disclosed publicly.
Across sectors
| Sector | Measured | Sites | Name an AI crawler | Blocked a script | Payment demand | Publish llms.txt |
|---|---|---|---|---|---|---|
| News and publishing | 2026-10-19 | 16 | 15 | 13 | 1 | 0 |
| OSINT and cyber threat intelligence | 2026-10-19 | 11 | 0 | 0 | 0 | 2 |
| E-commerce and payments | 2026-10-19 | 16 | 5 | 10 | 0 | 6 |
| Developer tooling and package registries | 2026-10-19 | 12 | 1 | 3 | 0 | 2 |
| E-commerce | 2026-09-29 | 16 | 3 | 8 | 0 | 6 |
| News and publishing | 2026-09-28 | 16 | 15 | 13 | 1 | 0 |
| Developer tooling and package registries | 2026-09-28 | 12 | 1 | 2 | 0 | 2 |
| OSINT and cyber threat intelligence | 2026-09-25 | 11 | 0 | 1 | 0 | not measured |
“Not measured” means that scan did not check it, not that the answer was zero. Counts are out of every site scanned. A site that could not be read counts as not having the feature, so these are floors. The limits of the scan say which sites could not be read.