Field scans
Whole sectors, measured first-hand.
Each scan reads the same things on every site in a sector: what robots.txt says to AI crawlers, what the site actually serves them, whether it asks for payment, and how much of the sector’s software is written with agents. 4 sectors so far.
- 2026-09-29 · 16 sites · 12 repositoriesE-commerceThe first surface in four sectors and 55 sites where an agent can find a machine-readable way to buy. Allbirds, a Shopify storefront, serves a Universal Commerce Protocol manifest at /.well-known/ucp. It declares cart, checkout, order and catalog capabilities, a live endpoint, and payment handlers. Asked without any credential what it offers, the endpoint listed 13 tools, among them create_checkout and complete_checkout. No purchase was attempted.
- 2026-09-28 · 16 sites · 10 repositoriesNews and publishingNews publishers write the most detailed AI policies of any sector scanned. 15 of 16 robots.txt files name AI crawlers, and the average site blocks 14 of the 27 tracked crawler tokens outright. CNN blocks 26, The Verge 24, The New York Times 23. In developer tooling the figure was one site in eleven.
- 2026-09-28 · 12 sites · 10 repositoriesDeveloper tooling and package registriesAgent-written code is concentrated, not spread. Across ten repositories and 8,850 commits in 90 days, 421 (4.8%) carry an AI-agent co-author trailer, and three repositories account for 371 of them: pnpm (311 of 2,035, 15%), rubygems.org (44 of 279, 16%) and the npm CLI (16 of 39, 41%). crates.io has none in 1,049 commits.
- 2026-09-25 · 11 sites · 11 repositoriesOSINT and cyber threat intelligenceThe agent story is inside the code, not the traffic. MISP: 360 commits in the window, 163 (45%) carrying AI-agent co-author trailers (Claude Fable 5.1, Opus 5, Opus 4.8). OpenCTI: 275 commits, about 73% bot- or agent-touched (119 Renovate, 74 AI co-authored, plus Copilot Autofix). These are the CTI platforms CERTs run on, and none of it is disclosed publicly.
Across sectors
| Sector | Measured | Sites | Name an AI crawler | Blocked a script | Payment demand | Publish llms.txt |
|---|---|---|---|---|---|---|
| E-commerce | 2026-09-29 | 16 | 3 | 8 | 0 | 6 |
| News and publishing | 2026-09-28 | 16 | 15 | 13 | 1 | 0 |
| Developer tooling and package registries | 2026-09-28 | 12 | 1 | 2 | 0 | 2 |
| OSINT and cyber threat intelligence | 2026-09-25 | 11 | 0 | 1 | 0 | not measured |
“Not measured” means that scan did not check it, not that the answer was zero. Counts are out of every site scanned. A site that could not be read counts as not having the feature, so these are floors. The limits of the scan say which sites could not be read.