# Overwing Atlas field scans

> First-hand scans of whole sectors: how each one treats AI crawlers and agents, and how much of its software is written with them. 4 sectors so far.

## [E-commerce](https://overwing.ai/atlas/scans/ecommerce-2026-09)

A first-hand scan of 16 sites and 12 repositories in e-commerce, measured on 2026-09-29. 3 of 16 sites name an AI crawler in robots.txt, 8 challenged or refused a scripted request, and 0 answered with a payment demand.

Markdown: https://overwing.ai/atlas/scans/ecommerce-2026-09.md

## [News and publishing](https://overwing.ai/atlas/scans/news-publishing-2026-09)

A first-hand scan of 16 sites and 10 repositories in news and publishing, measured on 2026-09-28. 15 of 16 sites name an AI crawler in robots.txt, 13 challenged or refused a scripted request, and 1 answered with a payment demand.

Markdown: https://overwing.ai/atlas/scans/news-publishing-2026-09.md

## [Developer tooling and package registries](https://overwing.ai/atlas/scans/devtools-registries-2026-09)

A first-hand scan of 12 sites and 10 repositories in developer tooling and package registries, measured on 2026-09-28. 1 of 12 sites name an AI crawler in robots.txt, 2 challenged or refused a scripted request, and 0 answered with a payment demand.

Markdown: https://overwing.ai/atlas/scans/devtools-registries-2026-09.md

## [OSINT and cyber threat intelligence](https://overwing.ai/atlas/scans/osint-cti-2026-09)

A first-hand scan of 11 sites and 11 repositories in OSINT and cyber threat intelligence, measured on 2026-09-25. 0 of 11 sites name an AI crawler in robots.txt, 1 challenged or refused a scripted request, and 0 answered with a payment demand.

Markdown: https://overwing.ai/atlas/scans/osint-cti-2026-09.md

## Across sectors

| Sector | Measured | Sites | Name an AI crawler | Payment demand | Publish llms.txt |
|---|---|---|---|---|---|
| E-commerce | 2026-09-29 | 16 | 3 | 0 | 6 |
| News and publishing | 2026-09-28 | 16 | 15 | 1 | 0 |
| Developer tooling and package registries | 2026-09-28 | 12 | 1 | 0 | 2 |
| OSINT and cyber threat intelligence | 2026-09-25 | 11 | 0 | 0 | not measured |

"Not measured" means the scan did not check it, not that the answer was zero. Counts are out of every site scanned. A site that could not be read counts as not having the feature, so these are floors. The limits of the scan say which sites could not be read.