News and publishing

Figures

Counts are out of every site scanned. A site that could not be read counts as not having the feature, so these are floors. The limits of the scan say which sites could not be read.

Findings

Measured means read first-hand, with the evidence kept. Inferred means a conclusion drawn from what was measured, and never above medium confidence.

1. Measured · highThe sector's robots.txt policies are unchanged in detail since September: 15 of 16 files name AI crawlers (Reuters, unreadable in September, names three and blocks one), and the average site still disallows 14 of the 27 tracked tokens outright. CNN blocks 25 (one fewer than in September), The Verge 24, The New York Times 23. Substack and The Wall Street Journal block none.
2. Measured · highEnforcement is at the edge, as in September: 13 of 16 sites challenged, refused or cut off at least one of four scripted requests, and only BBC, Substack and Wired served all four user agents the page. The same eight sites give at least one AI user agent a different answer than the Chrome user agent: CNN, Financial Times, NPR, The Atlantic, The Guardian, The New York Times, The Verge and The Washington Post. NPR drops the connection for all three AI agents while serving Chrome.
3. Measured · highThe Guardian and The Verge still serve GPTBot and refuse ClaudeBot and PerplexityBot, and The Verge's robots.txt still allows GPTBot while blocking 24 other tracked tokens. Which crawler gets in is decided per operator, and the pattern has held for three weeks.
4. Measured · highThe Atlantic still answers ClaudeBot alone with HTTP 402 and no price, payment header or x402 challenge, and remains the only site in the sector declaring Content-Signal (search=yes, ai-input=no, ai-train=no). No site publishes /llms.txt: 0 of the 13 where it could be requested. Publishers tell agents what they may not do and do not describe themselves to them.
5. Measured · highThe Guardian states its position in words in robots.txt comments: content is available under its terms, and any other use, including for large language models, machine learning or AI-related purposes, is not permitted; licensing@theguardian.com is given for assistance. Comments bind nobody, and the file's groups block 14 tracked tokens outright.
6. Measured · highPublishing software is being written with agents while publishers block them, a little more than in September: 296 of 7,363 commits (4.0%) across ten repositories in the 90 days to 2026-10-19 carry an agent trailer or come from an agent account, against 182 of 5,526 (3.3%) in September's window. Hugo leads at 45 of 186 (24.2%, all Claude). Astro's 93 of 535 (17.4%) are new and come almost entirely from factory[bot], Factory's agent account (85 commits), with no trailer on the commits themselves. Gutenberg has 125 of 2,296 (5.4%); Ghost 17 of 2,265.
7. Measured · highWordPress core still cannot be measured by this method: 1,083 commits on wordpress-develop in the window, none with a co-author trailer or an agent account, because the repository mirrors Subversion and credits contributors with a Props line.
8. Measured · highAcross 10 repositories and 7,363 commits in the 90 days to 2026-10-19, 296 (4.0%) carry an AI-agent co-author trailer or come from an agent account. Highest shares: Hugo 45 of 186 (24.2%), Astro 93 of 535 (17.4%), Gutenberg 125 of 2296 (5.4%).
9. Measured · highBy agent family across the sector: Claude 176, other 85, Copilot 23, Cursor 12 commits.
10. Measured · high15 of 16 sites whose robots.txt could be read name a tracked AI crawler; 13 of 16 challenged, refused or cut off at least one of four scripted requests; 1 answered with HTTP 402 or a payment header; 0 of 13 where it could be requested publish /llms.txt.

Method

Web: for each of 16 sites, fetched /robots.txt and /llms.txt once with a Chrome user agent, and the home page once each with Chrome, GPTBot, ClaudeBot and PerplexityBot user agents, 1.5 seconds apart, from one US address using a scripted HTTP client (no JavaScript, no cookies). Each response was classed as served (200 with readable text), challenged (a challenge page, a rate limit, or a 200 with no text), refused (a 4xx with no challenge), payment required (402), or reset (no HTTP response). Compared class and size of the four home page bodies; read Content-Signal in robots.txt and in response headers; looked for HTTP 402 and payment-required, x-payment and crawler-price headers. robots.txt was parsed into user-agent groups and checked against 27 tracked AI crawler tokens; a token counts as blocked when its group contains Disallow: /. Repositories: for each of 10 repositories, read every commit on the default branch from 2026-07-21 to 2026-10-19 through the GitHub commits API. A commit counts as AI co-authored when a Co-authored-by trailer carries an agent's name (Claude, Copilot, Cursor, Codex, Amp, Gemini, Devin and others) or an address an agent signs with, or when the commit author is a known agent account. A person's employer domain does not count. Automation accounts (Dependabot, Renovate, project bots) are counted separately. Funding was read from .github/FUNDING.yml in the repository or its organization. Scanner Overwing Atlas Scanner 3.0, source in the Overwing repository (scripts/atlas-scan.mts, rules in src/lib/atlas/scan-rules.ts).

Limits of this scan

Read these before quoting a figure.

• The web client sent a Chrome user agent string but is not a browser: it runs no JavaScript and has a different TLS fingerprint. A site that challenged it may serve a real browser normally. 'Challenged' describes what a script met, not what a reader meets.
• AI user agents were sent from an ordinary address, not from the operators' published ranges, and unsigned. A site that verifies crawlers by address or signature may admit the real crawler where it refused this one, or refuse it where it admitted this one.
• Only the home page was requested. One reading per site on one day; policies served by a CDN can vary by region and over time.
• The 90-day commit window overlaps the previous scan's window where the two scans are less than 90 days apart; shares are of the window, not of the period between scans.
• Co-author trailers are voluntary. They undercount agent-written code where contributors strip them or where a project runs its agent under its own account. Only default-branch commits are counted; squash merges collapse a pull request into one commit.
• bloomberg.com: /llms.txt is unmeasured (unmeasured: HTTP 403, 427 bytes, 425 chars of text: challenged).
• theatlantic.com: /llms.txt is unmeasured (unmeasured: HTTP 403, 449907 bytes, 450 chars of text, title "Your access has been blocked – The Atlantic": refused).
• wsj.com: /llms.txt is unmeasured (unmeasured: HTTP 403, 461082 bytes, 1557 chars of text: refused).
• Compared with news-publishing-2026-09: 16 site or repository values changed; the differences are listed in the scan's working file and summarised in the findings.

How to cite

Overwing Atlas. "News and publishing: field scan of 16 sites and 10 repositories." Measured 2026-10-19. https://overwing.ai/atlas/scans/news-publishing-2026-10

Quote a figure with its date and its denominator. These are single readings on one day, and policies change. Headline findings are free to quote with attribution. Per-target rows are in the sector-scans dataset, under the Atlas data license.