Field scan · 2026-10-19

E-commerce and payments

A first-hand scan of 16 sites and 12 repositories in e-commerce and payments, measured on 2026-10-19. 5 of 16 sites name an AI crawler in robots.txt, 10 challenged or refused a scripted request, and 0 answered with a payment demand.

16 sites · 12 repositories · 9 findings · Markdown

Figures

5 of 16
sites with an AI-crawler rule in robots.txt
0 of 16
sites that answered with a payment demand (HTTP 402, x402, pay per crawl). A demand is not always payable: see the findings
37%
of saleor/saleor’s recent commits carry AI-agent co-author trailers, the highest share in this scan
2 of 12
repositories dormant since 2024 or earlier
10 of 16
sites that challenged, refused or cut off a scripted request
6 of 16
sites publishing /llms.txt
1 of 16
sites publishing a manifest an agent can transact through
0 of 16
sites declaring Content-Signal

Counts are out of every site scanned. A site that could not be read counts as not having the feature, so these are floors. The limits of the scan say which sites could not be read.

Findings

Measured means read first-hand, with the evidence kept. Inferred means a conclusion drawn from what was measured, and never above medium confidence.

  1. Measured · highAllbirds is still the only site in the sector, and in four sectors and 55 sites, that serves a machine-readable way to buy: a Universal Commerce Protocol manifest at /.well-known/ucp, alongside /llms.txt. Since September the same storefront has started rate-limiting the crawlers that would read it: GPTBot and ClaudeBot now receive HTTP 429 while Chrome and PerplexityBot are served the page. The manifest is published and two of the three AI agents tested cannot reach the site that publishes it.
  2. Measured · highRetail has tightened at the edge in three weeks. Walmart and The Home Depot served all four user agents in September and now serve Chrome while challenging GPTBot, ClaudeBot and PerplexityBot; Best Buy and eBay drop the connection for the three AI agents and serve or refuse Chrome; Etsy, which served Chrome and challenged the agents in September, now challenges all four alike; Amazon still answers every scripted client with a script-only interstitial. 10 of 16 sites challenged, refused or cut off at least one request, up from 9. Costco, Instacart, PayPal, Shopify, Stripe and Target served all four.
  3. Measured · highTarget serves AI user agents a different page than a browser: GPTBot, ClaudeBot and PerplexityBot received a 130 KB page with more text (3,920 characters) than the 459 KB page sent to the Chrome user agent (3,185 characters). With Docker Hub in developer tooling, that makes two sites in four sectors giving agents a lighter, fuller page rather than a wall.
  4. Measured · higheBay's Robots & Agent Policy is unchanged: checkouts are strictly for human users, and automated scraping, buy-for-me agents, LLM-driven bots and any end-to-end flow that places orders without human review are strictly prohibited, in comments no crawler is required to read; the file names 13 tracked tokens and disallows 9 entirely. AliExpress has added AI rules since September: its robots.txt now names six tracked tokens and blocks all six. Amazon still blocks 22 of 27. 5 of 15 readable files name an AI crawler; no site declares Content-Signal, and no site answered with HTTP 402 or a payment header. Six publish /llms.txt: Allbirds, Etsy, PayPal, Shopify, Stripe and Target.
  5. Measured · highCommerce platforms remain the most agent-written code of the four sectors: 959 of 7,599 commits (12.6%) across twelve repositories in the 90 days to 2026-10-19 carry an agent trailer or come from an agent account, against 807 of 6,255 (12.9%) in September's window, and against 6.8% in developer tooling and 4.0% in publishing. WooCommerce 463 of 1,691 (27.4%, 454 of them Claude), Saleor 48 of 129 (37.2%, all Claude), Spree 247 of 1,892 (13.1%, Claude 211 and Cursor 35), Vendure 86 of 866, Shopify Hydrogen 35 of 219 (Codex 22, Claude 12). Magento and PrestaShop, the older platforms, are under 3%.
  6. Measured · highOf the four agent-payment protocols, two have gone quiet. The Agent Payments Protocol (AP2) last committed in April 2026 and the Agentic Commerce Protocol on 2026-07-18, three days before the window opens, so both are dormant for this scan. The Universal Commerce Protocol made 104 commits, none agent co-authored; x402 made 267, 32 of them (12%) agent co-authored (Claude 16, Cursor 16), about the same share as in September (31 of 287).
  7. Measured · highAcross 12 repositories and 7,599 commits in the 90 days to 2026-10-19, 959 (12.6%) carry an AI-agent co-author trailer or come from an agent account. Highest shares: Saleor 48 of 129 (37.2%), WooCommerce 463 of 1691 (27.4%), Shopify Hydrogen 35 of 219 (16.0%).
  8. Measured · highBy agent family across the sector: Claude 857, Cursor 61, Codex/ChatGPT 22, Copilot 18, Amp 1 commits.
  9. Measured · high5 of 15 sites whose robots.txt could be read name a tracked AI crawler; 10 of 16 challenged, refused or cut off at least one of four scripted requests; 0 answered with HTTP 402 or a payment header; 6 of 14 where it could be requested publish /llms.txt.

Sites scanned

AliExpress, Allbirds (Shopify storefront), Amazon, Best Buy, Costco, DoorDash, eBay, Etsy, The Home Depot, Instacart, PayPal, Shopify, Stripe, Target, Walmart, Wayfair.

Repositories scanned

Agentic Commerce Protocol, Agent Payments Protocol (AP2), Magento, Medusa, PrestaShop, Saleor, Shopify Hydrogen, Spree, Universal Commerce Protocol, Vendure, WooCommerce, x402.

Method

Web: for each of 16 sites, fetched /robots.txt and /llms.txt once with a Chrome user agent, and the home page once each with Chrome, GPTBot, ClaudeBot and PerplexityBot user agents, 1.5 seconds apart, from one US address using a scripted HTTP client (no JavaScript, no cookies). Each response was classed as served (200 with readable text), challenged (a challenge page, a rate limit, or a 200 with no text), refused (a 4xx with no challenge), payment required (402), or reset (no HTTP response). Compared class and size of the four home page bodies; read Content-Signal in robots.txt and in response headers; looked for HTTP 402 and payment-required, x-payment and crawler-price headers. robots.txt was parsed into user-agent groups and checked against 27 tracked AI crawler tokens; a token counts as blocked when its group contains Disallow: /. Checked /.well-known/ucp, /.well-known/agent-card.json, /.well-known/agent.json, /.well-known/mcp.json for a JSON manifest. Repositories: for each of 12 repositories, read every commit on the default branch from 2026-07-21 to 2026-10-19 through the GitHub commits API. A commit counts as AI co-authored when a Co-authored-by trailer carries an agent's name (Claude, Copilot, Cursor, Codex, Amp, Gemini, Devin and others) or an address an agent signs with, or when the commit author is a known agent account. A person's employer domain does not count. Automation accounts (Dependabot, Renovate, project bots) are counted separately. Funding was read from .github/FUNDING.yml in the repository or its organization. Scanner Overwing Atlas Scanner 3.0, source in the Overwing repository (scripts/atlas-scan.mts, rules in src/lib/atlas/scan-rules.ts).

Limits of this scan

Read these before quoting a figure.

  • The web client sent a Chrome user agent string but is not a browser: it runs no JavaScript and has a different TLS fingerprint. A site that challenged it may serve a real browser normally. 'Challenged' describes what a script met, not what a reader meets.
  • AI user agents were sent from an ordinary address, not from the operators' published ranges, and unsigned. A site that verifies crawlers by address or signature may admit the real crawler where it refused this one, or refuse it where it admitted this one.
  • Only the home page was requested. One reading per site on one day; policies served by a CDN can vary by region and over time.
  • The 90-day commit window overlaps the previous scan's window where the two scans are less than 90 days apart; shares are of the window, not of the period between scans.
  • Co-author trailers are voluntary. They undercount agent-written code where contributors strip them or where a project runs its agent under its own account. Only default-branch commits are counted; squash merges collapse a pull request into one commit.
  • doordash.com: robots.txt could not be read (HTTP 403, 5810 bytes, 16 chars of text, title "Just a moment...": challenged); its robots policy is unmeasured, not absent.
  • amazon.com: /llms.txt is unmeasured (unmeasured: HTTP 503, 2671 bytes, 28 chars of text, title "Sorry! Something went wrong!": refused).
  • doordash.com: /llms.txt is unmeasured (unmeasured: HTTP 403, 5804 bytes, 16 chars of text, title "Just a moment...": challenged).
  • Compared with ecommerce-2026-09: 31 site or repository values changed; the differences are listed in the scan's working file and summarised in the findings.

How to cite

Overwing Atlas. "E-commerce and payments: field scan of 16 sites and 12 repositories." Measured 2026-10-19. https://overwing.ai/atlas/scans/ecommerce-2026-10

Quote a figure with its date and its denominator. These are single readings on one day, and policies change. Headline findings are free to quote with attribution. Per-target rows are in the sector-scans dataset, under the Atlas data license.

Other sectors