Field scan · 2026-09-25

OSINT and cyber threat intelligence

A first-hand scan of 11 sites and 11 repositories in OSINT and cyber threat intelligence, measured on 2026-09-25. 0 of 11 sites name an AI crawler in robots.txt, 1 challenged or refused a scripted request, and 0 answered with a payment demand.

11 sites · 11 repositories · 6 findings · Markdown

Figures

0 of 11

sites with an AI-crawler rule in robots.txt

0 of 11

sites that answered with a payment demand (HTTP 402, x402, pay per crawl). A demand is not always payable: see the findings

45%

of MISP’s recent commits carry AI-agent co-author trailers, the highest share in this scan

5 of 11

repositories dormant since 2024 or earlier

1 of 11

sites that challenged, refused or cut off a scripted request

1 of 11

sites declaring Content-Signal

Counts are out of every site scanned. A site that could not be read counts as not having the feature, so these are floors. The limits of the scan say which sites could not be read.

Findings

Measured means read first-hand, with the evidence kept. Inferred means a conclusion drawn from what was measured, and never above medium confidence.

Measured · highThe agent story is inside the code, not the traffic. MISP: 360 commits in the window, 163 (45%) carrying AI-agent co-author trailers (Claude Fable 5.1, Opus 5, Opus 4.8). OpenCTI: 275 commits, about 73% bot- or agent-touched (119 Renovate, 74 AI co-authored, plus Copilot Autofix). These are the CTI platforms CERTs run on, and none of it is disclosed publicly.
Measured · highFive of eleven repositories are dormant: SpiderFoot's last commit was in 2023, recon-ng and holehe in 2024.
Measured · highZero of eleven sites carry an AI-crawler rule in robots.txt. The one policy signal is Have I Been Pwned's Content-Signal header: ai-train=no, ai-input=no, search=yes. OSINT Framework's robots.txt returns HTTP 500, so it has no policy at all.
Measured · highOnly Shodan challenges automated clients, and it blocks every non-JavaScript client equally rather than targeting AI. The other ten served GPTBot, ClaudeBot and PerplexityBot exactly what they served Chrome.
Measured · highNo agent-payable surface anywhere: zero HTTP 402, pay-per-crawl or x402 headers across all eleven targets. Funding pages show Patreon, Donorbox, PayPal and GitHub Sponsors only; no wallet addresses in any repository funding config.
Inferred · mediumThe money that is moving is undisclosed fiat spend on paid agent tooling (Claude, Copilot, CodeRabbit), inferred from commit trailers. No invoice or usage figure was found, so no dollar amount is attached.

Sites scanned

Shodan, Censys, Maltego, Intelligence X, Bellingcat, Have I Been Pwned, OCCRP Aleph, OSINT Framework, IntelTechniques, Hunchly, OWASP.

Repositories scanned

MISP, OpenCTI, theHarvester, subfinder, Amass, SpiderFoot, recon-ng, sherlock, holehe, Caldera, Aleph.

Method

First-hand measurement, not vendor statistics: robots.txt and Content-Signal reads, HTTP 402 and x402 header checks, user-agent comparisons (GPTBot, ClaudeBot, PerplexityBot versus Chrome), funding-page reads, and git commit-trailer analysis over each repository's recent commit window. No OSINT provider publishes request logs, so agent traffic to these sites cannot be measured from open sources; agent presence was measured where it is visible, in the code.

Limits of this scan

Read these before quoting a figure.

Cloudflare bot-share figures conflict across publications (57.5% versus 35.2%) because they use different measurement baskets; the spread is reported rather than a single number.
x402 cumulative volume ranges from about $15M to $50M depending on who counts; Allium's wash-trade-filtered estimate is about $1.6M per month. The spread is reported rather than the dramatic figure.

How to cite

Overwing Atlas. "OSINT and cyber threat intelligence: field scan of 11 sites and 11 repositories." Measured 2026-09-25. https://overwing.ai/atlas/scans/osint-cti-2026-09

Quote a figure with its date and its denominator. These are single readings on one day, and policies change. Headline findings are free to quote with attribution. Per-target rows are in the sector-scans dataset, under the Atlas data license.