# OSINT and cyber threat intelligence: how the sector treats AI agents

> A first-hand scan of 11 sites and 11 repositories in OSINT and cyber threat intelligence, measured on 2026-09-25. 0 of 11 sites name an AI crawler in robots.txt, 1 challenged or refused a scripted request, and 0 answered with a payment demand.

- Sector: OSINT and cyber threat intelligence
- Measured: 2026-09-25
- Scan id: osint-cti-2026-09
- Publisher: Overwing Atlas (https://overwing.ai/atlas)
- Canonical page: https://overwing.ai/atlas/scans/osint-cti-2026-09

## Figures

| Figure | Value |
|---|---|
| Sites naming an AI crawler in robots.txt | 0 of 11 |
| Sites answering with a payment demand (HTTP 402, x402, pay per crawl) | 0 of 11 |
| Highest share of AI co-authored commits in one repository (MISP) | 45% |
| Repositories dormant since 2024 or earlier | 5 of 11 |
| Sites that challenged, refused or cut off a scripted request | 1 of 11 |
| Sites declaring Content-Signal | 1 of 11 |

Counts are out of every site scanned. A site that could not be read counts as not having the feature, so these are floors. The limits of the scan say which sites could not be read.

## Findings

Each finding is marked measured (read first-hand, with evidence) or inferred (a conclusion drawn from what was measured).

1. **Measured, high confidence.** The agent story is inside the code, not the traffic. MISP: 360 commits in the window, 163 (45%) carrying AI-agent co-author trailers (Claude Fable 5.1, Opus 5, Opus 4.8). OpenCTI: 275 commits, about 73% bot- or agent-touched (119 Renovate, 74 AI co-authored, plus Copilot Autofix). These are the CTI platforms CERTs run on, and none of it is disclosed publicly.
2. **Measured, high confidence.** Five of eleven repositories are dormant: SpiderFoot's last commit was in 2023, recon-ng and holehe in 2024.
3. **Measured, high confidence.** Zero of eleven sites carry an AI-crawler rule in robots.txt. The one policy signal is Have I Been Pwned's Content-Signal header: ai-train=no, ai-input=no, search=yes. OSINT Framework's robots.txt returns HTTP 500, so it has no policy at all.
4. **Measured, high confidence.** Only Shodan challenges automated clients, and it blocks every non-JavaScript client equally rather than targeting AI. The other ten served GPTBot, ClaudeBot and PerplexityBot exactly what they served Chrome.
5. **Measured, high confidence.** No agent-payable surface anywhere: zero HTTP 402, pay-per-crawl or x402 headers across all eleven targets. Funding pages show Patreon, Donorbox, PayPal and GitHub Sponsors only; no wallet addresses in any repository funding config.
6. **Inferred, medium confidence.** The money that is moving is undisclosed fiat spend on paid agent tooling (Claude, Copilot, CodeRabbit), inferred from commit trailers. No invoice or usage figure was found, so no dollar amount is attached.

## What was scanned

Sites (11): Shodan, Censys, Maltego, Intelligence X, Bellingcat, Have I Been Pwned, OCCRP Aleph, OSINT Framework, IntelTechniques, Hunchly, OWASP.

Repositories (11): MISP, OpenCTI, theHarvester, subfinder, Amass, SpiderFoot, recon-ng, sherlock, holehe, Caldera, Aleph.

## Method

First-hand measurement, not vendor statistics: robots.txt and Content-Signal reads, HTTP 402 and x402 header checks, user-agent comparisons (GPTBot, ClaudeBot, PerplexityBot versus Chrome), funding-page reads, and git commit-trailer analysis over each repository's recent commit window. No OSINT provider publishes request logs, so agent traffic to these sites cannot be measured from open sources; agent presence was measured where it is visible, in the code.

## Limits of this scan

Read these before quoting a figure.

- Cloudflare bot-share figures conflict across publications (57.5% versus 35.2%) because they use different measurement baskets; the spread is reported rather than a single number.
- x402 cumulative volume ranges from about $15M to $50M depending on who counts; Allium's wash-trade-filtered estimate is about $1.6M per month. The spread is reported rather than the dramatic figure.

## How to cite

Overwing Atlas. "OSINT and cyber threat intelligence: field scan of 11 sites and 11 repositories." Measured 2026-09-25. https://overwing.ai/atlas/scans/osint-cti-2026-09

Quote a figure with its date and its denominator. These are single readings on one day, and policies change.

## Data

- Headline findings, free: https://overwing.ai/api/v1/atlas/summary
- Per-target rows (Atlas Pro or Team): https://overwing.ai/api/v1/atlas/datasets/sector-scans
- License: https://overwing.ai/atlas/license