# OSINT and cyber threat intelligence: how the sector treats AI agents

> A first-hand scan of 11 sites and 11 repositories in OSINT and cyber threat intelligence, measured on 2026-10-19. 0 of 11 sites name an AI crawler in robots.txt, 0 challenged or refused a scripted request, and 0 answered with a payment demand.

- Sector: OSINT and cyber threat intelligence
- Measured: 2026-10-19
- Scan id: osint-cti-2026-10
- Publisher: Overwing Atlas (https://overwing.ai/atlas)
- Canonical page: https://overwing.ai/atlas/scans/osint-cti-2026-10

## Figures

| Figure | Value |
|---|---|
| Sites naming an AI crawler in robots.txt | 0 of 11 |
| Sites answering with a payment demand (HTTP 402, x402, pay per crawl) | 0 of 11 |
| Highest share of AI co-authored commits in one repository (MISP/MISP) | 39% |
| Repositories dormant since 2024 or earlier | 4 of 11 |
| Sites that challenged, refused or cut off a scripted request | 0 of 11 |
| Sites publishing /llms.txt | 2 of 11 |
| Sites declaring Content-Signal | 1 of 11 |

Counts are out of every site scanned. A site that could not be read counts as not having the feature, so these are floors. The limits of the scan say which sites could not be read.

## Findings

Each finding is marked measured (read first-hand, with evidence) or inferred (a conclusion drawn from what was measured).

1. **Measured, high confidence.** MISP is being written with an agent at a rate higher than any repository in the three other sectors scanned except npm's command-line client: 295 of 766 commits (38.5%) in the 90 days to 2026-10-19 carry a Claude co-author trailer, every one of them Claude (Claude Fable 5.1 106, Claude Opus 5 with 1M context 101, Opus 5.5 29, Opus 4.8 24). In September's window the figure was 163 of 360 (45%); the share fell as the commit volume doubled.
2. **Measured, high confidence.** OpenCTI: 138 of 851 commits (16.2%) are agent co-authored (Claude 87, Copilot 48, Cursor 3) and 378 more come from automation accounts, so 516 of 851 (60.6%) were touched by a bot or an agent, down from about 73% in September's window. The two platforms CERTs and SOCs run on are both agent-written in part, and neither says so anywhere but in its commit trailers.
3. **Measured, high confidence.** The rest of the sector's open tooling is quiet. theHarvester (413 commits) and subfinder (67) carry no agent trailer, and four of eleven repositories had no commit on their default branch in 90 days: Amass (last commit April 2026), SpiderFoot (November 2023), holehe and Aleph (December 2025). The recon tools a generation of analysts learned on are not being maintained, by people or by agents. Caldera moved from mitre to apache and made 10 commits.
4. **Measured, high confidence.** Every one of the 11 sites served all four user agents the page: the sector neither challenges scripted clients nor singles out AI crawlers, the only sector of four where that is true. No robots.txt names an AI crawler, unchanged since September. Have I Been Pwned remains the only site with a machine-readable AI policy (Content-Signal: ai-train=no, ai-input=no, search=yes). Shodan and Censys publish /llms.txt, so the two commercial scanners now describe themselves to agents while nobody in the sector restricts them. osintframework.com's robots.txt returned a Cloudflare worker error. No site answered with HTTP 402 or a payment header.
5. **Measured, high confidence.** Sector check figures: 11 sites, 0 with an AI robots rule, 1 with Content-Signal, 2 with /llms.txt, 0 challenging automation, 0 with 402 or x402; 11 repositories, 4 dormant.
6. **Measured, high confidence.** Across 11 repositories and 2,120 commits in the 90 days to 2026-10-19, 433 (20.4%) carry an AI-agent co-author trailer or come from an agent account. Highest shares: MISP 295 of 766 (38.5%), OpenCTI 138 of 851 (16.2%), theHarvester 0 of 413 (0.0%).
7. **Measured, high confidence.** By agent family across the sector: Claude 382, Copilot 48, Cursor 3 commits.

## What was scanned

Sites (11): Shodan, Censys, Maltego, Intelligence X, Bellingcat, Have I Been Pwned, OCCRP Aleph, OSINT Framework, IntelTechniques, Hunchly, OWASP.

Repositories (11): MISP, OpenCTI, theHarvester, subfinder, Amass, SpiderFoot, recon-ng, sherlock, holehe, Caldera, Aleph.

## Method

Web: for each of 11 sites, fetched /robots.txt and /llms.txt once with a Chrome user agent, and the home page once each with Chrome, GPTBot, ClaudeBot and PerplexityBot user agents, 1.5 seconds apart, from one US address using a scripted HTTP client (no JavaScript, no cookies). Each response was classed as served (200 with readable text), challenged (a challenge page, a rate limit, or a 200 with no text), refused (a 4xx with no challenge), payment required (402), or reset (no HTTP response). Compared class and size of the four home page bodies; read Content-Signal in robots.txt and in response headers; looked for HTTP 402 and payment-required, x-payment and crawler-price headers. robots.txt was parsed into user-agent groups and checked against 27 tracked AI crawler tokens; a token counts as blocked when its group contains Disallow: /. Repositories: for each of 11 repositories, read every commit on the default branch from 2026-07-21 to 2026-10-19 through the GitHub commits API. A commit counts as AI co-authored when a Co-authored-by trailer carries an agent's name (Claude, Copilot, Cursor, Codex, Amp, Gemini, Devin and others) or an address an agent signs with, or when the commit author is a known agent account. A person's employer domain does not count. Automation accounts (Dependabot, Renovate, project bots) are counted separately. Funding was read from .github/FUNDING.yml in the repository or its organization. Scanner Overwing Atlas Scanner 3.0, source in the Overwing repository (scripts/atlas-scan.mts, rules in src/lib/atlas/scan-rules.ts).

## Limits of this scan

Read these before quoting a figure.

- The web client sent a Chrome user agent string but is not a browser: it runs no JavaScript and has a different TLS fingerprint. A site that challenged it may serve a real browser normally. 'Challenged' describes what a script met, not what a reader meets.
- AI user agents were sent from an ordinary address, not from the operators' published ranges, and unsigned. A site that verifies crawlers by address or signature may admit the real crawler where it refused this one, or refuse it where it admitted this one.
- Only the home page was requested. One reading per site on one day; policies served by a CDN can vary by region and over time.
- The 90-day commit window overlaps the previous scan's window where the two scans are less than 90 days apart; shares are of the window, not of the period between scans.
- Co-author trailers are voluntary. They undercount agent-written code where contributors strip them or where a project runs its agent under its own account. Only default-branch commits are counted; squash merges collapse a pull request into one commit.
- osintframework.com: robots.txt could not be read (HTTP 500, 4628 bytes, 572 chars of text, title "Worker threw exception | osintframework.com | Cloudflare": refused); its robots policy is unmeasured, not absent.
- The September scan of this sector recorded its targets by name rather than by domain or repository path, so no per-target diff was computed; the comparison with September is made in the findings from the published September figures.
- Caldera is now apache/caldera; GitHub redirects mitre/caldera and the scan followed the redirect.
- A first publication of this scan on 2026-10-19 classed inteltechniques.com as challenging automation because its pages load Cloudflare's passive detection beacon; the rule was corrected, the sector was re-measured the same day, and this publication replaced the first.

## How to cite

Overwing Atlas. "OSINT and cyber threat intelligence: field scan of 11 sites and 11 repositories." Measured 2026-10-19. https://overwing.ai/atlas/scans/osint-cti-2026-10

Quote a figure with its date and its denominator. These are single readings on one day, and policies change.

## Data

- Headline findings, free: https://overwing.ai/api/v1/atlas/summary
- Per-target rows (Atlas Pro or Team): https://overwing.ai/api/v1/atlas/datasets/sector-scans
- License: https://overwing.ai/atlas/license