AI crawlers / report
AI crawler report for wsj.com
24 of 29 AI crawlers are blocked; the rest read the site for free. robots.txt: found. It doesn't charge AI crawlers.
| AI training crawlers reading for free | 2 of 11 |
|---|---|
| Charges AI crawlers (HTTP 402) | No |
| AI license (RSL) | None published |
Fix it in five minutes
Blocking with robots.txt is free and works for crawlers that respect it. Charging lets the ones that pay in. A license tells every AI company your terms in a standard format. They work together.
AI training crawlers: 2 of 11 read for free
Copy your pages to train AI models. Blocking them costs you nothing in traffic: training doesn't send visitors. This is the group most sites block or charge.
| Crawler | Operator | Status | Rule |
|---|---|---|---|
GPTBot | OpenAI | Reads for free | named |
ClaudeBot | Anthropic | Blocked | default (*) |
anthropic-ai | Anthropic | Blocked | default (*) |
Google-Extended | Blocked | default (*) | |
CCBot | Common Crawl | Blocked | default (*) |
Bytespider | ByteDance | Blocked | default (*) |
Applebot-Extended | Apple | Reads for free | named |
meta-externalagent | Meta | Blocked | default (*) |
FacebookBot | Meta | Blocked | default (*) |
AI2Bot | Allen Institute for AI | Blocked | default (*) |
ElevenlabsCrawler | ElevenLabs | Blocked | default (*) |
AI search crawlers: 1 of 6 read for free
Index your pages so AI search can cite and link you. Blocking them can keep you out of AI answers, which do send some visitors. Most sites allow these.
| Crawler | Operator | Status | Rule |
|---|---|---|---|
OAI-SearchBot | OpenAI | Reads for free | named |
Claude-SearchBot | Anthropic | Blocked | default (*) |
PerplexityBot | Perplexity | Blocked | default (*) |
YouBot | You.com | Blocked | default (*) |
PetalBot | Huawei | Blocked | default (*) |
Timpibot | Timpi | Blocked | default (*) |
On-demand fetchers: 1 of 6 read for free
Fetch a page live when someone asks an assistant about it. These act for a person in the moment. Blocking them means assistants can't read your page when asked.
| Crawler | Operator | Status | Rule |
|---|---|---|---|
ChatGPT-User | OpenAI | Reads for free | named |
Claude-User | Anthropic | Blocked | default (*) |
Perplexity-User | Perplexity | Blocked | default (*) |
MistralAI-User | Mistral AI | Blocked | default (*) |
DuckAssistBot | DuckDuckGo | Blocked | default (*) |
ElevenlabsBot | ElevenLabs | Blocked | default (*) |
Mixed-use crawlers: 1 of 6 read for free
Collect pages for several uses, including AI. Judge these one by one on their pages.
| Crawler | Operator | Status | Rule |
|---|---|---|---|
GoogleOther | Reads for free | named | |
Amazonbot | Amazon | Blocked | default (*) |
cohere-ai | Cohere | Blocked | default (*) |
Diffbot | Diffbot | Blocked | default (*) |
ImagesiftBot | ImageSift | Blocked | default (*) |
omgili | Webz.io | Blocked | default (*) |
Share or show it
<a href="https://aipaygen.com/check/wsj.com"><img src="https://aipaygen.com/badge/wsj.com.svg" alt="AI crawler status for wsj.com"></a>Checked from robots.txt; results are kept for up to 6 hours. A crawler that ignores robots.txt, or pretends to be a browser, isn't stopped by it; pay-per-crawl also catches AI crawlers by the addresses their companies publish. Verify a bot from your logs, or check another site on the AI crawler list.