AI crawlers / report

AI crawler report for yale.edu

29 of 29

AI crawlers can read this site for free.

AI training crawlers reading free11 of 11
Charges AI crawlersNo
AI license (RSL)None
AI TermsNone. What's this?

robots.txt: not found (status 526).

Fix it in five minutes

Charge AI crawlers per pageBlock AI training onlyPublish an AI license (RSL)

Blocking with robots.txt is free and works for crawlers that respect it. Charging lets the ones that pay in. A license tells every AI company your terms in a standard format. They work together.

AI training crawlers: 11 of 11 read for free

Copy your pages to train AI models. Blocking them costs you nothing in traffic: training doesn't send visitors. This is the group most sites block or charge.

GPTBot: reads for freeClaudeBot: reads for freeanthropic-ai: reads for freeGoogle-Extended: reads for freeCCBot: reads for freeBytespider: reads for freeApplebot-Extended: reads for freemeta-externalagent: reads for freeFacebookBot: reads for freeAI2Bot: reads for freeElevenlabsCrawler: reads for free
The robots.txt rule behind each one
CrawlerOperatorStatusRule
GPTBotOpenAIReads for freeno robots.txt
ClaudeBotAnthropicReads for freeno robots.txt
anthropic-aiAnthropicReads for freeno robots.txt
Google-ExtendedGoogleReads for freeno robots.txt
CCBotCommon CrawlReads for freeno robots.txt
BytespiderByteDanceReads for freeno robots.txt
Applebot-ExtendedAppleReads for freeno robots.txt
meta-externalagentMetaReads for freeno robots.txt
FacebookBotMetaReads for freeno robots.txt
AI2BotAllen Institute for AIReads for freeno robots.txt
ElevenlabsCrawlerElevenLabsReads for freeno robots.txt

AI search crawlers: 6 of 6 read for free

Index your pages so AI search can cite and link you. Blocking them can keep you out of AI answers, which do send some visitors. Most sites allow these.

OAI-SearchBot: reads for freeClaude-SearchBot: reads for freePerplexityBot: reads for freeYouBot: reads for freePetalBot: reads for freeTimpibot: reads for free
The robots.txt rule behind each one
CrawlerOperatorStatusRule
OAI-SearchBotOpenAIReads for freeno robots.txt
Claude-SearchBotAnthropicReads for freeno robots.txt
PerplexityBotPerplexityReads for freeno robots.txt
YouBotYou.comReads for freeno robots.txt
PetalBotHuaweiReads for freeno robots.txt
TimpibotTimpiReads for freeno robots.txt

On-demand fetchers: 6 of 6 read for free

Fetch a page live when someone asks an assistant about it. These act for a person in the moment. Blocking them means assistants can't read your page when asked.

ChatGPT-User: reads for freeClaude-User: reads for freePerplexity-User: reads for freeMistralAI-User: reads for freeDuckAssistBot: reads for freeElevenlabsBot: reads for free
The robots.txt rule behind each one
CrawlerOperatorStatusRule
ChatGPT-UserOpenAIReads for freeno robots.txt
Claude-UserAnthropicReads for freeno robots.txt
Perplexity-UserPerplexityReads for freeno robots.txt
MistralAI-UserMistral AIReads for freeno robots.txt
DuckAssistBotDuckDuckGoReads for freeno robots.txt
ElevenlabsBotElevenLabsReads for freeno robots.txt

Mixed-use crawlers: 6 of 6 read for free

Collect pages for several uses, including AI. Judge these one by one on their pages.

GoogleOther: reads for freeAmazonbot: reads for freecohere-ai: reads for freeDiffbot: reads for freeImagesiftBot: reads for freeomgili: reads for free
The robots.txt rule behind each one
CrawlerOperatorStatusRule
GoogleOtherGoogleReads for freeno robots.txt
AmazonbotAmazonReads for freeno robots.txt
cohere-aiCohereReads for freeno robots.txt
DiffbotDiffbotReads for freeno robots.txt
ImagesiftBotImageSiftReads for freeno robots.txt
omgiliWebz.ioReads for freeno robots.txt

Share or show it

Share on X

AI crawler status for yale.edu

Code1 lines
<a href="https://aipaygen.com/check/yale.edu"><img src="https://aipaygen.com/badge/yale.edu.svg" alt="AI crawler status for yale.edu"></a>

Checked from robots.txt; results are kept for up to 6 hours. A crawler that ignores robots.txt, or pretends to be a browser, isn't stopped by it; pay-per-crawl also catches AI crawlers by the addresses their companies publish. Verify a bot from your logs, or check another site on the AI crawler list.