AI crawlers / report

AI crawler report for nextdoor.com

0 of 29

AI crawlers can read this site for free. Its robots.txt blocks the other 29.

AI training crawlers reading free0 of 11
Charges AI crawlersNo
AI license (RSL)None
AI TermsNone. What's this?

robots.txt: found.

Fix it in five minutes

Charge AI crawlers per pageBlock AI training onlyPublish an AI license (RSL)

Blocking with robots.txt is free and works for crawlers that respect it. Charging lets the ones that pay in. A license tells every AI company your terms in a standard format. They work together.

AI training crawlers: 0 of 11 read for free

Copy your pages to train AI models. Blocking them costs you nothing in traffic: training doesn't send visitors. This is the group most sites block or charge.

GPTBot: blockedClaudeBot: blockedanthropic-ai: blockedGoogle-Extended: blockedCCBot: blockedBytespider: blockedApplebot-Extended: blockedmeta-externalagent: blockedFacebookBot: blockedAI2Bot: blockedElevenlabsCrawler: blocked
The robots.txt rule behind each one
CrawlerOperatorStatusRule
GPTBotOpenAIBlockednamed
ClaudeBotAnthropicBlockeddefault (*)
anthropic-aiAnthropicBlockeddefault (*)
Google-ExtendedGoogleBlockeddefault (*)
CCBotCommon CrawlBlockeddefault (*)
BytespiderByteDanceBlockeddefault (*)
Applebot-ExtendedAppleBlockeddefault (*)
meta-externalagentMetaBlockeddefault (*)
FacebookBotMetaBlockeddefault (*)
AI2BotAllen Institute for AIBlockeddefault (*)
ElevenlabsCrawlerElevenLabsBlockeddefault (*)

AI search crawlers: 0 of 6 read for free

Index your pages so AI search can cite and link you. Blocking them can keep you out of AI answers, which do send some visitors. Most sites allow these.

OAI-SearchBot: blockedClaude-SearchBot: blockedPerplexityBot: blockedYouBot: blockedPetalBot: blockedTimpibot: blocked
The robots.txt rule behind each one
CrawlerOperatorStatusRule
OAI-SearchBotOpenAIBlockeddefault (*)
Claude-SearchBotAnthropicBlockeddefault (*)
PerplexityBotPerplexityBlockeddefault (*)
YouBotYou.comBlockeddefault (*)
PetalBotHuaweiBlockeddefault (*)
TimpibotTimpiBlockeddefault (*)

On-demand fetchers: 0 of 6 read for free

Fetch a page live when someone asks an assistant about it. These act for a person in the moment. Blocking them means assistants can't read your page when asked.

ChatGPT-User: blockedClaude-User: blockedPerplexity-User: blockedMistralAI-User: blockedDuckAssistBot: blockedElevenlabsBot: blocked
The robots.txt rule behind each one
CrawlerOperatorStatusRule
ChatGPT-UserOpenAIBlockeddefault (*)
Claude-UserAnthropicBlockeddefault (*)
Perplexity-UserPerplexityBlockeddefault (*)
MistralAI-UserMistral AIBlockeddefault (*)
DuckAssistBotDuckDuckGoBlockeddefault (*)
ElevenlabsBotElevenLabsBlockeddefault (*)

Mixed-use crawlers: 0 of 6 read for free

Collect pages for several uses, including AI. Judge these one by one on their pages.

GoogleOther: blockedAmazonbot: blockedcohere-ai: blockedDiffbot: blockedImagesiftBot: blockedomgili: blocked
The robots.txt rule behind each one
CrawlerOperatorStatusRule
GoogleOtherGoogleBlockeddefault (*)
AmazonbotAmazonBlockeddefault (*)
cohere-aiCohereBlockeddefault (*)
DiffbotDiffbotBlockeddefault (*)
ImagesiftBotImageSiftBlockeddefault (*)
omgiliWebz.ioBlockeddefault (*)

Share or show it

Share on X

AI crawler status for nextdoor.com

Code1 lines
<a href="https://aipaygen.com/check/nextdoor.com"><img src="https://aipaygen.com/badge/nextdoor.com.svg" alt="AI crawler status for nextdoor.com"></a>

Checked from robots.txt; results are kept for up to 6 hours. A crawler that ignores robots.txt, or pretends to be a browser, isn't stopped by it; pay-per-crawl also catches AI crawlers by the addresses their companies publish. Verify a bot from your logs, or check another site on the AI crawler list.