Pay-per-crawl / setup / Caddy
Block or charge AI crawlers on Caddy
Caddy can send AI crawlers to the gateway from your site block, with no plugin. Works for GPTBot, ClaudeBot, PerplexityBot, CCBot, Bytespider and 22 more AI crawlers; Googlebot and Bingbot are not affected.
What you need first: a terminal on the server that runs your site (SSH) and permission to use sudo, to edit the web server's config and add a scheduled job. If your host only gives you a control panel (cPanel, Hostinger, GoDaddy and similar), this guide isn't for you: use the WordPress plugin if your site is WordPress, or put the site behind Cloudflare (free) and follow the Cloudflare guide.
- 1Create your free routeOne field: your site. No wallet, no card.
- 2Add it to CaddyCopy one block and paste it. The code is folded below.
- 3Watch your reportSee which AI crawlers visit. Charging stays off until you turn it on.
Option 1: block them
Code2 lines
@ai_crawlers header_regexp User-Agent (?i)(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|anthropic-ai|PerplexityBot|Perplexity-User|GoogleOther|CCBot|Bytespider|Amazonbot|meta-externalagent|FacebookBot|cohere-ai|Diffbot|YouBot|MistralAI-User|DuckAssistBot|PetalBot|Timpibot|ImagesiftBot|omgili|AI2Bot|ElevenlabsCrawler|ElevenlabsBot)
respond @ai_crawlers 403Blocking also keeps you out of AI answers and citations. Pick crawlers one by one with the robots.txt generator.
Option 2: charge them per page
First create your free route (site, price, wallet); this page then fills in your route id.
Code38 lines
# Pay-per-crawl via AIPayGen (Caddy 2.6 or newer).
# Put these lines inside your site block, above your own reverse_proxy / file_server lines.
# If your site block is built from handle { } blocks, put them inside the handle block that
# serves your pages instead (at the top level they would never run).
# Replace YOUR_GATEWAY_SECRET with the gateway secret you got at sign-up.
#
# If AIPayGen is ever over capacity or failing (a 429 or 5xx answer), Caddy serves the request
# from your own site as if these lines weren't there, so an outage on our side can't hide your pages.
# This site's AI Terms (per-use terms for AI), served at /.well-known/ai-terms.
@aipaygen_terms path /.well-known/ai-terms
reverse_proxy @aipaygen_terms https://aipaygen.com {
rewrite /v1/gateway/terms/<your-route-id>
header_up Host {upstream_hostport}
@aipaygen_down status 429 5xx
handle_response @aipaygen_down {
}
}
# AI crawlers, and clients that already carry a payment, go through the gateway.
# The gateway's own fetch of a page it was paid for carries your secret and is served normally.
# robots.txt is free for every crawler: it is how they learn your rules and find your terms.
# Lines to add to it, written from your AI Terms (IETF aipref Content-Usage; training-only crawlers): https://aipaygen.com/v1/gateway/robots/<your-route-id>
@aipaygen_ai {
not header X-AIPayGen-Gateway-Secret YOUR_GATEWAY_SECRET
not path /robots.txt /.well-known/ai-terms
expression `header_regexp('User-Agent', '(?i)(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|anthropic-ai|PerplexityBot|Perplexity-User|GoogleOther|CCBot|Bytespider|Amazonbot|meta-externalagent|FacebookBot|cohere-ai|Diffbot|YouBot|MistralAI-User|DuckAssistBot|PetalBot|Timpibot|ImagesiftBot|omgili|AI2Bot|ElevenlabsCrawler|ElevenlabsBot)') || header({'Payment-Signature': '*'}) || header({'X-Payment': '*'})`
}
reverse_proxy @aipaygen_ai https://aipaygen.com {
rewrite /g/<your-route-id>{uri}
header_up Host {upstream_hostport}
@aipaygen_down status 429 5xx
handle_response @aipaygen_down {
# A payment that can't be checked is not accepted: the client is told to retry.
@aipaygen_paid expression `header({'Payment-Signature': '*'}) || header({'X-Payment': '*'})`
respond @aipaygen_paid `{"error":"PAYMENT_GATEWAY_UNAVAILABLE","message":"The payment service could not be reached. Nothing was charged. Try again shortly."}` 503
}
}Paste it inside your site block in the Caddyfile, above your own reverse_proxy or file_server lines, replace YOUR_GATEWAY_SECRET, and run caddy reload.
Details and options
If your site block is built from handle blocks, paste it inside the handle block that serves your pages. Needs Caddy 2.6 or newer (check with caddy version). If AIPayGen is ever over capacity or failing, Caddy serves crawlers from your own site as if the lines weren't there. On Caddy, every AI crawler request goes through AIPayGen (the other setups answer it on your server), which is also how your report gets its counts. In report-only mode the crawler still gets your page: AIPayGen fetches it from your site with your gateway secret and passes it on, free. To charge only training crawlers, remove the AI search and assistant names (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot, YouBot, MistralAI-User, ElevenlabsBot, PetalBot, Timpibot) from the bot list in this config.
Before anything is sold through AIPayGen, your site is checked once: this setup serves your AI Terms at /.well-known/ai-terms, which shows it is yours, so it verifies itself within about an hour of installing (we check for that file). Or add a DNS TXT record _aipaygen.yourdomain.com, a file at https://yourdomain.com/.well-known/aipaygen-verify.txt, or the tag <meta name="aipaygen-verify" content="..."> on your site, each holding the value shown on your report page; then press Check my site now in your report's settings (it asks for your gateway secret), or send POST /v1/gateway/<route>/verify with your API key.
Lost your gateway secret? Get a new one by email.
Using a page cache or CDN?
If a CDN or a cache in front of this server stores your HTML (for example a Cloudflare "Cache Everything" rule), it can answer AI crawlers with the saved copy before this check runs. Make it bypass the cache for all 27 AI crawler user agents this setup checks (listed below, one per line), or don't cache HTML at the edge. Prices are always sent with no-store, so a cache never keeps one.
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Claude-User
Claude-SearchBot
anthropic-ai
PerplexityBot
Perplexity-User
GoogleOther
CCBot
Bytespider
Amazonbot
meta-externalagent
FacebookBot
cohere-ai
Diffbot
YouBot
MistralAI-User
DuckAssistBot
PetalBot
Timpibot
ImagesiftBot
omgili
AI2Bot
ElevenlabsCrawler
ElevenlabsBotCheck it works
Report-only (the free start, no wallet):
curl -I -A GPTBot https://your-site.com/ # 200: report-only serves the page to AI crawlers too
curl -I https://your-site.com/ # 200 for everyone elseBoth answer 200 in report-only mode; then open your report: that visit shows up within about a minute or two. It may be listed as GPTBot (fake): it came from your address, not one OpenAI publishes, which is the check working. Your report's Test my setup button also checks it for you.
Charging (with a wallet):
curl -i -A GPTBot https://your-site.com/ # 402 with your price
curl -I https://your-site.com/ # 200 for everyone elseThe 402 is counted too: open your report: that visit shows up within about a minute or two. It may be listed as GPTBot (fake): it came from your address, not one OpenAI publishes, which is the check working.
Or run the free AI crawler check on your site. Other platforms: WordPress · Cloudflare · Nginx · Apache · Next.js · Vercel (not Next.js) · Netlify · Express / Node.js · Django / Python.
Show it on your site
This badge shows the date your AI Terms file last passed our check, and links to a page that lists your terms in plain words. Until your file passes, it says "not checked": check your file. Replace your-site.com with your site's address.
HTML
Code1 lines
<a href="https://aipaygen.com/ai-terms/site/your-site.com"><img src="https://aipaygen.com/badge/ai-terms.svg?site=your-site.com" alt="AI Terms checked by AIPayGen" height="20"></a>Markdown
[](https://aipaygen.com/ai-terms/site/your-site.com)It says only that the file passed our check on that date. It is not an approval, and it can't tell you whether any AI company follows the terms.
Uninstall
Remove the AIPayGen lines from your site block and run caddy reload. Your site then serves every visitor as before.
To delete your route and its data at AIPayGen too, use delete my sites and data (by the email you confirmed). Your sites and their reports: my sites.