Pay-per-crawl / setup / Nginx
Block or charge AI crawlers on Nginx
Nginx can send AI crawlers to the gateway with a map and an internal proxy location, no modules beyond the standard build. Running Ghost? A Ghost-CLI install already sits behind Nginx, so this is the page for you. Works for GPTBot, ClaudeBot, PerplexityBot, CCBot, Bytespider and 22 more AI crawlers; Googlebot and Bingbot are not affected.
What you need first: a terminal on the server that runs your site (SSH) and permission to use sudo, to edit the web server's config and add a scheduled job. If your host only gives you a control panel (cPanel, Hostinger, GoDaddy and similar), this guide isn't for you: use the WordPress plugin if your site is WordPress, or put the site behind Cloudflare (free) and follow the Cloudflare guide.
- 1Create your free routeOne field: your site. No wallet, no card.
- 2Add it to NginxCopy one block and paste it. The code is folded below.
- 3Watch your reportSee which AI crawlers visit. Charging stays off until you turn it on.
Install in one step (Debian or Ubuntu)
One command, as root. It asks for your gateway secret (hidden, never on the command line), adds one line to your site's block, checks the configuration and undoes everything if that check fails, then starts the free report. Charging is turned on later in your settings.
curl -fsSL https://aipaygen.com/install/nginx | sudo sh -s -- --route <your-route-id>Prefer to read it first? Download it, check it, read it, run it:
Code4 lines
curl -fsSLo aipaygen-install.sh https://aipaygen.com/install/nginx
sha256sum aipaygen-install.sh # should print d90a6e11acb378d6bcf0eee4187021d3f02801bfee8cb355176ca3ec05dffca5
less aipaygen-install.sh
sudo sh aipaygen-install.sh --route <your-route-id>Compare with the public copy before running: github.com/Damien829/pay-per-crawl carries the same file, with the same SHA-256. It changes nothing when your site has more than one block for it, or the layout isn't the usual one: it prints the one line to add by hand instead. Remove it all later with sudo sh aipaygen-install.sh --uninstall (or curl -fsSL https://aipaygen.com/install/nginx | sudo sh -s -- --uninstall).
Or set it up by hand, below.
Free AI crawler report (start here)
See which AI crawlers read your site and what they would have paid, without charging anyone. Nginx logs only AI crawler visits to a small file, and a cron script sends the counts to your report every 10 minutes. Works in every mode, including report-only.
Code37 lines
# AIPayGen AI crawler report for nginx.
# 1. FIRST install the reporter script and run it once. It creates /etc/nginx/aipaygen-ai-ips.conf and
# /etc/nginx/aipaygen-bots.conf, which the lines below read (nginx -t fails while they're missing):
# sudo curl -fsS "https://aipaygen.com/downloads/aipaygen-nginx-report.sh?route=<your-route-id>" -o /usr/local/bin/aipaygen-report.sh
# sudo chmod 700 /usr/local/bin/aipaygen-report.sh # then put your gateway secret in it
# sudo /usr/local/bin/aipaygen-report.sh
# 2. Add the lines below, then: sudo nginx -t && sudo nginx -s reload
# 3. Run the script every 10 minutes from root's crontab (sudo crontab -e):
# */10 * * * * /usr/local/bin/aipaygen-report.sh
# It sends the visit counts and keeps the address list and the crawler names fresh, reloading nginx only
# when one changed and nginx -t passes. If AIPayGen can't be reached it keeps what it has: your site is unaffected.
# In the http {} block, next to your existing access_log line:
# The AI crawler names (and which of them can be called fake): the script above keeps this file fresh.
include /etc/nginx/aipaygen-bots.conf;
# AI crawlers that hide their name, by address: the lists AI companies publish, named at
# https://aipaygen.com/ai-crawlers#checkable (the script above keeps this file fresh).
geo $aipaygen_ai_ip { default 0; include /etc/nginx/aipaygen-ai-ips.conf; }
# The page as the crawler asked for it, without the query ($uri changes when a rewrite runs).
map $request_uri $aipaygen_path { default ""; "~^([^?]*)" $1; }
log_format aipaygen_ai escape=json '$aipaygen_seen\t$aipaygen_path';
access_log /var/log/nginx/aipaygen-ai.log aipaygen_ai if=$aipaygen_seen;
# If a server {} block has its own access_log lines, it doesn't inherit this one: copy the line above into it too.
# In your site's server {} block: this site's AI Terms (per-use terms for AI) at /.well-known/ai-terms.
location = /.well-known/ai-terms {
# Looked up when a request needs it, not when nginx starts, so nginx always starts. If this server
# can't reach 1.1.1.1 for DNS, use the nameserver from /etc/resolv.conf (127.0.0.53 on Ubuntu).
resolver 1.1.1.1 1.0.0.1 valid=300s ipv6=off;
resolver_timeout 5s;
set $aipaygen_up https://aipaygen.com;
proxy_pass $aipaygen_up/v1/gateway/terms/<your-route-id>;
proxy_set_header Host aipaygen.com;
proxy_ssl_server_name on;
proxy_set_header Cookie "";
proxy_set_header Authorization "";
}Follow the three steps at the top, in order: run the script first, because it creates the address list file the config reads (nginx -t fails until it exists). In step 1, open the script with sudo nano /usr/local/bin/aipaygen-report.sh and replace YOUR_GATEWAY_SECRET in the line SECRET="YOUR_GATEWAY_SECRET" with your gateway secret (the one-line installer above does this for you). The script needs only sh, awk and curl; your gateway secret stays on your server. Download the script. The first counts show in your report within about 10 minutes of the first AI crawler visit.
Option 1: block them
Code4 lines
# http {} block
map $http_user_agent $ai_bot { default 0; "~*(GPTBot|OAI\-SearchBot|ChatGPT\-User|ClaudeBot|Claude\-User|Claude\-SearchBot|anthropic\-ai|PerplexityBot|Perplexity\-User|GoogleOther|CCBot|Bytespider|Amazonbot|meta\-externalagent|FacebookBot|cohere\-ai|Diffbot|YouBot|MistralAI\-User|DuckAssistBot|PetalBot|Timpibot|ImagesiftBot|omgili|AI2Bot|ElevenlabsCrawler|ElevenlabsBot)" 1; }
# server {} block
if ($ai_bot) { return 403; }Blocking also keeps you out of AI answers and citations. Pick crawlers one by one with the robots.txt generator.
Option 2: charge them per page
First create your free route (site, price, wallet); this page then fills in your route id.
Code57 lines
# Pay-per-crawl via AIPayGen (nginx): charging, added ON TOP of the AI crawler report setup.
# Install the report setup first and keep it: it defines $aipaygen_ai_ip (AI crawlers that hide their
# name, by the address lists OpenAI, Perplexity, DuckDuckGo, Common Crawl, Anthropic, Mistral and Amazon publish),
# serves your AI Terms at /.well-known/ai-terms, and its cron script sends your visit counts and keeps
# the address list and your price fresh (it reloads nginx only when a file changed and nginx -t passes).
# 1. Create the price file once (your route must be charging, with a wallet):
# sudo sh -c 'curl -fsS "https://aipaygen.com/v1/gateway/quote/<your-route-id>?format=nginx" -o /etc/nginx/aipaygen-quote.conf.new && mv /etc/nginx/aipaygen-quote.conf.new /etc/nginx/aipaygen-quote.conf'
# 2. Add the lines below, replace YOUR_GATEWAY_SECRET, then: sudo nginx -t && sudo nginx -s reload
# In the http {} block:
map $http_user_agent $aipaygen_bot { default 0; "~*(GPTBot|OAI\-SearchBot|ChatGPT\-User|ClaudeBot|Claude\-User|Claude\-SearchBot|anthropic\-ai|PerplexityBot|Perplexity\-User|GoogleOther|CCBot|Bytespider|Amazonbot|meta\-externalagent|FacebookBot|cohere\-ai|Diffbot|YouBot|MistralAI\-User|DuckAssistBot|PetalBot|Timpibot|ImagesiftBot|omgili|AI2Bot|ElevenlabsCrawler|ElevenlabsBot)" 1; }
# In your site's server {} block (next to the report's /.well-known/ai-terms location):
include /etc/nginx/aipaygen-quote.conf;
set $aipaygen 0;
if ($aipaygen_bot) { set $aipaygen 1; }
if ($aipaygen_ai_ip) { set $aipaygen 1; }
# A crawler retrying with a payment goes to AIPayGen, which verifies it, fetches the page and pays you.
if ($http_payment_signature != "") { set $aipaygen 2; }
if ($http_x_payment != "") { set $aipaygen 2; }
# The gateway fetching a page it was paid for: serve it normally. Replace with your gateway secret.
if ($http_x_aipaygen_gateway_secret = "YOUR_GATEWAY_SECRET") { set $aipaygen 0; }
# Your AI Terms file and robots.txt stay readable by every crawler: they are how it learns your rules and the price.
# Lines to add to robots.txt, written from your AI Terms (IETF aipref Content-Usage; training-only crawlers): https://aipaygen.com/v1/gateway/robots/<your-route-id>
if ($uri = /.well-known/ai-terms) { set $aipaygen 0; }
if ($uri = /robots.txt) { set $aipaygen 0; }
# Charging switched off (report-only): the price file is empty, so every request is served normally.
if ($aipaygen_quote_b64 = "") { set $aipaygen 0; }
# Unpaid AI crawlers get your price from here; they never reach AIPayGen.
if ($aipaygen = 1) { rewrite ^ /__aipaygen_402 last; }
if ($aipaygen = 2) { rewrite ^(.*)$ /__aipaygen$1 last; }
location = /__aipaygen_402 {
internal;
default_type application/json;
add_header payment-required $aipaygen_quote_b64 always;
add_header cache-control no-store always;
# AI Terms: this route's machine-readable terms, which a 402 must link.
add_header link '<https://aipaygen.com/v1/gateway/terms/<your-route-id>>; rel="ai-terms"' always;
return 402 $aipaygen_quote_json;
}
location ^~ /__aipaygen/ {
internal;
# Looked up when a request needs it, not when nginx starts, so nginx always starts. If this server
# can't reach 1.1.1.1 for DNS, use the nameserver from /etc/resolv.conf (127.0.0.53 on Ubuntu).
resolver 1.1.1.1 1.0.0.1 valid=300s ipv6=off;
resolver_timeout 5s;
set $aipaygen_up https://aipaygen.com;
rewrite ^/__aipaygen(.*)$ /g/<your-route-id>$1 break;
proxy_pass $aipaygen_up;
proxy_set_header Host aipaygen.com;
proxy_ssl_server_name on;
# The payment and the usual request headers go on; the visitor's cookies and login do not.
proxy_set_header Cookie "";
proxy_set_header Authorization "";
}Add this on top of the free report setup above, which stays in place.
Details and options
That setup serves your AI Terms, defines the AI crawler address list and sends the visit counts, so this block does not repeat them (nginx would refuse a second /.well-known/ai-terms location). Create the price file with the command at the top, replace YOUR_GATEWAY_SECRET with the gateway secret you got at sign-up, then nginx -t && nginx -s reload. The report script then refreshes your price about hourly and the address list daily, and reloads nginx only when a file changed and nginx -t passes. Nginx answers unpaid AI crawlers itself; only paid requests reach AIPayGen. To charge only training crawlers, remove the AI search and assistant names (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot, YouBot, MistralAI-User, ElevenlabsBot, PetalBot, Timpibot) from the bot list in this config; the other setups follow your route's setting automatically.
Before anything is sold through AIPayGen, your site is checked once: this setup serves your AI Terms at /.well-known/ai-terms, which shows it is yours, so it verifies itself within about an hour of installing (we check for that file). Or add a DNS TXT record _aipaygen.yourdomain.com, a file at https://yourdomain.com/.well-known/aipaygen-verify.txt, or the tag <meta name="aipaygen-verify" content="..."> on your site, each holding the value shown on your report page; then press Check my site now in your report's settings (it asks for your gateway secret), or send POST /v1/gateway/<route>/verify with your API key.
Lost your gateway secret? Get a new one by email.
Using a page cache or CDN?
If a CDN or a cache in front of this server stores your HTML (for example a Cloudflare "Cache Everything" rule), it can answer AI crawlers with the saved copy before this check runs. Make it bypass the cache for all 27 AI crawler user agents this setup checks (listed below, one per line), or don't cache HTML at the edge. Prices are always sent with no-store, so a cache never keeps one.
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Claude-User
Claude-SearchBot
anthropic-ai
PerplexityBot
Perplexity-User
GoogleOther
CCBot
Bytespider
Amazonbot
meta-externalagent
FacebookBot
cohere-ai
Diffbot
YouBot
MistralAI-User
DuckAssistBot
PetalBot
Timpibot
ImagesiftBot
omgili
AI2Bot
ElevenlabsCrawler
ElevenlabsBotCheck it works
Report-only (the free start, no wallet):
curl -I -A GPTBot https://your-site.com/ # 200: report-only serves the page to AI crawlers too
curl -I https://your-site.com/ # 200 for everyone elseBoth answer 200 in report-only mode; then open your report: that visit shows up within about 10 minutes (the cron script runs every 10 minutes). It may be listed as GPTBot (fake): it came from your address, not one OpenAI publishes, which is the check working. Your report's Test my setup button also checks it for you.
Charging (with a wallet):
curl -i -A GPTBot https://your-site.com/ # 402 with your price
curl -I https://your-site.com/ # 200 for everyone elseThe 402 is counted too: open your report: that visit shows up within about 10 minutes (the cron script runs every 10 minutes). It may be listed as GPTBot (fake): it came from your address, not one OpenAI publishes, which is the check working.
Or run the free AI crawler check on your site. Other platforms: WordPress · Cloudflare · Caddy · Apache · Next.js · Vercel (not Next.js) · Netlify · Express / Node.js · Django / Python.
Show it on your site
This badge shows the date your AI Terms file last passed our check, and links to a page that lists your terms in plain words. Until your file passes, it says "not checked": check your file. Replace your-site.com with your site's address.
HTML
Code1 lines
<a href="https://aipaygen.com/ai-terms/site/your-site.com"><img src="https://aipaygen.com/badge/ai-terms.svg?site=your-site.com" alt="AI Terms checked by AIPayGen" height="20"></a>Markdown
[](https://aipaygen.com/ai-terms/site/your-site.com)It says only that the file passed our check on that date. It is not an approval, and it can't tell you whether any AI company follows the terms.
Uninstall
Remove the lines you added from your site's config (the map, geo and include lines and the AIPayGen location blocks), run sudo nginx -t && sudo nginx -s reload, remove the aipaygen-report.sh line from root's crontab (sudo crontab -e), then delete /usr/local/bin/aipaygen-report.sh and /etc/nginx/aipaygen-*. Your site then serves every visitor as before.
To delete your route and its data at AIPayGen too, use delete my sites and data (by the email you confirmed). Your sites and their reports: my sites.