Pay-per-crawl / setup / Apache
Block or charge AI crawlers on Apache
Apache answers AI crawlers on your own server; a small cron script sends the visit counts and keeps your price fresh. Works for GPTBot, ClaudeBot, PerplexityBot, CCBot, Bytespider and 22 more AI crawlers; Googlebot and Bingbot are not affected.
What you need first: a terminal on the server that runs your site (SSH) and permission to use sudo, to edit the web server's config and add a scheduled job. If your host only gives you a control panel (cPanel, Hostinger, GoDaddy and similar), this guide isn't for you: use the WordPress plugin if your site is WordPress, or put the site behind Cloudflare (free) and follow the Cloudflare guide.
- 1Create your free routeOne field: your site. No wallet, no card.
- 2Add it to ApacheCopy one block and paste it. The code is folded below.
- 3Watch your reportSee which AI crawlers visit. Charging stays off until you turn it on.
Install in one step (Debian or Ubuntu)
One command, as root. It asks for your gateway secret (hidden, never on the command line), adds one line to your site's block, checks the configuration and undoes everything if that check fails, then starts the free report. Charging is turned on later in your settings.
curl -fsSL https://aipaygen.com/install/apache | sudo sh -s -- --route <your-route-id>Prefer to read it first? Download it, check it, read it, run it:
Code4 lines
curl -fsSLo aipaygen-install.sh https://aipaygen.com/install/apache
sha256sum aipaygen-install.sh # should print 4c1473dddd171fb410d1a49ec55a4f16d7fc1dc07c57420e44900d9e569d6358
less aipaygen-install.sh
sudo sh aipaygen-install.sh --route <your-route-id>Compare with the public copy before running: github.com/Damien829/pay-per-crawl carries the same file, with the same SHA-256. It changes nothing when your site has more than one block for it, or the layout isn't the usual one: it prints the one line to add by hand instead. Remove it all later with sudo sh aipaygen-install.sh --uninstall (or curl -fsSL https://aipaygen.com/install/apache | sudo sh -s -- --uninstall).
Or set it up by hand, below.
Option 1: block them
Code3 lines
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|OAI\-SearchBot|ChatGPT\-User|ClaudeBot|Claude\-User|Claude\-SearchBot|anthropic\-ai|PerplexityBot|Perplexity\-User|GoogleOther|CCBot|Bytespider|Amazonbot|meta\-externalagent|FacebookBot|cohere\-ai|Diffbot|YouBot|MistralAI\-User|DuckAssistBot|PetalBot|Timpibot|ImagesiftBot|omgili|AI2Bot|ElevenlabsCrawler|ElevenlabsBot) [NC]
RewriteRule ^ - [F]Blocking also keeps you out of AI answers and citations. Pick crawlers one by one with the robots.txt generator.
Option 2: charge them per page
First create your free route (site, price, wallet); this page then fills in your route id.
Code88 lines
# Pay-per-crawl via AIPayGen (Apache 2.4). Turn on the modules these lines use first (Debian/Ubuntu):
# sudo a2enmod rewrite headers setenvif proxy proxy_http ssl
# The last three let a paying crawler's retry reach AIPayGen, which settles the payment and pays you. Without them
# a paid retry gets your price again (402), never your page for free, so you can't earn until they're on.
#
# 1. FIRST install the helper script and run it once. It creates /var/lib/aipaygen, which the lines
# below write to: Apache will not start if that folder is missing.
# sudo curl -fsS "https://aipaygen.com/downloads/aipaygen-apache.sh?route=<your-route-id>" -o /usr/local/bin/aipaygen-apache.sh
# sudo chmod 700 /usr/local/bin/aipaygen-apache.sh # then put your gateway secret in it
# sudo /usr/local/bin/aipaygen-apache.sh
# 2. Paste the lines below inside your site's <VirtualHost> block (not .htaccess), above your own
# Rewrite and Proxy lines, and replace YOUR_GATEWAY_SECRET.
# 3. sudo apachectl configtest && sudo apachectl graceful
# 4. Run the script every 10 minutes from root's crontab (sudo crontab -e):
# */10 * * * * /usr/local/bin/aipaygen-apache.sh
#
# If this virtual host has no CustomLog line of its own, add your usual one too (for example
# CustomLog ${APACHE_LOG_DIR}/access.log combined): a virtual host with any CustomLog stops
# writing to the server-wide access log.
# Which requests are AI crawlers: by address first (the lists AI companies publish, named at
# https://aipaygen.com/ai-crawlers#checkable; the script keeps this file fresh), then by name.
IncludeOptional /var/lib/aipaygen/ai-ips.conf
# The AI crawler names, and "<name> (fake)" for a name whose operator publishes its addresses used from outside them
# (only once ai-ips.conf has loaded): the script keeps this file fresh.
IncludeOptional /var/lib/aipaygen/bots.conf
# Their visits, one line each (crawler name and path only), for your report.
CustomLog /var/lib/aipaygen/visits.log "%<{aipaygen_bot}e\t%U" env=aipaygen_bot
<Directory /var/lib/aipaygen>
Require all granted
ForceType application/json
Header set Access-Control-Allow-Origin "*"
</Directory>
# An unpaid AI crawler gets your price from a file on this server; it never reaches AIPayGen.
ErrorDocument 402 /__aipaygen_402
Header always set payment-required "expr=%{file:/var/lib/aipaygen/quote.b64}" "expr=%{REQUEST_STATUS} -eq 402"
Header always set cache-control "no-store" "expr=%{REQUEST_STATUS} -eq 402"
# AI Terms: this route's machine-readable terms, which a 402 must link.
Header always set link "<https://aipaygen.com/v1/gateway/terms/<your-route-id>>; rel=\"ai-terms\"" "expr=%{REQUEST_STATUS} -eq 402"
RewriteEngine On
# This site's AI Terms (per-use terms for AI), served at /.well-known/ai-terms.
RewriteCond /var/lib/aipaygen/ai-terms.json -f
RewriteRule ^/\.well-known/ai-terms$ /var/lib/aipaygen/ai-terms.json [L]
RewriteCond %{ENV:REDIRECT_STATUS} =402
RewriteRule ^/__aipaygen_402$ /var/lib/aipaygen/quote.json [L]
# Charge: only while your route charges (the script creates /var/lib/aipaygen/charging then and
# removes it in report-only mode, so you switch modes in your AIPayGen settings, not here).
RewriteCond %{ENV:aipaygen_bot} .
# The gateway fetching a page it was paid for: serve it normally. Replace with your gateway secret.
RewriteCond %{HTTP:X-AIPayGen-Gateway-Secret} !=YOUR_GATEWAY_SECRET
# robots.txt and the terms file are free for every crawler: they are how it learns your rules and the price.
# Lines to add to robots.txt, written from your AI Terms (IETF aipref Content-Usage; training-only crawlers): https://aipaygen.com/v1/gateway/robots/<your-route-id>
RewriteCond %{REQUEST_URI} !^/(robots\.txt|\.well-known/ai-terms|__aipaygen_402)$
RewriteCond %{HTTP:Payment-Signature}%{HTTP:X-Payment} ^$
RewriteCond /var/lib/aipaygen/charging -f
RewriteRule ^ - [R=402,L]
# A crawler retrying with a payment goes to AIPayGen, which verifies it, fetches the page and pays you.
<IfModule mod_ssl.c>
SSLProxyEngine on
</IfModule>
<IfModule mod_proxy_http.c>
<IfModule mod_ssl.c>
RewriteCond %{HTTP:X-AIPayGen-Gateway-Secret} !=YOUR_GATEWAY_SECRET
RewriteCond %{REQUEST_URI} !^/(robots\.txt|\.well-known/ai-terms)$
RewriteCond %{HTTP:Payment-Signature}%{HTTP:X-Payment} .
RewriteCond /var/lib/aipaygen/charging -f
RewriteRule ^/?(.*)$ https://aipaygen.com/g/<your-route-id>/$1 [P,L,QSA]
</IfModule>
</IfModule>
# Without those modules a paid retry can't reach AIPayGen: it gets the price again, never the page for free.
<IfModule !mod_proxy_http.c>
RewriteCond %{HTTP:X-AIPayGen-Gateway-Secret} !=YOUR_GATEWAY_SECRET
RewriteCond %{REQUEST_URI} !^/(robots\.txt|\.well-known/ai-terms|__aipaygen_402)$
RewriteCond %{HTTP:Payment-Signature}%{HTTP:X-Payment} .
RewriteCond /var/lib/aipaygen/charging -f
RewriteRule ^ - [R=402,L]
</IfModule>
<IfModule !mod_ssl.c>
RewriteCond %{HTTP:X-AIPayGen-Gateway-Secret} !=YOUR_GATEWAY_SECRET
RewriteCond %{REQUEST_URI} !^/(robots\.txt|\.well-known/ai-terms|__aipaygen_402)$
RewriteCond %{HTTP:Payment-Signature}%{HTTP:X-Payment} .
RewriteCond /var/lib/aipaygen/charging -f
RewriteRule ^ - [R=402,L]
</IfModule>Follow the four numbered steps at the top of the config, in order, and in step 1 open the script with sudo nano /usr/local/bin/aipaygen-apache.sh and put your gateway secret in place of YOUR_GATEWAY_SECRET in its line SECRET="YOUR_GATEWAY_SECRET" (the one-line installer above does this for you).
Details and options
Run the script first, because Apache won't start if its folder is missing (apachectl configtest catches that before you reload). This goes in the virtual host, not .htaccess; on shared hosting without access to it, use the WordPress plugin or another setup. One config covers both modes: in report-only mode it only counts visits, and when you switch your route to charging, crawlers start getting your price within the hour, with no change on the server. Apache answers crawlers itself, from files the script refreshes, so your site doesn't depend on AIPayGen being reachable; only a crawler's paid retry goes through AIPayGen. With SELinux enforcing (RHEL, Alma, Rocky), also run: sudo chcon -R -t httpd_sys_rw_content_t /var/lib/aipaygen (we have not tested on SELinux). To charge only training crawlers, remove the AI search and assistant names (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot, YouBot, MistralAI-User, ElevenlabsBot, PetalBot, Timpibot) from the bot list in this config.
Before anything is sold through AIPayGen, your site is checked once: this setup serves your AI Terms at /.well-known/ai-terms, which shows it is yours, so it verifies itself within about an hour of installing (we check for that file). Or add a DNS TXT record _aipaygen.yourdomain.com, a file at https://yourdomain.com/.well-known/aipaygen-verify.txt, or the tag <meta name="aipaygen-verify" content="..."> on your site, each holding the value shown on your report page; then press Check my site now in your report's settings (it asks for your gateway secret), or send POST /v1/gateway/<route>/verify with your API key.
Lost your gateway secret? Get a new one by email.
Using a page cache or CDN?
If a CDN or a cache in front of this server stores your HTML (for example a Cloudflare "Cache Everything" rule), it can answer AI crawlers with the saved copy before this check runs. Make it bypass the cache for all 27 AI crawler user agents this setup checks (listed below, one per line), or don't cache HTML at the edge. Prices are always sent with no-store, so a cache never keeps one.
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Claude-User
Claude-SearchBot
anthropic-ai
PerplexityBot
Perplexity-User
GoogleOther
CCBot
Bytespider
Amazonbot
meta-externalagent
FacebookBot
cohere-ai
Diffbot
YouBot
MistralAI-User
DuckAssistBot
PetalBot
Timpibot
ImagesiftBot
omgili
AI2Bot
ElevenlabsCrawler
ElevenlabsBotCheck it works
Report-only (the free start, no wallet):
curl -I -A GPTBot https://your-site.com/ # 200: report-only serves the page to AI crawlers too
curl -I https://your-site.com/ # 200 for everyone elseBoth answer 200 in report-only mode; then open your report: that visit shows up within about 10 minutes (the cron script runs every 10 minutes). It may be listed as GPTBot (fake): it came from your address, not one OpenAI publishes, which is the check working. Your report's Test my setup button also checks it for you.
Charging (with a wallet):
curl -i -A GPTBot https://your-site.com/ # 402 with your price
curl -I https://your-site.com/ # 200 for everyone elseThe 402 is counted too: open your report: that visit shows up within about 10 minutes (the cron script runs every 10 minutes). It may be listed as GPTBot (fake): it came from your address, not one OpenAI publishes, which is the check working.
Or run the free AI crawler check on your site. Other platforms: WordPress · Cloudflare · Nginx · Caddy · Next.js · Vercel (not Next.js) · Netlify · Express / Node.js · Django / Python.
Show it on your site
This badge shows the date your AI Terms file last passed our check, and links to a page that lists your terms in plain words. Until your file passes, it says "not checked": check your file. Replace your-site.com with your site's address.
HTML
Code1 lines
<a href="https://aipaygen.com/ai-terms/site/your-site.com"><img src="https://aipaygen.com/badge/ai-terms.svg?site=your-site.com" alt="AI Terms checked by AIPayGen" height="20"></a>Markdown
[](https://aipaygen.com/ai-terms/site/your-site.com)It says only that the file passed our check on that date. It is not an approval, and it can't tell you whether any AI company follows the terms.
Uninstall
Remove the lines you pasted into your <VirtualHost>, run sudo apachectl configtest and reload Apache, remove the aipaygen-apache.sh line from root's crontab (sudo crontab -e), then delete /usr/local/bin/aipaygen-apache.sh and the AIPayGen files it made. Your site then serves every visitor as before.
To delete your route and its data at AIPayGen too, use delete my sites and data (by the email you confirmed). Your sites and their reports: my sites.