Charge AI crawlers per page. People still browse free.

AI crawlers and agents now fetch a large share of web pages, and the sites pay for the bandwidth without getting anything back. Pay-per-crawl answers them with HTTP 402: they pay a fraction of a cent per page in USDC through x402, straight to your wallet, and get the page. Visitors see nothing different.

Set it up (5 minutes)How the gateway works

How it works

  1. An AI crawler asks for a page. Your site passes it to the AIPayGen gateway on the same URL.
  2. The gateway answers 402 Payment Required with your price and your wallet address.
  3. The crawler signs a USDC payment and asks again. The gateway checks the payment, fetches the page from your site with a secret header, and settles the payment on Base.
  4. The crawler gets the page; the money lands in your wallet. If your site doesn't answer with a 2xx, nobody is charged.

Your price is listed in the AIPayGen marketplace and in x402 discovery, so agents can find your content and budget for it. Crawlers that don't pay just don't get the page, the same as a block in robots.txt, but now there's a way for them to say yes.

Set it up here (no command line)

Free. Creates an AIPayGen key and a gateway route for your site. Nothing is charged to you.

Setup from the command line

1. Get a free API key with curl -X POST https://aipaygen.com/v1/keys, then register your site as a gateway route. Prices run from $0.001 to $10 a page.

curl -X POST https://aipaygen.com/v1/gateway \
  -H "Authorization: Bearer $AIPAYGEN_KEY" -H "content-type: application/json" \
  -d '{"name": "example.com pages","upstream": "https://example.com","price_usd": 0.002,"pay_to":"0xYourWallet"}'

# -> {"id":"3f9a1c0b7d2e","url":"https://aipaygen.com/g/3f9a1c0b7d2e","gateway_secret":"...",...}
# Buyers call https://aipaygen.com/g/3f9a1c0b7d2e/forecast?city=Austin and pay per call.

Keep the gateway_secret from the answer; it's shown once.

2. Put this in front of your site. On Cloudflare, as a Worker on your site's route:

// Pay-per-crawl via AIPayGen. Set the secret: npx wrangler secret put AIPAYGEN_SECRET
const GATEWAY = "https://aipaygen.com/g/<your-route-id>";
const AI_BOTS = /GPTBot|OAI\-SearchBot|ChatGPT\-User|ClaudeBot|Claude\-User|Claude\-SearchBot|anthropic\-ai|PerplexityBot|Perplexity\-User|Google\-Extended|GoogleOther|CCBot|Bytespider|Amazonbot|Applebot\-Extended|meta\-externalagent|FacebookBot|cohere\-ai|Diffbot|YouBot|MistralAI\-User|DuckAssistBot|PetalBot|Timpibot|ImagesiftBot|omgili|AI2Bot/i;
export default {
  async fetch(req, env) {
    // The gateway fetching a page it was paid for: serve it.
    if (env.AIPAYGEN_SECRET && req.headers.get("x-aipaygen-gateway-secret") === env.AIPAYGEN_SECRET) return fetch(req);
    const paying = req.headers.get("payment-signature") || req.headers.get("x-payment");
    if (!AI_BOTS.test(req.headers.get("user-agent") || "") && !paying) return fetch(req); // people browse free
    const u = new URL(req.url);
    return fetch(GATEWAY + u.pathname + u.search, { method: req.method, headers: req.headers, body: req.body });
  },
};

Or in an Express app:

// Pay-per-crawl via AIPayGen. Put AIPAYGEN_SECRET (the gateway_secret) in your environment.
const GATEWAY = "https://aipaygen.com/g/<your-route-id>";
const AI_BOTS = /GPTBot|OAI\-SearchBot|ChatGPT\-User|ClaudeBot|Claude\-User|Claude\-SearchBot|anthropic\-ai|PerplexityBot|Perplexity\-User|Google\-Extended|GoogleOther|CCBot|Bytespider|Amazonbot|Applebot\-Extended|meta\-externalagent|FacebookBot|cohere\-ai|Diffbot|YouBot|MistralAI\-User|DuckAssistBot|PetalBot|Timpibot|ImagesiftBot|omgili|AI2Bot/i;
app.use(async (req, res, next) => {
  if (process.env.AIPAYGEN_SECRET && req.get("x-aipaygen-gateway-secret") === process.env.AIPAYGEN_SECRET) return next();
  const paying = req.get("payment-signature") || req.get("x-payment");
  if (!AI_BOTS.test(req.get("user-agent") || "") && !paying) return next();
  const headers = { ...req.headers }; delete headers.host;
  const r = await fetch(GATEWAY + req.originalUrl, { method: "GET", headers });
  r.headers.forEach((v, k) => { if (!["content-encoding", "content-length", "transfer-encoding"].includes(k)) res.set(k, v); });
  res.status(r.status).send(Buffer.from(await r.arrayBuffer()));
});

3. Check it. curl -A GPTBot https://example.com/ should answer 402 with your price; a normal browser still gets the page. See earnings with GET https://aipaygen.com/v1/gateway.

Which crawlers are asked to pay

GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, CCBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent, FacebookBot, cohere-ai, Diffbot, YouBot, MistralAI-User, DuckAssistBot, PetalBot, Timpibot, ImagesiftBot, omgili, AI2Bot. Search engine crawlers (Googlebot, Bingbot) are not on the list, so your search ranking doesn't change. Edit the list in the snippet to suit you.

Fees

None from us. The payment goes from the crawler's wallet to yours; the x402 facilitator pays the network fee.