An AI training crawler was 58% of our traffic. We charged it $0.001 a page.
November 15, 2026. What happened when we put our own pay-per-crawl in front of the crawler, with the numbers.
What we saw
Our free database plan allows 5 million row reads a day, and we were at 4.8 million. Watching live traffic for 75 seconds showed why: 52 of 89 requests (58%) came from one user agent, ElevenlabsCrawler/1.0.
ElevenLabs documents it as the crawler that collects content that may be used to train its generative AI models. It was walking all 20,000 listing pages in our marketplace, one at a time. Pages are cached, but a crawler rarely asks for the same page twice, so almost every visit missed the cache and ran a search against the database.
What we did
At 16:08 UTC we turned on pay-per-crawl for those listing pages, the same thing we offer other sites:
- AI crawlers (by user agent, and by the IP ranges their operators publish) get
402 Payment Requiredwith a price of $0.001 in USDC, payable over x402. Pay it and you get the page. - The 402 says in plain words how to pay, including without a wallet.
- People and search engines see exactly what they saw before.
- Our product pages, docs and home page stay free to every crawler, so we keep showing up in AI answers.
What happened
| Before | After | |
|---|---|---|
| ElevenlabsCrawler share of requests | 52 of 89 (58%) | 0 of 49, about 20 minutes later |
| Database rows read, last 24 hours | 4.81 million | 3.96 million, still falling |
| Paid by the crawler | $0. It didn't pay; it stopped. | |
Both traffic numbers are short samples, so read them as a snapshot, not a study. The database figure is a rolling 24 hours, so it keeps dropping for a day after a change.
What it means if you run a site
robots.txt can only say no, and only to crawlers that choose to listen. A 402 says "yes, for a price". Today most AI crawlers can't pay, so in practice the price works like a block that stops free bulk copying, without touching people, search engines or the AI assistants you want citing you. As agents get wallets, the same response starts earning.
Two other things we learned:
- Your biggest visitor may not be a person. We only found out by reading raw traffic. Most analytics tools hide bots.
- A crawler's user agent is a calling card. ElevenLabs links its bot page from its user agent, which is how we learned who it was. Ours does the same: AIPayGenBot.
Try it on your site
Set up pay-per-crawl Check which AI crawlers you allow Verify a bot from your logs
Free for site owners; payments go straight to your wallet. WordPress plugin, and snippets for Cloudflare, Nginx, Apache, Next.js and Express, are open source.