Marketplace · API · x402

crawl-preflight.postnov01.workers.dev/crawl-check/:domain

Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.

Endpoint	`https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain`
Trust	100% uptime over 1 checks · ~382 ms
Pricing	Pay per call with x402, no account. Call it to get an exact quote before paying.
network	eip155:8453
scheme	exact
pay to	0x02d15E7f25d29743c27021d32a7B280d760B2bC2
method	GET
last updated	2026-11-02T02:18:12.268Z

Use it

Call it (an x402 client pays automatically)

curl -i https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain
# 402 reply -> sign PAYMENT-REQUIRED with an x402 client (e.g. @x402/fetch) -> retry with PAYMENT-SIGNATURE

This listing as JSON

curl https://aipaygen.com/v1/catalog/api/x402-d032363f55b0ec95

Search the whole marketplace from your agent (MCP)

{
  "mcpServers": {
    "aipaygen": {
      "url": "https://aipaygen.com/mcp"
    }
  }
}