Site Crawl to Markdown

Crawl a website from a start URL and return up to 20 same-site HTML pages as clean Markdown with title and description (breadth-first, 3 in parallel, skips files and non-HTML). Optional path_prefix keeps the crawl inside a section such as /docs. Charged only if at least one page was read. Params: url (required), max_pages (1 to 20, default 5), max_chars per page (500 to 20000, default 4000), path_prefix.

Call it

import { wrapFetchWithPayment, x402Client } from "@x402/fetch";
import { registerExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";

const client = new x402Client();
registerExactEvmScheme(client, { signer: privateKeyToAccount(process.env.EVM_PRIVATE_KEY) });
const payFetch = wrapFetchWithPayment(fetch, client);

const res = await payFetch("https://agentsvc.io/api/v1/proxy/site-crawl", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({"url":"https://agentsvc.io/docs","max_pages":3,"max_chars":2000}),
});
const { data, payment } = await res.json();  // payment.transaction = on-chain receipt

Input

Output

Example response (data)

{
  "site": "https://agentsvc.io",
  "pages_crawled": 2,
  "ok_count": 2,
  "skipped_non_html": 2,
  "discovered_urls": 37,
  "pages": [
    {
      "url": "https://agentsvc.io/docs",
      "ok": true,
      "status": 200,
      "title": "Docs: how agents pay per call with x402 · agentsvc.io",
      "description": "Quickstart, payment protocol (x402 v1 and v2), free trial, error codes and the reference for every service on agentsvc.io.",
      "markdown": "Documentation\n\n# Pay per call. Nothing else to set up.\n\nEvery service is a JSON endpoint behind HTTP 402. If you already use an x402 client, you are done: po…",
      "word_count": 104
    },
    {
      "url": "https://agentsvc.io/services/erc20-info",
      "ok": true,
      "status": 200,
      "title": "ERC20 Token Info: $0.002 USDC per call · agentsvc.io",
      "description": "Read an ERC20 token contract on Base or Ethereum mainnet: name, symbol, decimals, total supply, and optionally the balance of one holder address. Verifies that ",
      "markdown": "[services](/services) / erc20-info\n\ndataoperational · 240 ms\n\n# ERC20 Token Info\n\nRead an ERC20 token contract on Base or Ethereum mainnet: name, symbol, dec…",
      "word_count": 83
    }
  ],
  "took_ms": 125,
  "crawled_at": "2026-10-06T14:17:50.682Z"
}