Site Map (URL discovery)

Discover the URLs of a website: reads robots.txt, follows sitemaps and sitemap indexes, and adds links from the homepage. Returns up to 1,000 same-site URLs with lastmod and source. Use before site-crawl or web-read-batch. Params: url (site or domain, required), max_urls (1 to 1000, default 200).

Call it

import { wrapFetchWithPayment, x402Client } from "@x402/fetch";
import { registerExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";

const client = new x402Client();
registerExactEvmScheme(client, { signer: privateKeyToAccount(process.env.EVM_PRIVATE_KEY) });
const payFetch = wrapFetchWithPayment(fetch, client);

const res = await payFetch("https://agentsvc.io/api/v1/proxy/site-map", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({"url":"https://agentsvc.io","max_urls":50}),
});
const { data, payment } = await res.json();  // payment.transaction = on-chain receipt

Input

Output

Example response (data)

{
  "site": "https://agentsvc.io",
  "count": 3,
  "sitemaps": [
    "https://agentsvc.io/sitemap.xml"
  ],
  "from_sitemap": 3,
  "urls": [
    {
      "url": "https://agentsvc.io/",
      "lastmod": "2026-10-06T14:17:50.554Z",
      "source": "sitemap"
    },
    {
      "url": "https://agentsvc.io/services",
      "lastmod": "2026-10-06T14:17:50.554Z",
      "source": "sitemap"
    }
  ],
  "robots_txt_found": true,
  "checked_at": "2026-10-06T14:17:50.557Z"
}