Full URL list
Get back a flat array of every page URL the site exposes, ready to feed straight into another endpoint.
POST /v1/map
Every URL on a site, in one call.
Discover every page URL for a domain before you crawl or extract.
Point your agent at a domain and get back a flat list of every page URL, parsed from robots.txt and sitemap.xml. Built in Rust, it resolves sitemap indexes recursively so one request can surface thousands of links, ready to feed into batch or crawl.
What you get
Everything in one call.
Recursive sitemap resolution
Nested sitemap indexes are followed automatically, so a single request can surface thousands of links from one domain.
robots.txt aware
It reads robots.txt for sitemap references first, then falls back to /sitemap.xml when none are declared.
Fast crawl planning
Map a site before you scrape it to scope the work, skip dead routes, and feed the right URLs into batch.
How it works
From URL to output in four steps.
01
Send a domain
POST the base URL of the site you want to map with your API key.
02
Sources are checked
We read robots.txt for declared sitemaps, then look for /sitemap.xml if none are listed.
03
Indexes resolved
Sitemap indexes are expanded recursively to collect every listed page in one pass.
04
URL list returned
You get a deduplicated array of discovered URLs plus a total count.
API
One request, structured back.
POST /v1/map
A sitemap API takes a domain and returns the list of page URLs that site publishes, parsed from its robots.txt and sitemap.xml. It is how an agent learns what pages exist before deciding what to crawl or extract.
Ship an agent that actually sees the web.
One credit pool, every endpoint. Cancel anytime, or self-host the open-source core for free.