Many URLs, one request
Pass an array of URLs and webclaw scrapes them all, instead of you firing one request per page.
POST /v1/batch
Scrape many URLs in one call.
Send a list of URLs, get back clean content for every page in a single request.
Feed your agent or pipeline a whole list of URLs and get back LLM-ready markdown, JSON, or structured text for each one in a single call. Pages run in parallel on the server, so you skip the loop, the rate-limit juggling, and the dozens of round trips.
What you get
Everything in one call.
Parallel by default
Pages are fetched concurrently on the server, with a concurrency you control, so a large list finishes far faster than a sequential loop.
Per-URL failure isolation
A dead link or a 404 on one page never sinks the batch, every result carries its own error field so you know exactly what failed.
LLM-ready output
Each page comes back as clean markdown, JSON, or structured text with around 90% fewer tokens than raw HTML, ready to feed straight to a model.
How it works
From URL to output in four steps.
01
Send your URL list
POST an array of URLs to /v1/batch with the formats you want and an optional concurrency limit.
02
Parallel fetch
webclaw scrapes the pages concurrently in Rust, handling JavaScript rendering and bot protection automatically per page.
03
Extract per page
Each response is stripped of nav, ads, and boilerplate, then converted into your chosen format with its own metadata.
04
Results returned
You get one results array with content for the pages that succeeded and an error field on the ones that didn't.
API
One request, structured back.
POST /v1/batch
POST a urls array to /v1/batch. webclaw scrapes every page in parallel on the server and returns one results array, so you do not have to loop over a scrape endpoint yourself.
Ship an agent that actually sees the web.
One credit pool, every endpoint. Cancel anytime, or self-host the open-source core for free.