Many URLs, one request
Pass an array of URLs and webclaw scrapes them all, instead of you firing one request per page.
POST /v1/batch
Scrape many URLs in one call.
Send a list of URLs, get back clean content for every page in a single request.
Feed your agent or pipeline a whole list of URLs and get back LLM-ready markdown, JSON, or structured text for each one in a single call. Pages run in parallel on the server, so you skip the loop, the rate-limit juggling, and the dozens of round trips.
What you get
Everything in one call.
Parallel by default
Pages are fetched concurrently on the server, with a concurrency you control, so a large list finishes far faster than a sequential loop.
Per-URL failure isolation
A dead link or a 404 on one page never sinks the batch, every result carries its own error field so you know exactly what failed.
LLM-ready output
Each page comes back as clean markdown, JSON, or structured text with around 90% fewer tokens than raw HTML, ready to feed straight to a model.
How it works
From URL to output in four steps.
01
Send your URL list
POST an array of URLs to /v1/batch with the formats you want and an optional concurrency limit.
02
Parallel fetch
webclaw scrapes the pages concurrently in Rust, handling JavaScript rendering and bot protection automatically per page.
03
Extract per page
Each response is stripped of nav, ads, and boilerplate, then converted into your chosen format with its own metadata.
04
Results returned
You get one results array with content for the pages that succeeded and an error field on the ones that didn't.
API
One request, structured back.
POST /v1/batch
Common questions
Frequently asked questions
How do I scrape multiple URLs in one API request?
POST a urls array to /v1/batch. webclaw scrapes every page in parallel on the server and returns one results array, so you do not have to loop over a scrape endpoint yourself.
How many URLs can I send in a single batch?
Send a list of URLs and tune the concurrency field to control how many run at once. The default concurrency is 5, which keeps things fast without hammering target servers. Keep it in the 5 to 10 range for large lists.
What happens if one URL in the batch fails?
Nothing else breaks. Individual failures are isolated, so a 404 or a dead link only affects that one entry. Check the error field on each result to see which pages failed and why.
Am I billed for failed requests?
No. Credits are only consumed on successful responses. A standard page is 1 credit; heavier work like JS rendering or protected-site access costs a few extra credits.
Is batch scraping faster than calling the scrape endpoint in a loop?
Yes. Pages are fetched concurrently in Rust inside a single request, so you skip the per-call overhead and the round trips. A long list finishes much faster than scraping URLs one at a time.
Ship an agent that actually sees the web.
One credit pool, every endpoint. Cancel anytime, or self-host the open-source core for free.