Every page as markdown
Each crawled page comes back as clean markdown with title and word count, stripped of nav, ads, and boilerplate.
POST /v1/crawl
Crawl whole sites, page by page.
Point it at one URL and get clean markdown for every page on the site.
Feed your LLM or agent a whole knowledge base in one job. Start an async crawl from a single URL, follow same-origin links with depth and page limits you set, and poll for clean markdown on every page. Built in Rust, handles JS rendering and bot protection automatically.
What you get
Everything in one call.
Depth and page limits
Set max_depth and max_pages to control exactly how far the crawler walks and how many pages it pulls.
Sitemap seeding
Turn on use_sitemap to seed the queue with sitemap URLs and reach pages that links alone never expose.
Async with polling
Start a job, get a UUID back, and poll until it completes without holding a request open.
How it works
From URL to output in four steps.
01
Post a start URL
Send one URL with optional max_depth, max_pages, and use_sitemap, and get a crawl job ID in return.
02
Same-origin traversal
The crawler walks links breadth-first, staying on the same origin until it hits your depth or page limit.
03
Clean each page
Every page is rendered if needed and converted to markdown, around 90% fewer tokens than the raw HTML.
04
Poll for results
Poll GET /v1/crawl/{id} until status is completed, then read the full pages array of markdown.
API
One request, structured back.
POST /v1/crawl
Send one POST to /v1/crawl with a start URL. The crawler follows same-origin links breadth-first and returns clean markdown for every page it reaches. Set max_depth and max_pages to bound how far it goes.
Ship an agent that actually sees the web.
One credit pool, every endpoint. Cancel anytime, or self-host the open-source core for free.