Markdown out of the box, no Turndown or Readability to maintain
POST /v1/scrape
Website to Markdown for AI and LLM context windows
Convert public web pages into content-oriented Markdown in one call.
Convert public page content to Markdown for documentation ingestion, prompts, and context windows. Remove navigation, ads, and boilerplate before sending content to a model.
Why webclaw
Built for website to markdown.
Remove HTML markup before passing content to a model
Cache controls for repeated page requests
Headings, lists, tables, and links preserved as structure
Protected pages handled automatically, nothing to configure
What you get
Everything this use case needs.
- URL to Markdown in one call
- Boilerplate, nav, and ad stripping
- Heading, list, table, and link fidelity
- Markdown, JSON, llm, text, and rawHtml formats
- Page metadata: title, description, status, timing
Where it fits
Built for the messy parts.
Feeding raw HTML to an LLM wastes the context window on navigation, ads, cookie banners, and scripts. Hand-rolling an HTML-to-Markdown converter means maintaining Readability heuristics, Turndown rules, and a headless browser, and it still breaks on bot-protected pages.
webclaw's /v1/scrape endpoint returns clean Markdown directly. It strips boilerplate and preserves headings, lists, tables, and links. JS rendering and bot-protection handling kick in automatically when a page needs them.
How it works
Build it step by step.
The real flow, one step at a time. Switch between TypeScript, Python, and cURL on any snippet.
- 1
Send a URL
POST the page URL to /v1/scrape with formats set to markdown.
// POST the page URL to /v1/scrape, asking for Markdownconst result = await webclaw.scrape({ url: "https://example.com/docs/getting-started", formats: ["markdown"],}); - 2
Page is fetched
The engine fetches the page and escalates to JS rendering or bot-protection handling only when the page needs it.
- 3
HTML is cleaned
Navigation, ads, and scripts are stripped while headings, lists, tables, and links are kept as Markdown.
- 4
Markdown returned
Clean Markdown plus title, status, and timing metadata comes back, ready for a prompt, vector store, or docs index.
// Clean Markdown plus page metadata comes backconsole.log(result.markdown); // boilerplate strippedconsole.log(result.metadata); // title, description, status, timing// Drop straight into a prompt or context windowconst prompt = `Summarize this page:\n\n${result.markdown}`;
Turndown converts whatever HTML you hand it, including nav, ads, and scripts, so you still need to fetch the page and clean it first. webclaw fetches the page, strips boilerplate, and returns article-grade Markdown in one call, and it handles JS rendering and bot-protected pages that a plain HTTP fetch cannot reach.
For AI agents
Or hand it to your agent.
Add the webclaw MCP server to Claude, Cursor, or any MCP client, then paste this prompt. The agent calls the webclaw tools and hands the result back to your model — no code to write.
PROMPT FOR YOUR AGENT
Using the webclaw tools, call scrape on [the page URL] and convert it into clean, LLM-ready Markdown. Strip out the navigation, ads, cookie banners, and scripts, but keep the real structure intact: headings, lists, tables, and links should survive as proper Markdown. Return the cleaned Markdown along with the page's title and any description metadata. If I give you several URLs, use batch to scrape them all in parallel and return one Markdown block per page, each labeled with its source URL.
Ready to build? Start extracting.
Cancel anytime. Clean, structured data on every call.