Content-oriented output reduces the page markup sent for embedding
POST /v1/batch
Web scraping for RAG pipelines.
Feed your vector database with fresh, clean web content.
Build retrieval augmented generation systems with real, up-to-date web data. webclaw returns LLM-optimized markdown that embeds cleanly into any vector database, with page markup and boilerplate removed.
Why webclaw
Built for rag pipelines.
Configurable caching supports repeated indexing workloads
Content diffing for incremental updates instead of full re-crawls
CSS selector filtering to extract only article content, not nav/footer
Built-in batch endpoint for parallel multi-URL ingestion
What you get
Everything this use case needs.
- Markdown and LLM-optimized output formats
- Batch endpoint for parallel ingestion
- Content diff endpoint for incremental updates
- CSS selector filtering
- Firecrawl v2 compatible (drop-in replacement)
Where it fits
Built for the messy parts.
Your RAG pipeline is only as good as the documents you index. Most web scrapers return messy HTML full of navigation, ads, and boilerplate that wastes embedding costs and poisons retrieval quality. Re-crawling frequently is slow and expensive with browser-based tools.
webclaw returns clean markdown with boilerplate stripped, semantic structure preserved, and token count minimized. Choose an indexing schedule based on your dataset, source latency, and freshness requirements. Built-in content diffing lets you incrementally update only what changed.
How it works
Build it step by step.
The real flow, one step at a time. Switch between TypeScript, Python, and cURL on any snippet.
- 1
Collect source URLs
Gather the docs and article pages you want indexed, or discover them with /v1/map.
// Discover every page under a docs siteconst sitemap = await webclaw.map({ url: "https://example.com/docs" });const urls = sitemap.links;console.log(`Found ${urls.length} pages to index`); - 2
Scrape to markdown
Call /v1/scrape (or /v1/batch in parallel) to get clean, boilerplate-stripped markdown for each page.
// Parallel scrape, LLM-optimized markdown outputconst results = await webclaw.batch({ urls, formats: ["markdown"] });for (const r of results.data) { console.log(r.url, r.markdown.length);} - 3
Chunk and embed
Split the markdown, embed each chunk, and upsert into your vector database keyed by URL.
// Split each page, embed every chunk, upsert keyed by URLfor (const r of results.data) { const chunks = chunk(r.markdown); for (const [i, text] of chunks.entries()) { const embedding = await embed(text); await vectordb.upsert({ id: `${r.url}#${i}`, text, embedding }); }} - 4
Diff to re-index
Use /v1/diff against the last snapshot so you only re-embed pages whose content actually changed.
// Only re-embed pages whose content actually changedconst result = await webclaw.diff({ url: r.url, previous: lastSnapshot, current: r.markdown,});if (result.changed) await reindex(r.url, r.markdown);
webclaw strips navigation, ads, scripts, and boilerplate before returning content. Removing markup can reduce embedding input size; measure the token reduction on your own page corpus.
For AI agents
Or hand it to your agent.
Add the webclaw MCP server to Claude, Cursor, or any MCP client, then paste this prompt. The agent calls the webclaw tools and hands the result back to your model — no code to write.
PROMPT FOR YOUR AGENT
Using the webclaw tools, build a clean, up-to-date corpus I can feed into my RAG pipeline's vector database from [the docs or site section to index]. First call map on [the base URL or domain] to discover all the relevant page URLs, then use batch to scrape them all in parallel into LLM-optimized markdown with the navigation, ads, and boilerplate stripped out. For each page return its source URL, a short title, the cleaned markdown body, and an estimated token count, organized so each page is a self-contained chunk ready to embed. Skip or flag any pages that came back empty or look like login walls. Finally, list which URLs changed since [the date or snapshot of my last index] using diff so I only re-embed what's actually new, and summarize the whole run in a table of URL, status (new / changed / unchanged / failed), and token count.
Ready to build? Start extracting.
Cancel anytime. Clean, structured data on every call.