Web extraction, LLMs, and building in public.
Technical deep dives on web extraction, content parsing for LLMs, anti-bot bypass, and building open-source infrastructure in Rust. Written by the team behind webclaw.
webclaw turns any website into clean, structured content for AI applications. These posts cover the engineering decisions, trade-offs, and lessons learned building a web extraction toolkit from scratch.
HTML to Markdown for LLMs and RAG
Convert HTML to Markdown for LLMs with boilerplate removed, links preserved, and cleaner RAG input for agents and summarization.
Web scraping for AI agents: 3 hidden problems
Most scraping tools were built for data pipelines, not AI agents. Three things quietly break your pipeline and how to fix them.
Why I built webclaw (Rust scraper for LLMs)
I was tired of scrapers that return 403 or need headless Chrome for basic HTML. So I built one in Rust that actually works.
Stop reading. Start scraping.
Cancel anytime. Turn any page into clean, structured content your agent can actually use.