Web extraction, LLMs, and building in public.
Technical deep dives on web extraction, content parsing for LLMs, anti-bot bypass, and building open-source infrastructure in Rust. Written by the team behind webclaw.
webclaw turns any website into clean, structured content for AI applications. These posts cover the engineering decisions, trade-offs, and lessons learned building a web extraction toolkit from scratch.
How to Convert HTML to Markdown: The Complete 2026 Guide
Convert HTML to Markdown the right way: Pandoc for local files, Turndown and markdownify in code, and a URL-to-Markdown API for JavaScript-rendered pages.
Apify Alternative for LLM Web Scraping and AI Agents
Compare Apify actors, the Apify marketplace, and Webclaw for any-URL markdown extraction, structured JSON, crawling, MCP access, and AI agent web tooling.
Bright Data Alternative for LLM Web Scraping
Compare Bright Data, Web Unlocker, and Webclaw for proxy infrastructure, markdown extraction, structured JSON, crawling, batching, and AI agent workflows.
Jina Reader Alternative That Handles Cloudflare (2026)
Jina Reader breaks on Cloudflare and DataDome. Same r.jina.ai-style URL to markdown, plus crawling, batching, and anti-bot bypass that returns content.
Crawl4AI vs Playwright: Which to Use for Scraping (2026)
Crawl4AI vs Playwright for web scraping: which one to pick, where each breaks, and when you need neither. Markdown output, browser control, RAG input.
JavaScript Rendering API: When You Actually Need a Browser
Most pages do not need a headless browser. How to detect an empty React shell, when a JavaScript rendering API is worth it, and how to skip the slow path.
Anti-Bot Scraping API 2026: signals that force browser fallback
The exact block markers, JA4 fingerprints, empty shells, anti-bot cookies, JavaScript heuristics, and content-quality signals that decide when a scraping API should escalate to a browser.
Anti-Bot Scraping API in 2026: Skip Browser-First, Stay Fast
An anti-bot scraping API that detects the block first, then escalates to a browser only when needed. Faster and cheaper, with clean markdown or JSON out.
How to evaluate web scraping APIs for AI agents
A practical checklist for testing web scraping APIs on real agent and RAG workflows, not toy URLs like example.com.
Stop reading. Start scraping.
Cancel anytime. Turn any page into clean, structured content your agent can actually use.