Web scraping for AI agents. What actually works.

If you're building anything with LLMs right now, you've probably hit this wall: your agent needs to read a webpage, and suddenly you're deep into scraping infrastructure instead of working on your actual product.

I've spent the last months building webclaw specifically for this problem. Not scraping in general. Scraping for AI agents. It's a different problem than most people think.

Why traditional scraping tools don't work for AI

Traditional web scraping was built for a different world. You'd write a scraper, schedule it to run every night, dump results into a database. The output was structured data: prices, product names, stock levels. You knew the exact CSS selectors because you picked them yourself.

AI agents don't work like that. An agent doesn't know what page it's going to visit next. It can't have a pre-written selector for every website on the internet. It needs to visit any URL, extract the useful content, and move on. In real time, not as a batch job.

This changes everything about what a scraping tool needs to do.

The three problems nobody talks about

Problem 1: Token waste

Give a raw HTML page to an LLM and watch your costs explode. A typical webpage is 50,000 to 200,000 tokens of HTML. The actual content? Maybe 800 tokens.

You're paying for navigation menus, footer links, cookie consent banners, inline SVGs, CSS classes, data attributes. All noise. Your LLM processes all of it, reasons over all of it, and bills you for all of it.

This is why output format matters more than speed. A fast scraper that returns raw HTML is useless for AI. You need clean, optimized output that preserves the information but strips the noise.

webclaw runs a 9-step optimization pipeline on every extraction. Image stripping, emphasis removal, link deduplication, stat merging, whitespace collapse. On average, you get 67% fewer tokens compared to standard markdown conversion. On some pages it's over 90%.

Problem 2: Getting blocked

Modern websites use multiple layers of bot protection. Cloudflare, DataDome, AWS WAF, Akamai. Your scraping tool sends a request, gets a 403 or a challenge page, and returns either an error or the challenge HTML pretending it's real content.

Most scraping APIs solve this by routing through proxy networks and charging you per request. Which works, but you're paying 5 to 10 cents per page for something that should cost a fraction of a cent.

The actual fix is TLS fingerprinting. Websites identify bots by looking at the TLS handshake, not just the User-Agent header. If your TLS fingerprint doesn't match a real browser, you get blocked before the server even reads your request.

webclaw impersonates real browser TLS fingerprints. Chrome 142, Firefox 144, whatever the target expects. The request goes out over raw HTTP, no browser needed, but the server sees a legitimate browser connection. This gets through most anti-bot systems without proxies, without extra cost, without 3-second wait times.

Problem 3: JavaScript-rendered content

Some pages need JavaScript to render their content. React apps, Next.js sites, SPAs. The HTML response is just a loading spinner and a bundle URL.

The old solution was headless Chrome. Spin up a full browser, load the page, wait for JavaScript, extract the DOM. It works but it's slow (2-5 seconds per page), resource-heavy (200MB+ of Chromium), and hard to scale.

webclaw takes a smarter approach. Most pages don't actually need JavaScript rendering. The content is in the initial HTML, in server-side rendered markup, in JSON data islands embedded in the page. React hydration data, Next.js payloads, JSON-LD, Contentful CMS data. webclaw extracts all of this from the raw HTML before ever thinking about JavaScript.

For the pages that genuinely need rendering, webclaw uses a lightweight rendering engine instead of Chrome. It's purpose-built for fast JS rendering and significantly lighter than headless Chrome. You get the rendered DOM without the overhead.

What AI agents actually need from a scraping API

After building webclaw and watching how people use it with their agents, the pattern is clear. AI agents need:

Speed. An agent waiting 5 seconds for a page to load is an agent that feels broken to the user. webclaw averages 118ms for static pages. Even with JavaScript rendering, you're under a second.

Any URL, zero config. The agent doesn't know what website it's visiting next. There's no time to write custom selectors or configure extraction rules. The scraping tool needs to handle everything automatically.

Clean, structured output. Markdown for general content. JSON for structured data. Schema-based extraction when you know what shape the data should be. The agent shouldn't need to parse HTML.

Tool integration. The agent needs to call the scraper as a tool, not shell out to a CLI. MCP (Model Context Protocol) is the standard here. webclaw ships an MCP server with 8 tools that works with Claude Desktop, Claude Code, and any MCP-compatible client.

Using webclaw with AI agents

The fastest way to connect webclaw to an AI agent is through MCP. You add webclaw-mcp to your Claude Desktop config and your agent gets access to scraping, crawling, search, sitemap discovery, content diffing, brand extraction, summarization, and structured data extraction.

{
  "mcpServers": {
    "webclaw": {
      "command": "webclaw-mcp"
    }
  }
}

That's it. Your agent can now call scrape with any URL and get back clean markdown. Or call extract with a JSON schema and get structured data. Or call crawl to recursively extract an entire documentation site.

If you're not using MCP, the REST API covers everything. Every extraction feature is a JSON endpoint. You can use it from any language, any framework, any agent architecture.

curl -X POST https://api.webclaw.io/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "format": "llm"}'

The llm format runs the full optimization pipeline and returns the smallest possible output that preserves all the useful content.

Coming from Firecrawl

If you're already using Firecrawl, webclaw has drop-in compatible /v2 endpoints. Same API shape, same request format. Change your base URL and everything keeps working.

Why switch? Faster extraction, better anti-bot handling, lower token counts, and the full optimization pipeline is included in every plan. No per-page charges for anti-bot or JavaScript rendering.

The honest trade-offs

webclaw is not the right tool for everything. If you need to scrape a million product pages from Amazon with rotating residential proxies and CAPTCHA solving at scale, there are services built specifically for that.

webclaw is built for AI agents and LLM applications. Real-time extraction, clean output, tool integration. If your use case is "my agent needs to read web pages," this is what I built it for.

The code is open source and MIT licensed. You can self-host it, run the cloud API, or use the MCP server locally. Whatever fits your stack.

Try it at webclaw.io or check the documentation to get started.