Native MCP server, works with Claude Desktop, Cursor, Windsurf out of the box
webclaw-mcp (stdio)
Real-time web data for AI agents.
Give your AI agent live eyes on the web. One MCP install.
AI agents that can only read their training data are stuck in the past. webclaw exposes 12 web tools over the Model Context Protocol so Claude, Cursor, and any MCP-compatible agent can scrape, crawl, search, and extract from the live web in real time.
Why webclaw
Built for ai agents.
Automatic bot-protection handling for sites that block plain requests
12 tools exposed: scrape, search, crawl, map, batch, extract, summarize, diff, brand, research, list_extractors, vertical_scrape
Choose cache settings and timeouts for your agent’s workload
Structured output (JSON, markdown) that agents can reason over
What you get
Everything this use case needs.
- MCP server with 12 tools over stdio
- Works with Claude Desktop, Claude Code, Cursor, Windsurf, OpenCode
- Automatic bot-protection handling
- Real-time web access (no stale training data)
- Structured JSON or markdown output
Where it fits
Built for the messy parts.
Your AI agent can write code, analyze documents, and reason about complex problems, but it cannot read a webpage reliably. Most scraping tools fail on modern bot-protected sites, and integrating them requires custom middleware for every agent framework.
webclaw-mcp is a Model Context Protocol server that exposes 8 web extraction tools over stdio. Install it once and your AI agent gets scrape, crawl, search, batch, extract, summarize, diff, and screenshot tools that handle bot-protected sites automatically. No middleware, no custom code.
How it works
Build it step by step.
The real flow, one step at a time. Switch between TypeScript, Python, and cURL on any snippet.
- 1
Install the MCP server
Add webclaw over MCP (an API key unlocks the cloud tools) so the agent gets 12 web tools over stdio.
claude_desktop_config.json{ "mcpServers": { "webclaw": { "command": "npx", "args": ["-y", "@webclaw/mcp"], "env": { "WEBCLAW_API_KEY": "wc_..." } } }} - 2
Agent calls a tool
When the agent needs live data it invokes scrape, search, or extract instead of relying on stale training data.
// The agent invokes a tool when it needs live dataconst result = await webclaw.scrape({ url: "https://news.ycombinator.com", formats: ["markdown"],}); - 3
Fetch and clean
webclaw fetches the URL, handles bot protection, and returns structured markdown or JSON.
// webclaw handled bot protection and returned clean contentconst markdown = result.markdown;const title = result.metadata?.title;console.log(markdown); - 4
Reason over the result
The agent reads the clean output and continues its task with fresh, real-world context.
webclaw-mcp works with any MCP-compatible client: Claude Desktop, Claude Code, Cursor, Windsurf, OpenCode, Codex, Antigravity, and any custom agent built on the official MCP SDK.
For AI agents
Or hand it to your agent.
Add the webclaw MCP server to Claude, Cursor, or any MCP client, then paste this prompt. The agent calls the webclaw tools and hands the result back to your model — no code to write.
PROMPT FOR YOUR AGENT
Using the webclaw tools, get me current real-world information on [your topic or question] instead of relying on what you already know. Start by calling search with the query [your search query] to find the most relevant live pages, then call scrape on the top results to pull their full, clean content. If you need specific structured fields from any page, call extract with a schema covering [the fields you care about, e.g. name, price, date, author]; if a source is long, call summarize to condense it first. Reason over the fresh results and reply with a short written answer that directly addresses my question, followed by a table of the key facts you found and a list of the source URLs you used so I can verify each one.
Ready to build? Start extracting.
Cancel anytime. Clean, structured data on every call.