MCP Server

The webclaw MCP (Model Context Protocol) server exposes the full extraction engine as tools that AI agents can call directly. Works with Claude Desktop, Claude Code, Cursor, Windsurf, OpenCode, Codex, Antigravity, and any MCP-compatible client.

What is MCP

Model Context Protocol is an open standard for connecting AI models to external tools and data sources. Instead of making HTTP calls manually, an AI agent discovers available tools through the MCP server and calls them natively. The webclaw MCP server communicates over stdio transport and exposes 8 tools covering scraping, crawling, extraction, and more.

Setup

Claude Desktop

Add webclaw to your Claude Desktop config file:

{
  "mcpServers": {
    "webclaw": {
      "command": "webclaw-mcp",
      "env": {
        "WEBCLAW_API_KEY": "<YOUR_API_KEY>"
      }
    }
  }
}

Replace webclaw-mcp with the full path if not in PATH. The WEBCLAW_API_KEY enables automatic cloud fallback for bot-protected sites (Cloudflare, DataDome, AWS WAF) and JS-rendered SPAs. Without it, extraction works for ~80% of sites via local HTTP.

Claude Code

claude mcp add webclaw webclaw-mcp

Or add the JSON config above to your Claude Desktop config file. Claude Code auto-discovers MCP servers from the same config.

Cursor

Add webclaw to your Cursor MCP config:

{
  "mcpServers": {
    "webclaw": {
      "command": "webclaw-mcp",
      "env": {
        "WEBCLAW_API_KEY": "<YOUR_API_KEY>"
      }
    }
  }
}

Windsurf

Add webclaw to your Windsurf MCP config:

{
  "mcpServers": {
    "webclaw": {
      "command": "webclaw-mcp",
      "env": {
        "WEBCLAW_API_KEY": "<YOUR_API_KEY>"
      }
    }
  }
}

OpenCode

Add webclaw to your OpenCode config:

{
  "mcp": {
    "webclaw": {
      "type": "local",
      "command": ["~/.webclaw/webclaw-mcp"],
      "enabled": true
    }
  }
}

Codex

Add webclaw to your Codex config. Codex supports both a CLI and desktop app:

[mcp_servers.webclaw]
command = "~/.webclaw/webclaw-mcp"
args = []
enabled = true

Antigravity

Antigravity uses the same mcpServers JSON format as Claude Desktop:

{
  "mcpServers": {
    "webclaw": {
      "command": "webclaw-mcp",
      "env": {
        "WEBCLAW_API_KEY": "<YOUR_API_KEY>"
      }
    }
  }
}

Other MCP clients

Any MCP client that supports stdio transport can connect to webclaw-mcp. Point the client at the binary and it will discover all available tools through the standard MCP handshake.

Smart Fetch

The MCP server uses a local-first architecture. Most scrapes happen locally over HTTP (free, no API credits). When bot protection or JS rendering is detected, it automatically falls back to the webclaw cloud API which has antibot solvers.

Environment variables

Tools

The MCP server exposes 12 tools. Each tool maps to a corresponding REST API endpoint.

Example conversations

Here is how an AI agent might use the webclaw MCP tools in practice.

{
  "url": "https://stripe.com/pricing",
  "prompt": "Extract all plan names, monthly prices, and included features"
}

{
  "url": "https://nextjs.org/docs"
}

{
  "urls": [
    "https://nextjs.org/docs",
    "https://nextjs.org/docs/getting-started",
    "https://nextjs.org/docs/routing",
    "https://nextjs.org/docs/rendering",
    "https://nextjs.org/docs/data-fetching"
  ],
  "format": "llm"
}