MCP server
Web scraping inside Cursor IDE
Give your Cursor agent access to the live web.
Cursor is an AI-first code editor with full MCP support. Add webclaw over MCP and your Cursor agent can read documentation, scrape API references, and research codebases on the live web while you're coding.
You are set up. Now try it.
Check it connected
Reload Cursor, then open Customize → MCP. webclaw should be listed with a green dot and 12 tools.
Then paste this
PROMPT FOR YOUR AGENT
Use webclaw to scrape the API reference at <url>, then write a typed client for the three endpoints this project actually calls.
Why webclaw for Cursor
- Installs as a standard MCP server
- Cursor agent gets 12 tools automatically
- Reliable access to docs sites and forums that block plain requests
- Fast enough to not slow down your flow
Common use cases
- Cursor agent reads real library documentation in real time
- Scrape API specs and reference pages on demand
- Research Stack Overflow or GitHub issues while coding
- Extract code examples from docs
Cursor reads ~/.cursor/mcp.json on startup. Add the webclaw entry there, then restart Cursor. The tools will show up in the MCP panel.