RAW HTTP — NO HEADLESS BROWSER OVERHEAD•MARKDOWN · JSON · HTML · LLM-READY FORMATS•MCP SERVER FOR AI AGENTS•TLS FINGERPRINT IMPERSONATION•EXTRACT · SUMMARIZE · DIFF · BRAND•SITEMAP DISCOVERY & DEEP CRAWLING•SELF-HOST OR USE OUR CLOUD API•BUILT IN RUST — FAST BY DEFAULT•DEEP RESEARCH — AI SYNTHESIZES REPORTS FROM 50+ SOURCES•WEB SEARCH — QUERY AND SCRAPE SEARCH RESULTS IN ONE CALL•AGENT SCRAPE — GIVE A GOAL, AI EXTRACTS WHAT YOU NEED•URL MONITORING — WATCH PAGES FOR CHANGES WITH WEBHOOKS•BONUS CREDITS — EARN FREE CREDITS BY STARRING AND REFERRING•RAW HTTP — NO HEADLESS BROWSER OVERHEAD•MARKDOWN · JSON · HTML · LLM-READY FORMATS•MCP SERVER FOR AI AGENTS•TLS FINGERPRINT IMPERSONATION•EXTRACT · SUMMARIZE · DIFF · BRAND•SITEMAP DISCOVERY & DEEP CRAWLING•SELF-HOST OR USE OUR CLOUD API•BUILT IN RUST — FAST BY DEFAULT•DEEP RESEARCH — AI SYNTHESIZES REPORTS FROM 50+ SOURCES•WEB SEARCH — QUERY AND SCRAPE SEARCH RESULTS IN ONE CALL•AGENT SCRAPE — GIVE A GOAL, AI EXTRACTS WHAT YOU NEED•URL MONITORING — WATCH PAGES FOR CHANGES WITH WEBHOOKS•BONUS CREDITS — EARN FREE CREDITS BY STARRING AND REFERRING•

MCP SERVER

Web scraping for OpenAI Codex

Give Codex reliable web access through MCP.

Codex supports MCP servers in the CLI and IDE extension using shared config.toml configuration. Add webclaw-mcp and Codex can fetch live documentation, scrape protected pages, summarize sources, and extract structured data while it edits your codebase.

Setup

~/.codex/config.toml

[mcp_servers.webclaw]
command = "webclaw-mcp"

[mcp_servers.webclaw.env]
WEBCLAW_API_KEY = "wc_..."

Why webclaw for Codex

  • Fits Codex's MCP config.toml format
  • Works from both Codex CLI and IDE extension configuration
  • Fast scrape and search tools without browser setup
  • LLM-ready markdown keeps Codex context focused

Common use cases

  • Let Codex inspect current docs before changing code
  • Research libraries, API references, and migration guides
  • Scrape product pages or support docs into implementation notes
  • Build repeatable web-data workflows as coding tasks

Frequently asked questions

Where do I configure webclaw for Codex?

Add a [mcp_servers.webclaw] table to ~/.codex/config.toml, or to a project-scoped .codex/config.toml in a trusted project. Codex CLI and the IDE extension share this configuration.

Can I add webclaw with the Codex CLI instead of editing TOML?

Yes. Codex also supports codex mcp add for managing MCP servers from the CLI. Editing config.toml gives you the exact final configuration in one place.

Get started

500 pages/month free. No credit card. Open source.

GET STARTEDCODEX WEBSITE

OTHER INTEGRATIONS

LangChainLlamaIndexCrewAIMastraClaude DesktopClaude CodeCursorn8nZapier