Documentation

Introduction

webclaw is a web extraction toolkit built in Rust. It converts public web content to Markdown, JSON, plain text, and output optimized for LLMs. Results depend on the source page and its access restrictions.

Three binaries, one engine

webclaw ships as three standalone binaries, all powered by the same extraction core:

webclaw

The CLI. Extract, crawl, summarize, and track changes from the terminal. Pipe output to files, chain with other tools, or use interactively.

webclaw-server

The stateless OSS REST server. Offers extraction endpoints with optional Bearer authentication. Its crawl endpoint returns results synchronously; hosted jobs and account features are separate.

webclaw-mcp

The MCP server. Exposes 12 tools over the Model Context Protocol (stdio transport) for use with Claude Desktop, Claude Code, Cursor, Windsurf, OpenCode, Codex, Antigravity, and any MCP-compatible AI client.

Key features

Fast and lightweight. Extracts static HTML over HTTP. Run the supported binaries on a laptop or server; native builds need the documented toolchain.

Hosted output formats. Markdown, plain text, JSON, LLM-optimized, links, rawHtml, attributes, query, and structured extract. Screenshots use a separate option; query and attributes cannot combine with screenshots or browser actions.

CSS selector filtering. Include or exclude content by CSS selector. Extract only article bodies, skip navbars and footers.

Crawling and sitemap discovery. BFS same-origin crawler with configurable depth, concurrency, and delay. Sitemap.xml and robots.txt discovery built in.

Content change tracking. Snapshot pages as JSON and diff against future extractions to detect what changed.

Brand extraction. Extract brand identity -- colors, fonts, logo URL, favicon -- from DOM structure and CSS analysis.

LLM integration. JSON schema extraction, prompt-based extraction, and summarization. Local tools need a configured model provider; the hosted API manages model access.

YouTube transcript extraction. The hosted API detects YouTube URLs and attempts metadata and caption extraction. Captions may be unavailable; inspect the returned content.

PDF extraction. Auto-detected via Content-Type header. Text extraction from text-based PDF documents. Scanned pages may require a separate OCR workflow.

Proxy rotation. Proxy pool rotation from a file. Auto-loads proxies.txt from the working directory.

Handles protected sites. The hosted API can retry protected or JavaScript-heavy pages. Access restrictions and upstream failures can still prevent extraction.

Open source

The CLI, MCP server, extraction engine, and OSS REST server are AGPL-3.0 licensed. The managed cloud platform is separate. The open-source repository is at github.com/0xMassi/webclaw.

Architecture

The project is a Rust workspace split into focused crates. The core extraction engine has zero network dependencies and is WASM-compatible.

workspace

webclaw/
  crates/
    webclaw-core/     # Extraction engine. WASM-safe. Zero network deps.
                      # Readability scoring, noise filtering, markdown
                      # conversion, LLM optimization, CSS selector
                      # filtering, diff engine, brand extraction.

    webclaw-fetch/    # HTTP client. Crawler. Sitemap discovery.
                      # Batch operations. Proxy pool rotation.

    webclaw-llm/      # LLM provider chain (Ollama -> OpenAI -> Anthropic).
                      # JSON schema extraction, prompt extraction,
                      # summarization.

    webclaw-pdf/      # PDF text extraction via pdf-extract.

    webclaw-server/   # Stateless REST API; synchronous crawl.

    webclaw-mcp/      # MCP server over stdio transport. 12 tools for
                      # AI agents.

    webclaw-cli/      # CLI binary.

webclaw-core

The pure extraction engine. Takes raw HTML as a string, returns structured output. No network calls, no I/O -- just parsing and scoring. This is what makes the core WASM-compatible.

Key modules: readability-style content scoring with text density and link density penalties, shared noise filtering (tags, ARIA roles, class/ID patterns, Tailwind-safe), JSON data island extraction for React SPAs and Next.js, HTML to markdown conversion with URL resolution, and an LLM optimization pipeline.

webclaw-fetch

The HTTP layer. Fetches pages and reports fetch failures. Handles BFS crawling with configurable depth and concurrency, sitemap.xml and robots.txt discovery, multi-URL batch operations, and proxy pool rotation.

webclaw-llm

LLM provider chain with automatic fallback: tries Ollama first (local, no API key needed), then OpenAI, then Anthropic. Supports JSON schema extraction, prompt-based extraction, and summarization.

Note

The core crate never makes network requests. It takes &str HTML and returns structured data. All HTTP, LLM calls, and PDF parsing happen in the other crates.

Next steps

Getting started -- install webclaw and run your first extraction.

CLI reference -- common commands and options for the command-line tool.

REST API -- programmatic access to the full extraction engine.