RAW HTTP — NO HEADLESS BROWSER OVERHEAD•MARKDOWN · JSON · HTML · LLM-READY FORMATS•MCP SERVER FOR AI AGENTS•TLS FINGERPRINT IMPERSONATION•EXTRACT · SUMMARIZE · DIFF · BRAND•SITEMAP DISCOVERY & DEEP CRAWLING•SELF-HOST OR USE OUR CLOUD API•BUILT IN RUST — FAST BY DEFAULT•DEEP RESEARCH — AI SYNTHESIZES REPORTS FROM 50+ SOURCES•WEB SEARCH — QUERY AND SCRAPE SEARCH RESULTS IN ONE CALL•AGENT SCRAPE — GIVE A GOAL, AI EXTRACTS WHAT YOU NEED•URL MONITORING — WATCH PAGES FOR CHANGES WITH WEBHOOKS•BONUS CREDITS — EARN FREE CREDITS BY STARRING AND REFERRING•RAW HTTP — NO HEADLESS BROWSER OVERHEAD•MARKDOWN · JSON · HTML · LLM-READY FORMATS•MCP SERVER FOR AI AGENTS•TLS FINGERPRINT IMPERSONATION•EXTRACT · SUMMARIZE · DIFF · BRAND•SITEMAP DISCOVERY & DEEP CRAWLING•SELF-HOST OR USE OUR CLOUD API•BUILT IN RUST — FAST BY DEFAULT•DEEP RESEARCH — AI SYNTHESIZES REPORTS FROM 50+ SOURCES•WEB SEARCH — QUERY AND SCRAPE SEARCH RESULTS IN ONE CALL•AGENT SCRAPE — GIVE A GOAL, AI EXTRACTS WHAT YOU NEED•URL MONITORING — WATCH PAGES FOR CHANGES WITH WEBHOOKS•BONUS CREDITS — EARN FREE CREDITS BY STARRING AND REFERRING•

CLI TOOL

Extraction from your terminal.

A single binary that scrapes, crawls, extracts structured data, runs web searches, deep research, and AI-guided agents. Built in Rust. Runs anywhere.

Install

Three ways to get started.

curl -fsSL https://github.com/0xMassi/webclaw/releases/latest/download/webclaw-aarch64-apple-darwin \
  -o webclaw && chmod +x webclaw
Platform support
Pre-built binaries are available for macOS (Apple Silicon & Intel), Linux (x86_64 & aarch64), and Windows (x86_64). Check the releases page for all targets.

Usage

Real commands you can run right now.

Basic extraction

bash
# Basic extraction
webclaw https://example.com

# LLM-optimized format
webclaw https://example.com --format llm

# Filter content
webclaw https://example.com --include "article" --exclude "nav,footer"

Crawling and discovery

bash
# Crawl a site
webclaw https://docs.example.com --crawl --depth 2 --max-pages 50

# Discover URLs
webclaw https://example.com --map

# Batch extract
webclaw url1 url2 url3 --format markdown

LLM features

bash
# LLM extraction (requires Ollama or API key)
webclaw https://example.com --extract-prompt "Get all pricing tiers"
webclaw https://example.com --summarize

# Brand analysis
webclaw https://example.com --brand

Search, research, and agents

bash
# Web search
webclaw --search "best rust web frameworks 2026"

# Deep research with AI synthesis
webclaw --research "compare Next.js vs Remix for production apps"

# AI-guided scraping
webclaw https://example.com --agent "find the pricing page and extract all plan details"

Advanced

bash
# Change tracking
webclaw https://example.com -f json > snap.json
webclaw https://example.com --diff-with snap.json

# Browser impersonation
webclaw https://example.com --browser firefox

# Proxy rotation
webclaw https://example.com --proxy-file proxies.txt
LLM features
Extract, summarize, research, and agent commands require either a local Ollama instance (auto-detected at localhost:11434) or an OPENAI_API_KEY / ANTHROPIC_API_KEY environment variable.

All flags

Complete reference for every flag.

FlagValuesDescription
--format, -fmarkdown, text, llm, json, htmlOutput format
--includeCSS selectorsOnly extract matching elements
--excludeCSS selectorsRemove matching elements
--crawl-Enable BFS crawl mode
--depth1-10Crawl depth (default: 2)
--max-pagesnumberMaximum pages to crawl
--map-Discover URLs via sitemap
--extract-promptstringLLM extraction prompt
--summarize-Generate AI summary
--brand-Extract brand identity
--diff-withfile pathDiff against previous snapshot
--search"query"Web search via Serper
--research"query"Deep research with AI synthesis
--agent"goal"AI-guided scraping with a goal
--browserchrome, firefox, safariBrowser to impersonate
--proxy-filefile pathProxy list for rotation
--timeoutsecondsRequest timeout (default: 30)
--output, -ofile pathWrite output to file
--verbose, -v-Verbose logging

Try it now.

Install the binary, point it at a URL. No account required for local extraction.

VIEW ON GITHUBREAD FULL DOCS