CLI TOOL

Extraction from your terminal.

A single binary that scrapes, crawls, extracts structured data, and tracks changes. Built in Rust. Runs anywhere.

Install

Three ways to get started.

curl -fsSL https://github.com/0xMassi/webclaw/releases/latest/download/webclaw-aarch64-apple-darwin \
  -o webclaw && chmod +x webclaw

Platform support

Pre-built binaries are available for macOS (Apple Silicon & Intel), Linux (x86_64 & aarch64), and Windows (x86_64). Check the releases page for all targets.

Usage

Real commands you can run right now.

Basic extraction

bash

# Basic extraction
webclaw https://example.com

# LLM-optimized format
webclaw https://example.com --format llm

# Filter content
webclaw https://example.com --include "article" --exclude "nav,footer"

Crawling and discovery

bash

# Crawl a site
webclaw https://docs.example.com --crawl --depth 2 --max-pages 50

# Discover URLs
webclaw https://example.com --map

# Batch extract
webclaw url1 url2 url3 --format markdown

LLM features

bash

# LLM extraction (requires Ollama or API key)
webclaw https://example.com --extract-prompt "Get all pricing tiers"
webclaw https://example.com --summarize

# Brand analysis
webclaw https://example.com --brand

Advanced

bash

# Change tracking
webclaw https://example.com -f json > snap.json
webclaw https://example.com --diff-with snap.json

# Browser impersonation
webclaw https://example.com --browser firefox

# Proxy rotation
webclaw https://example.com --proxy-file proxies.txt

LLM features

Extract and summarize commands require either a local Ollama instance (auto-detected at localhost:11434) or an OPENAI_API_KEY / ANTHROPIC_API_KEY environment variable.

All flags

Complete reference for every flag.

Flag	Values	Description
`--format, -f`	markdown, text, llm, json, html	Output format
`--include`	CSS selectors	Only extract matching elements
`--exclude`	CSS selectors	Remove matching elements
`--crawl`	-	Enable BFS crawl mode
`--depth`	1-10	Crawl depth (default: 2)
`--max-pages`	number	Maximum pages to crawl
`--map`	-	Discover URLs via sitemap
`--extract-prompt`	string	LLM extraction prompt
`--summarize`	-	Generate AI summary
`--brand`	-	Extract brand identity
`--diff-with`	file path	Diff against previous snapshot
`--browser`	chrome, firefox, safari	Browser to impersonate
`--proxy-file`	file path	Proxy list for rotation
`--timeout`	seconds	Request timeout (default: 30)
`--output, -o`	file path	Write output to file
`--verbose, -v`	-	Verbose logging

Try it now.

Install the binary, point it at a URL. No account required for local extraction.

VIEW ON GITHUB READ FULL DOCS