Self-Hosting

The OSS release ships a single binary, webclaw-server, that exposes the extraction engine over HTTP. Its JSON contracts overlap with hosted endpoints but are not identical. Stateless, no database, no job queue. Run it in Docker or build from source.

Warning

Self-hosting gives you the extraction pipeline, not the full cloud platform. See the capability matrix below before deciding where to point your traffic.

OSS vs hosted

Capability	Self-hosted	api.webclaw.io
Scrape, map, batch, diff, brand	Yes	Yes
Crawl	Synchronous; default 50 pages, 0 requests no page cap	Async jobs; default 50 pages, configurable cap and credit budget
Extract, summarize	Bring your own LLM (Ollama local, or OpenAI / Anthropic key)	Managed — no LLM keys needed
Bot-protected site handling	No	Yes
JS rendering (SPAs)	No	Yes
Research and hosted account features	No	Yes
Watch (scheduled monitors)	No	Yes
Auth	Single bearer token	OAuth, API keys, multi-tenant billing
Infrastructure	Single binary, stateless	Managed service

Rule of thumb: self-host when you want to keep the extraction pipeline on your own box and point it at sites that don't fight you. Use the hosted API when you hit bot-protected sites, JS-rendered SPAs, or need async crawl jobs. You can also mix both: use local extraction for suitable pages and the hosted API when needed.

Docker

Fastest way to run the server. The image ships three binaries (webclaw CLI, webclaw-mcp, webclaw-server) and picks up on WEBCLAW_PORT, WEBCLAW_HOST, and WEBCLAW_API_KEY as env vars.

Run the server

docker run --rm -p 127.0.0.1:3000:3000 -e WEBCLAW_API_KEY=local-example-key ghcr.io/0xmassi/webclaw:latest webclaw-server

The image binds inside the container, so it requires an API key. The example publishes the port on your host's loopback interface. Replace the example key before sharing access. The trailing webclaw-server is important. Without it, the default CMD runs the CLI and prints its --help, which is correct for docker run IMAGE https://example.com but not for running a server.

Configure authentication

bash

docker run -p 3000:3000 \
  -e WEBCLAW_API_KEY=mysecret \
  ghcr.io/0xmassi/webclaw:latest webclaw-server

When WEBCLAW_API_KEY is set, every request to /v1/* must present Authorization: Bearer mysecret. The comparison is constant-time. /health stays public.

Docker Compose with Ollama

To use /v1/extract and /v1/summarize, you need an LLM. Ollama runs locally for free.

docker-compose.yml

services:
  webclaw:
    image: ghcr.io/0xmassi/webclaw:latest
    command: webclaw-server
    ports:
      - "3000:3000"
    environment:
      - WEBCLAW_API_KEY=mysecret
      - OLLAMA_HOST=http://ollama:11434
    depends_on:
      - ollama

  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama

volumes:
  ollama_data:

Tip

After the stack is up, pull a model: docker compose exec ollama ollama pull qwen3:8b. The extract/summarize endpoints will fall back to OpenAI or Anthropic if Ollama isn't reachable and the corresponding API key is set.

From source

Requires Rust 1.85+ (edition 2024).

bash

git clone https://github.com/0xMassi/webclaw.git
cd webclaw
cargo build --release

Produces three binaries in target/release/:

Binary	Role
`webclaw`	CLI (single-shot extraction, crawl, no server).
`webclaw-server`	Stateless REST API (this page).
`webclaw-mcp`	MCP server over stdio for AI agents.

Start the server

./target/release/webclaw-server --port 3000 --api-key mysecret

Default bind is 127.0.0.1 so the CLI case stays safe on a laptop. Pass --host 0.0.0.0 (or WEBCLAW_HOST=0.0.0.0) when you want to expose it on a network. The Docker image already flips this to 0.0.0.0.

Environment variables

The following variables configure the OSS server. Set WEBCLAW_API_KEY before binding beyond loopback; public binding without a key is refused unless WEBCLAW_ALLOW_OPEN_PUBLIC=1 is explicitly set.

Server

Variable	Default	Description
`WEBCLAW_PORT`	3000	HTTP port.
`WEBCLAW_HOST`	127.0.0.1 (binary), 0.0.0.0 (Docker)	Bind address.
`WEBCLAW_API_KEY`	--	Bearer token required on /v1/*. Without a key, loopback development is allowed.
`RUST_LOG`	info,webclaw_server=info	Tracing filter (e.g. `debug`, `webclaw_fetch=trace`).

LLM providers (for /v1/extract and /v1/summarize)

Provider chain tries Ollama first, then OpenAI, then Anthropic. Set at least one. If none are reachable the endpoints return 422 with a readable error.

Variable	Default	Description
`OLLAMA_HOST`	http://localhost:11434	Ollama API endpoint.
`OLLAMA_MODEL`	qwen3:8b	Default model.
`OPENAI_API_KEY`	--	Fallback provider.
`OPENAI_BASE_URL`	--	OpenAI-compatible endpoint (for proxies or self-hosted inference).
`OPENAI_RESPONSE_FORMAT_TYPE`	json_object	Response format for OpenAI-compatible backends: `json_object`, `json_schema`, or `text`. Use `text` or `json_schema` for LM Studio.
`ANTHROPIC_BASE_URL`	https://api.anthropic.com/v1	Anthropic-compatible endpoint. Set this when routing Anthropic requests through a compatible gateway.
`ANTHROPIC_API_KEY`	--	Fallback provider.

Tip

Official OpenAI behavior stays unchanged when OPENAI_RESPONSE_FORMAT_TYPE is unset. Set it only for compatible backends that require json_schema or text.

Endpoints

Check each OSS request and response against your integration. Hosted-only options, asynchronous jobs, and account features do not become available by changing the base URL.

GET/health

Liveness check. Always open, even with auth enabled.

POST/v1/scrape

Extract one URL. Formats: markdown, text, llm, json, html.

POST/v1/crawl

Synchronous BFS crawl. Default 50 pages; max_pages: 0 removes the page cap.

POST/v1/map

Discover URLs via robots.txt + sitemap.xml.

POST/v1/batch

Fetch multiple URLs; no URL-count cap, at most 20 concurrent requests. Supports NDJSON streaming.

POST/v1/search

Search with your own SERPER_API_KEY. Returns 501 when not configured; result scraping defaults to false.

POST/v1/extract

LLM-backed structured extraction. Pass a JSON Schema or a prompt. Needs an LLM provider.

POST/v1/summarize

N-sentence summary. Needs an LLM provider.

POST/v1/diff

Compare current content against a prior snapshot.

POST/v1/brand

Extract colors, fonts, logo, favicon from a page.

Smoke test

# Liveness
curl http://localhost:3000/health

# Scrape a page
curl -X POST http://localhost:3000/v1/scrape \
  -H "content-type: application/json" \
  -H "authorization: Bearer mysecret" \
  -d '{"url":"https://example.com","formats":["markdown"]}'

Why your self-hosted server still sees 403s on some sites

The OSS server uses the same HTTP client as the CLI, which is enough for most of the open web. It does not do: bot-protection handling, JS rendering, or managed proxying and session handling. Those run on dedicated infrastructure behind api.webclaw.io and are intentionally closed source. If you need them, point the stubborn targets at the hosted API while keeping the easy ones on your box.