REST API

The webclaw REST API gives you programmatic access to the full extraction engine. The endpoint reference describes JSON requests and responses; streaming endpoints use SSE or NDJSON.

Base URL

Use the cloud endpoint for managed infrastructure, or point at your own instance when self-hosting.

# Cloud (managed)
https://api.webclaw.io

# Self-hosted
http://localhost:3000

Authentication

Authenticated hosted operations require an API key in the Authorization header. Public routes such as health and the verified free tools have separate access rules.

Authorization: Bearer <api_key>

Cloud: Create API keys from your dashboard at webclaw.io. Keys are prefixed with wc_.

Self-hosted: Pass --api-key when starting the server, or set the WEBCLAW_API_KEY environment variable. If neither is set, the server runs without authentication.

Request format

All POST endpoints accept a JSON body. Set the Content-Type header accordingly.

Content-Type: application/json

Response format

Non-streaming API responses use JSON. Successful responses return endpoint-specific data. Errors use a consistent shape:

{
  "error": "Human-readable error message"
}

Rate limiting

Cloud API rate limits are based on your plan tier. Self-hosted instances have no rate limits by default. See the Cloud API page for plan details.

Output formats

The /v1/scrape endpoint supports 9 output formats. Pass one or more in the formats array. Request screenshots with the separate screenshot boolean. Query and attributes cannot be combined with browser actions or screenshots.

Document parsing

webclaw auto-detects document types from Content-Type headers and file extensions. In addition to PDF, the following formats are supported natively and converted to your requested output format:

YouTube transcript extraction

YouTube URLs (youtube.com/watch) are auto-detected and processed differently. webclaw extracts structured markdown containing the video title, channel name, view count, publish date, duration, description, and transcript text when captions can be retrieved.

curl -X POST https://api.webclaw.io/v1/scrape \
  -H "Authorization: Bearer wc_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ", "formats": ["markdown"]}'

Endpoints

The full list of available endpoints.

Core endpoints

X (Twitter)

Monitor X profiles, searches, lists, and replies for new posts. Paid feature; free or lapsed accounts get a 403.

Webhooks

Firecrawl v2 compatibility

Compatibility routes for supported Firecrawl v2 operations. Change the base URL and API key, then verify the parameters and response fields your client uses.

Results and history

Authenticated cloud support endpoints for dashboard history, request inspection, replay output lookup, and cached result lookup.

curl https://api.webclaw.io/v1/results/run/87ac93f1-1ff8-4622-a1f4-1ac2a8117fa7 \
  -H "Authorization: Bearer wc_your_api_key"

Utility

POST /v1/search

Perform a web search and optionally scrape all result pages in parallel. Combines discovery and extraction into a single call.

Request body

{
  "query": "best rust web frameworks 2026",
  "num_results": 5,
  "scrape": true,
  "formats": ["markdown"],
  "country": "us",
  "lang": "en"
}

Example

curl -X POST https://api.webclaw.io/v1/search \
  -H "Authorization: Bearer wc_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "best rust web frameworks 2026",
    "num_results": 3,
    "formats": ["llm"],
    "country": "us"
  }'

POST /v1/extract

Extract structured JSON data from a public URL. Provide a JSON schema to guide the output, or a natural language prompt. Check extracted values against the source. When only a prompt is provided (no schema), the LLM generates a JSON schema first, then extracts data matching it. The response includes a generated_schema field showing the auto-generated schema.

Prompt-to-schema example

curl -X POST https://api.webclaw.io/v1/extract \
  -H "Authorization: Bearer wc_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "prompt": "Extract all pricing tiers with name, price, and features"
  }'

When only prompt is provided, the response includes both the extracted data and the generated schema:

{
  "data": {
    "tiers": [
      {"name": "Hobby", "price": 9, "features": ["1 seat", "Community support"]},
      {"name": "Pro", "price": 49, "features": ["5 seats", "Priority support"]}
    ]
  },
  "generated_schema": {
    "type": "object",
    "properties": {
      "tiers": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "name": {"type": "string"},
            "price": {"type": "number"},
            "features": {"type": "array", "items": {"type": "string"}}
          }
        }
      }
    }
  }
}

Enhanced scrape parameters

The /v1/scrape endpoint supports additional parameters for browser automation, screenshots, mobile emulation, cache control, page-level Q&A, and attribute extraction.

Query format example

curl -X POST https://api.webclaw.io/v1/scrape \
  -H "Authorization: Bearer wc_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "formats": ["query"],
    "query": "What is the price of the Pro plan?"
  }'

Attributes format example

curl -X POST https://api.webclaw.io/v1/scrape \
  -H "Authorization: Bearer wc_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": ["attributes"],
    "attribute_selectors": [
      {"selector": "img", "attribute": "alt"},
      {"selector": "a", "attribute": "href"}
    ]
  }'

Enhanced crawl parameters

The /v1/crawl endpoint now supports path filtering, subdomain and external link control, and webhook notifications.

Crawl sub-endpoints

Enhanced crawl example

curl -X POST https://api.webclaw.io/v1/crawl \
  -H "Authorization: Bearer wc_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.stripe.com",
    "max_depth": 3,
    "max_pages": 200,
    "include_patterns": ["/docs/**"],
    "exclude_patterns": ["/docs/changelog/**"],
    "allow_subdomains": false,
    "webhook_url": "https://your-app.com/webhooks/crawl-done"
  }'

Webhooks

Register webhook URLs to receive POST notifications when async operations (crawls, batch jobs) complete. Manage webhooks with full CRUD.

Create webhook request

{
  "url": "https://your-app.com/webhooks/webclaw",
  "events": ["crawl.completed", "crawl.failed", "batch.completed"]
}

Example

curl -X POST https://api.webclaw.io/v1/webhooks \
  -H "Authorization: Bearer wc_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-app.com/webhooks/webclaw",
    "events": ["crawl.completed"]
  }'

Firecrawl v2 compatibility

These routes translate supported Firecrawl v2 requests and responses. They do not guarantee complete feature parity. Test your client's options and error handling before migrating.

Quick example

curl -X POST https://api.webclaw.io/v1/scrape \
  -H "Authorization: Bearer wc_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["markdown"]}'