Scrape
Extract content from a single URL. This is the core endpoint -- one URL in, clean structured content out.
POST
/v1/scrapeExtract content from a single URL in one or more output formats.
Markdown and structured extraction
Request extract alongside Markdown to get both from the same page. Provide an extract object with a JSON Schema, a nonempty prompt, or both. The page costs one credit, including extraction. The response includes extract beside the requested content, metadata and cache status. Unknown request fields return an error.
json
{
"url": "https://example.com",
"formats": ["markdown", "extract"],
"extract": {
"prompt": "Extract the main heading",
"schema": {
"type": "object",
"properties": { "title": { "type": "string" } },
"required": ["title"]
}
}
}The existing Extract endpoint remains available with its data response envelope.
Request body
json
{
"url": "https://example.com",
"formats": ["markdown", "llm", "text", "json"],
"include_selectors": [".article-content"],
"exclude_selectors": ["nav", ".sidebar"],
"only_main_content": true
}Parameters
| Field | Type | Required | Description |
|---|---|---|---|
url | string | Yes | The URL to scrape. |
formats | string[] | No | Output formats to include. Options: markdown, llm, text, json, extract (with extraction options). Defaults to ["markdown"]. |
include_selectors | string[] | No | CSS selectors to extract exclusively. Only content matching these selectors will be included. |
exclude_selectors | string[] | No | CSS selectors to remove from the page before extraction. |
only_main_content | boolean | No | Defaults to false: automatic cleanup can still remove headers and footers. When true, selects the first article, main, or role=main element and keeps its scope even when short or empty. If none exists, uses automatic extraction. Include selectors take precedence; exclusions still apply. |
Tip
Use
formats for an array, or format for one string. Omission or an empty array defaults to Markdown. Supplying both, null, the wrong type, or an unsupported name returns 400. The text and llm formats use top-level response fields; json uses extraction. Additional formats are links, rawHtml, attributes, and query. Request a screenshot with the separate screenshot: true parameter. Query and attributes cannot be combined with actions or screenshots; these combinations return 400.Detected empty default extractions and recognized access, login, or upstream errors can return 502 with an error_code, including empty_content, bot_protection_unbypassed, upstream_error_page, or login_required. An include selector or a matching main region may intentionally return empty content. Target-site availability varies; check the content before treating a scrape as complete.
Tip
The
llm format runs a content optimization pipeline that strips images, collapses whitespace, and deduplicates links. Token savings depend on the page and tokenizer; check that the result retains the information your task needs.Response
The response includes the requested formats alongside extracted metadata. Only the formats you request are populated.
json
{
"url": "https://example.com",
"metadata": {
"title": "Example Domain",
"description": "This domain is for use in illustrative examples.",
"author": null,
"published_date": null,
"language": "en",
"site_name": "Example",
"image": null,
"favicon": "https://example.com/favicon.ico",
"word_count": 1234
},
"markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
"text": "Example Domain\n\nThis domain is for use in illustrative examples...",
"llm": "> URL: https://example.com\n> Title: Example Domain\n\nThis domain is for use in illustrative examples...\n\n## Links\n- ...",
"extraction": { "metadata": { "title": "Example Domain", "word_count": 28 }, "content": { "markdown": "# Example Domain", "plain_text": "Example Domain", "links": [], "images": [], "code_blocks": [] } }
}Metadata fields
| Field | Type | Description |
|---|---|---|
title | string | Page title from OG, meta, or title tag. |
description | string | Page description from meta or OG tags. |
author | string? | Author name if detected. |
published_date | string? | Publication date if found in metadata. |
language | string? | Page language code (e.g. "en"). |
site_name | string? | Site name from OG metadata. |
image | string? | Primary image URL (OG or Twitter Card). |
favicon | string? | Favicon URL. |
word_count | number | Total word count of extracted content. |
Examples
Basic extraction
curl
curl -X POST https://api.webclaw.io/v1/scrape \
-H "Authorization: Bearer wc_your_api_key" \
-H "Content-Type: application/json" \
-d '{
"url": "https://openai.com/blog/gpt-4",
"formats": ["markdown"]
}'LLM-optimized with selector filtering
curl
curl -X POST https://api.webclaw.io/v1/scrape \
-H "Authorization: Bearer wc_your_api_key" \
-H "Content-Type: application/json" \
-d '{
"url": "https://docs.stripe.com/payments/checkout",
"formats": ["llm"],
"include_selectors": [".content-container"],
"exclude_selectors": ["nav", "footer", ".sidebar"],
"only_main_content": true
}'Multiple output formats
curl
curl -X POST https://api.webclaw.io/v1/scrape \
-H "Authorization: Bearer wc_your_api_key" \
-H "Content-Type: application/json" \
-d '{
"url": "https://en.wikipedia.org/wiki/Web_scraping",
"formats": ["markdown", "llm", "text"]
}'Note
PDFs are auto-detected via Content-Type. If the URL serves a PDF, webclaw extracts text using its PDF engine instead of HTML parsing.
Error responses
400 Bad Request
{
"error": "Missing required field: url"
}401 Unauthorized
{
"error": "invalid or missing API key"
}503 Service Unavailable — query provider
{
"error": "The page question could not be answered. Please retry later."
}