Extract structured data from any URL in one call

You scraped the page. You got clean markdown. Now what?

If you're building a price comparison tool, you don't need the whole article. You need the price, the product name, and whether it's in stock. If you're enriching leads, you need the company name, team size, and tech stack. Not the entire "About Us" page converted to markdown.

Most scraping tools stop at "here's the content." They give you text and leave the parsing to you. Which means you're back to writing regex, CSS selectors, or feeding the entire page to an LLM with a prompt like "please find the price somewhere in here."

There's a better way.

Why selectors break

The traditional approach to pulling specific data from a webpage is CSS selectors. Find the element, grab the text.

price = soup.select_one(".product-price .amount").text

This works until it doesn't. And it always stops working.

The site redesigns. The class name changes from product-price to pdp-price-container. The price moves from a <span> to a <div>. The format changes from "$29.99" to "US$29.99/mo". Your selector returns None and your pipeline breaks at 3am.

Selectors are brittle because they depend on implementation details. You're coupling your data extraction to someone else's frontend code. Every deploy on their end is a potential breakpoint on yours.

For one site, you can maintain selectors. For ten sites, it's a part time job. For "any URL an agent decides to visit," it's impossible.

Schema-based extraction

webclaw's /v1/extract endpoint takes a different approach. You describe what data you want as a JSON schema. The extraction engine reads the page, understands the content, and returns data matching your schema.

No selectors. No XPath. No regex. You define the shape of the data you need, webclaw fills it in.

curl -X POST https://api.webclaw.io/v1/extract \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://store.example.com/product/wireless-headphones",
    "schema": {
      "type": "object",
      "properties": {
        "product_name": {"type": "string"},
        "price": {"type": "number"},
        "currency": {"type": "string"},
        "in_stock": {"type": "boolean"},
        "rating": {"type": "number"},
        "review_count": {"type": "integer"}
      }
    }
  }'

Response:

{
  "data": {
    "product_name": "Sony WH-1000XM5",
    "price": 279.99,
    "currency": "USD",
    "in_stock": true,
    "rating": 4.7,
    "review_count": 3842
  }
}

The site can redesign completely. As long as the information is somewhere on the page, the extraction still works. You're extracting meaning, not DOM positions.

How it works under the hood

The extraction pipeline has three steps.

First, webclaw fetches the page with TLS fingerprinting, the same way it handles any scrape. If the page needs JavaScript rendering, it renders. If it's behind anti-bot protection, it gets through. The extract endpoint inherits all of webclaw's fetch capabilities.

Second, the page content gets cleaned through the same 9-step optimization pipeline from the regular scrape. Navigation, ads, cookie banners, footers. All stripped. What's left is the actual content.

Third, an LLM reads the clean content against your schema and extracts the matching fields. Because the content is already optimized, the LLM focuses on actual information instead of wading through noise. This makes the extraction more accurate and costs fewer tokens.

A schema can ask the model to normalize prices, ratings, and counts into separate fields. Validate ambiguous values against the source: “Starting from $29/mo” should retain its monthly billing period and starting-price qualifier, and a review count should not be confused with a rating.

You can also just ask in plain English

Not everything fits neatly into a JSON schema. Sometimes you don't know the exact structure upfront. For those cases, the extract endpoint accepts a prompt parameter alongside or instead of a schema.

curl -X POST https://api.webclaw.io/v1/extract \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://company.com/about",
    "prompt": "Find the founding year, number of employees, and what the company does in one sentence"
  }'

You get back structured data without having to define every field type. Useful for exploration, quick lookups, and cases where the data shape varies across pages.

For production pipelines where you need consistent output, use the schema. For ad-hoc extraction and agent workflows, the prompt is faster to set up.

What you can extract

The same endpoint works across completely different types of pages. Same approach, different schema.

Product pages. Name, price, availability, specs, reviews. Works on any e-commerce site regardless of their frontend framework or layout.

Job listings. Title, company, location, salary range, requirements, remote status. Same schema works on LinkedIn, Greenhouse, Lever, Indeed. No per-site configuration.

Contact pages. Email, phone, address, social media links. Useful for lead enrichment at scale.

Event listings. Conference pages, meetup groups, concert venues. Pull dates, locations, speakers, prices into a consistent format regardless of how each site presents the information.

Pricing pages. Plan names, features, prices, billing frequency. Competitive analysis without manually checking each competitor's site every week.

The pattern is always the same. Define the data shape. Point it at a URL. Get clean JSON back.

Extract vs scrape: when to use which

Use scrape when you want the content of a page. Articles, documentation, blog posts. You want to read it, summarize it, feed it to a RAG pipeline, or show it to a user. The output is text.

Use extract when you want specific data points. Prices, names, dates, structured fields. You want to store it in a database, compare it across sites, or use it in a calculation. The output is JSON.

You can combine them. Scrape a page to get the full content for context, then extract specific fields for your database. Or run extract on a batch of URLs to build a structured dataset from pages that all look completely different.

Using extract through MCP

If you're using webclaw through MCP with an AI agent, the agent gets the extract tool automatically. During a conversation, the agent can call extract with a schema and get back structured data without you writing any code.

You say: "Compare the pricing of these three SaaS tools." The agent calls extract on each pricing page with the same schema, gets back consistent JSON, and builds a comparison table. Three pages, three seconds, structured output.

This is where extract gets really useful. The agent decides what to extract based on the conversation. You describe the outcome, the agent figures out the schema and the URLs.

{
  "mcpServers": {
    "webclaw": {
      "command": "npx",
      "args": ["-y", "@webclaw/mcp"]
    }
  }
}

The extract tool is one of 12 tools in webclaw-mcp. It works alongside scrape, crawl, search, map, summarize, diff, and brand. Install once, your agent gets all of them.

Accuracy and trade-offs

The extraction uses an LLM for the parsing step, which means it handles messy, inconsistent pages well. But it also means there are trade-offs worth knowing.

Accuracy. Check extracted fields against the source, especially prices, dates, and inferred categories. Schema validity does not establish factual accuracy, and missing or ambiguous content can produce incomplete results.

Cost. A successful hosted extract and a standard scrape each consume one credit under the current page-operation model. See the pricing reference for endpoint costs and plan allowances.

Missing data. Allow null values where your schema permits missing information. An LLM can still infer an unsupported value, so validate source evidence before using the output.

Timing. Model processing adds latency. Measure representative pages and schemas, use explicit timeouts, and keep concurrent extraction requests within your plan limits. The batch scrape endpoint does not accept structured extraction requests.

Getting started

The extract endpoint is available on the webclaw cloud API. If you have an API key, you can start using it right now.

curl -X POST https://api.webclaw.io/v1/extract \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "schema": {
      "type": "object",
      "properties": {
        "plans": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "name": {"type": "string"},
              "price": {"type": "number"},
              "billing": {"type": "string"},
              "features": {"type": "array", "items": {"type": "string"}}
            }
          }
        }
      }
    }
  }'

webclaw SDKs for Python, TypeScript, and Go are coming soon with native extract() methods. For now, the REST API and MCP cover everything.

Define the data you need. Point it at any page. Get structured JSON back. No selectors to maintain, no parsers to update, no pipelines to fix when a site changes its CSS.

Check the API documentation for the full schema reference and response format. Sign up at webclaw.io to get your API key.

Read next: Build a RAG pipeline with live web data | Web scraping for AI agents | API reference