REST API
Web scraping for Dify AI applications
Add web tools to Dify agents and workflows.
Dify is an open-source LLM application development platform. Add webclaw as a custom tool via the Tools > Custom section, and your Dify agents can scrape, extract, and research the web at runtime.
Setup
Dify custom tool (OpenAPI-style)
openapi: 3.0.0
info:
title: webclaw
description: Web extraction API for LLM agents
version: 1.0.0
servers:
- url: https://api.webclaw.io
paths:
/v1/scrape:
post:
summary: Scrape any URL to markdown
operationId: scrape
security:
- bearerAuth: []
requestBody:
content:
application/json:
schema:
type: object
required: [url]
properties:
url:
type: string
formats:
type: array
items:
type: string
responses:
'200':
description: Success
components:
securitySchemes:
bearerAuth:
type: http
scheme: bearerWhy webclaw for Dify
- Registers as a standard OpenAPI custom tool
- Works with Dify agents and workflows
- Bot bypass for production Dify apps
- LLM-optimized markdown works with Dify models
Common use cases
- Dify agents with real-time web access
- Chatbots that answer from live documentation
- Knowledge workflows with fresh web data
- RAG pipelines built in Dify
Frequently asked questions
How do I register webclaw as a Dify custom tool?
Go to Tools > Custom in your Dify workspace. Paste the OpenAPI spec above and configure your webclaw API key as the bearer token. The tool will be available to all agents in the workspace.
Can Dify workflow steps call webclaw?
Yes. Once registered as a custom tool, webclaw endpoints appear in workflow nodes as callable actions. You can scrape, extract, or search as part of any Dify flow.
Ready to connect?
Start extracting.
7-day Starter trial. Cancel anytime. Deploy in under a minute or self-host forever. Open source.
OTHER INTEGRATIONS