SDK INTEGRATION
Web scraping for CrewAI agents
Give your CrewAI agents real-time web access as a tool.
CrewAI is a Python framework for building multi-agent systems where specialized agents collaborate on tasks. webclaw plugs in as a crew tool that any agent can call, providing scrape, search, crawl, and extract capabilities with automatic bot bypass.
Setup
CrewAI Python — custom tool
from crewai import Agent, Task, Crew
from crewai.tools import tool
from webclaw import Webclaw
wc = Webclaw(api_key="wc_...")
@tool("Web Scraper")
def scrape_url(url: str) -> str:
"""Scrape any URL and return clean markdown."""
result = wc.scrape(url=url, formats=["markdown"])
return result.markdown
researcher = Agent(
role="Research Analyst",
goal="Find and summarize information from the web",
tools=[scrape_url],
)
task = Task(
description="Research the latest Rust async runtime benchmarks",
agent=researcher,
)
crew = Crew(agents=[researcher], tasks=[task])
crew.kickoff()Why webclaw for CrewAI
- Drop-in @tool wrapping of webclaw SDK
- Bot bypass lets agents access any public site
- Structured output agents can reason over
- Fast enough for interactive crew execution
Common use cases
- Multi-agent research crews
- Sales intelligence agents
- Content aggregation and summarization
- Automated competitive analysis
Frequently asked questions
How do I add webclaw as a tool in a CrewAI crew?
Wrap the webclaw SDK call in a @tool-decorated function and pass it to your agent's tools list. The agent can then call scrape, crawl, or extract as needed during task execution.
Can multiple agents share the same webclaw tool?
Yes. Define the tool once and include it in each agent's tools list. All agents will share the same API key and rate limit pool.
Ready to connect?
Start extracting.
7-day Starter trial. Cancel anytime. Deploy in under a minute or self-host forever. Open source.
OTHER INTEGRATIONS