SDK integration
Web scraping for CrewAI agents
Give your CrewAI agents real-time web access as a tool.
CrewAI is a Python framework for building multi-agent systems where specialized agents collaborate on tasks. webclaw plugs in as a crew tool that any agent can call, providing scrape, search, crawl, and extract capabilities with automatic bot bypass.
Setup
CrewAI Python — custom tool
python
from crewai import Agent, Task, Crew
from crewai.tools import tool
from webclaw import Webclaw
wc = Webclaw(api_key="wc_...")
@tool("Web Scraper")
def scrape_url(url: str) -> str:
"""Scrape any URL and return clean markdown."""
result = wc.scrape(url=url, formats=["markdown"])
return result.markdown
researcher = Agent(
role="Research Analyst",
goal="Find and summarize information from the web",
tools=[scrape_url],
)
task = Task(
description="Research the latest Rust async runtime benchmarks",
agent=researcher,
)
crew = Crew(agents=[researcher], tasks=[task])
crew.kickoff()Why webclaw for CrewAI
- Drop-in @tool wrapping of webclaw SDK
- Bot bypass lets agents access any public site
- Structured output agents can reason over
- Fast enough for interactive crew execution
Common use cases
- Multi-agent research crews
- Sales intelligence agents
- Content aggregation and summarization
- Automated competitive analysis
Wrap the webclaw SDK call in a @tool-decorated function and pass it to your agent's tools list. The agent can then call scrape, crawl, or extract as needed during task execution.