Crawl and diff competitor sites on a schedule
POST /v1/crawl + /v1/diff
Automated competitive intelligence.
Track what your competitors ship, publish, and price.
Monitor competitor websites, blogs, pricing pages, changelogs, and documentation for meaningful changes. webclaw extracts structured data, detects diffs over time, and can summarize changes with built-in LLM analysis.
How it works
Build it step by step.
The real flow, one step at a time. Switch between TypeScript, Python, and cURL on any snippet.
- 1
Map the target site
Run /v1/map on a competitor domain to discover its blog, pricing, changelog, and docs URLs.
const competitor = "https://competitor.com";// Discover the blog, pricing, changelog, and docs URLsconst { links } = await webclaw.map({ url: competitor });const targets = links.filter((u) => /\/(blog|pricing|changelog|docs)/.test(u),); - 2
Scrape key sections
Scrape the pages that matter into clean markdown so changes are easy to track.
// Scrape each target page into clean markdownconst pages = await Promise.all( targets.map((url) => webclaw.scrape({ url, formats: ["markdown"] }), ),); - 3
Diff over time
Compare each scrape against the previous snapshot to surface only new or changed content.
// Compare each page against yesterday's saved snapshotconst diffs = await Promise.all( pages.map(async (page) => { const previous = await loadSnapshot(page.url); return webclaw.diff({ url: page.url, previous, current: page.markdown, }); }),); - 4
Summarize the deltas
Send changed content through /v1/summarize to get a readable digest of product and pricing moves.
// Digest only the pages that actually changedconst changed = diffs.filter((d) => d.hasChanges);for (const d of changed) { const summary = await webclaw.summarize({ url: d.url, focus: "product announcements and pricing changes", }); console.log(d.url, summary);}
Why webclaw
Built for competitive intelligence.
LLM summarization of meaningful changes
Brand extraction for design and positioning tracking
Works on bot-protected competitor sites
Structured output for feeding dashboards
What you get
Everything this use case needs.
- BFS crawler with depth and concurrency controls
- Content diff across snapshots
- LLM summarization of changes
- Brand identity extraction
- Bot protection bypass
Where it fits
Built for the messy parts.
Keeping up with competitors means manually checking their sites, Twitter, blog, changelog, and pricing page on a schedule. It is repetitive, easy to miss, and impossible to scale across multiple competitors.
webclaw crawls specified sections of competitor sites, snapshots content, diffs new vs previous scrapes, and summarizes changes with LLM analysis. Feed the output into a dashboard or Slack channel for your team to review.
Common questions
Frequently asked questions
Can I crawl a competitor's entire blog automatically?
Yes. Use /v1/crawl with the blog URL and a maxDepth of 2-3. webclaw will BFS the same-origin pages, extract content, and return structured results. Combine with /v1/diff to only surface new or changed posts.
How does brand extraction help with competitive intelligence?
The /v1/brand endpoint extracts logo, colors, fonts, and design tokens from any site. Track changes over time to detect rebrands, design refreshes, or positioning shifts across competitors.
Is it legal to scrape competitor sites for competitive intelligence?
Scraping public, non-gated content is generally legal in most jurisdictions (see HiQ v LinkedIn). webclaw respects robots.txt by default and you should comply with each site's terms of service. Consult your legal team for specifics.
For AI agents
Or hand it to your agent.
Add the webclaw MCP server to Claude, Cursor, or any MCP client, then paste this prompt. The agent calls the webclaw tools and hands the result back to your model — no code to write.
PROMPT FOR YOUR AGENT
Using the webclaw tools, run a competitive intelligence check on [the competitor domain]. First call map to discover the site's key URLs, then scrape (or crawl for a whole section) the pages that matter most — homepage, pricing, changelog, blog, and docs — into clean markdown. For each page, use diff to compare today's content against [the previous snapshot or last run] and isolate only what actually changed, then pass each meaningful change through summarize to turn it into a readable digest of product launches, pricing moves, and positioning shifts. Optionally call brand on the domain to flag any logo, color, or font updates. Return a single report: a table with columns Page | What Changed | Why It Matters, followed by a short bulleted summary of the most significant moves and an alert line at the top if anything pricing-related changed.
Ready to build? Start extracting.
Cancel anytime. Clean, structured data on every call.