POST /v1/crawl + /v1/diff
Automated competitive intelligence
Track what your competitors ship, publish, and price.
Monitor competitor websites, blogs, pricing pages, changelogs, and documentation for meaningful changes. webclaw extracts structured data, detects diffs over time, and can summarize changes with built-in LLM analysis.
The problem
Keeping up with competitors means manually checking their sites, Twitter, blog, changelog, and pricing page on a schedule. It is repetitive, easy to miss, and impossible to scale across multiple competitors.
The webclaw solution
webclaw crawls specified sections of competitor sites, snapshots content, diffs new vs previous scrapes, and summarizes changes with LLM analysis. Feed the output into a dashboard or Slack channel for your team to review.
Why webclaw for competitive intelligence
- Crawl and diff competitor sites on a schedule
- LLM summarization of meaningful changes
- Brand extraction for design and positioning tracking
- Works on bot-protected competitor sites
- Structured output for feeding dashboards
Code example
TypeScript — monitor competitor blog
import { Webclaw } from "@webclaw/sdk";
const wc = new Webclaw({ apiKey: "wc_..." });
// Crawl competitor blog
const crawl = await wc.crawl({
url: "https://competitor.com/blog",
maxDepth: 2,
maxPages: 50,
});
// Wait for completion
const result = await wc.crawl.wait(crawl.id);
// Diff against yesterday's snapshot
const diff = await wc.diff({
current: result.pages,
previous: await loadYesterdaySnapshot(),
});
// Summarize meaningful changes
const summary = await wc.summarize({
content: diff.changedContent,
focus: "product announcements and pricing changes",
});
console.log(summary);webclaw features for this use case
- BFS crawler with depth and concurrency controls
- Content diff across snapshots
- LLM summarization of changes
- Brand identity extraction
- Bot protection bypass
Frequently asked questions
Can I crawl a competitor's entire blog automatically?
Yes. Use /v1/crawl with the blog URL and a maxDepth of 2-3. webclaw will BFS the same-origin pages, extract content, and return structured results. Combine with /v1/diff to only surface new or changed posts.
How does brand extraction help with competitive intelligence?
The /v1/brand endpoint extracts logo, colors, fonts, and design tokens from any site. Track changes over time to detect rebrands, design refreshes, or positioning shifts across competitors.
Is it legal to scrape competitor sites for competitive intelligence?
Scraping public, non-gated content is generally legal in most jurisdictions (see HiQ v LinkedIn). webclaw respects robots.txt by default and you should comply with each site's terms of service. Consult your legal team for specifics.