Everything we shipped, in order.

Added

• Local JavaScript rendering. Built with the `browser` feature (`cargo build --release --features browser`), webclaw renders pages in your local Chrome when the HTTP response has no content yet.
• Page actions. `--actions` (JSON inline or `@FILE`) runs ordered steps in the rendered page before webclaw reads it: `click`, `type`, `press`, `scroll`, `wait`, `waitForSelector` and `executeJavascript`, whose values come back as `js_results…
• Screenshots, PDFs and MHTML. `--screenshot FILE` (PNG or JPEG, `--full-page`), `--save-pdf FILE` (`--paper`, `--landscape`, `--print-background`, `--scale`) and `--save-mhtml FILE`.
• Network capture. `--capture-network[=PATTERN]` returns the XHR and fetch responses a rendered page loads, so you get the JSON an app reads from its API.
• Persistent browser profile. `--browser-profile DIR` (MCP: `WEBCLAW_BROWSER_PROFILE`) keeps cookies, local storage and IndexedDB between runs, so a login made once holds. A profile in use by another browser is refused with the owner named.
• Cloudflare challenge pages in the local browser. `--solve` (MCP: `solve` on `scrape`) loads pages that first show a Cloudflare challenge. The README covers how it works and its limits.
• Resource blocking while rendering. `--block images,fonts,media` (MCP: `block` on `scrape` and `crawl`).
• Local renders use your Chrome's own User-Agent.
• CSS selectors and XPath. `--select` and `--xpath` pick values from a page, and `--field NAME=SELECTOR` builds one JSON record per match. The new MCP `select` tool does the same. Both run without an LLM.
• Data tables as JSON. `--tables` returns each data table as `{caption, headers, rows}`, with spanning cells laid out like a browser does and layout tables skipped. MCP: `tables` on `select`.
• Find by text, and find similar. `--find-text`, `--find-regex` and `--similar`: give one example element and get every element like it, plus a CSS selector that picks them all. MCP: `text`, `regex` and `similar` on `select`.
• Keep only what answers a question. `--query "..."` returns the paragraphs that answer it under their headings instead of the whole page. MCP: `query` on `scrape`.
• Question-guided crawls. `--crawl --query "..."` fetches the links most likely to answer first and lists pages best match first, each with a `relevance`. `--stop-when-answered` ends the crawl once the answer looks found.
• Chunks for retrieval. `--chunk TOKENS` and `--chunk-overlap TOKENS` split pages into chunks that keep sentences, tables and code blocks whole, each with its heading path and a token estimate.
• Markdown with numbered link citations. `--citations` moves inline links into a numbered list at the end of the page. MCP: `citations` on `scrape`, `crawl` and `batch`.
• WARC archives. `--warc FILE` records the HTTP responses of one page, several, or a whole crawl in the format web archives use.
• Opt-in disk cache. `--cache` stores fetched pages in `WEBCLAW_CACHE_DIR` or `~/.cache/webclaw`, so a rerun answers from disk.
• MCP over HTTP. `webclaw-mcp --http [ADDR]` serves the tools to several agents or another machine. It requires a bearer token from `WEBCLAW_MCP_TOKEN` unless started with `--no-auth`.
• Every browser profile by name. `--browser` (MCP: `browser` on `scrape`) accepts `chrome`, `chrome-macos`, `firefox`, `safari`, `safari-ios`, `edge` and `random`.
• More sources and filters for `--map`. `--map-source sitemap,cc` adds URLs from the public Common Crawl index, and `--path-prefix`, `--include-paths`, `--exclude-paths` and `--query` now apply to `--map`.
• Crash-safe crawls. `-f jsonl` streams one JSON object per page as it finishes, and `--crawl-state` checkpoints about once a second, so a killed crawl resumes where it stopped and keeps its page budget.
• A Python package that bundles the CLI in `python/`, built with maturin. It is not published yet.
• Reproducible benchmarks. Each script in `benchmarks/` reruns one comparison against open-source tools with a single command.

Changed

• Package metadata names the licence `AGPL-3.0-only`, the current SPDX identifier. `AGPL-3.0` was retired and meant the same; the licence itself is unchanged.
• Crawls respect robots.txt by default. Disallowed URLs are skipped, redirects are checked again, and `Crawl-delay` spaces requests to its host. Opt out with `--ignore-robots` (MCP and the OSS server: `ignore_robots`).
• `--delay` spaces requests per host. It is the shortest time between two requests to the same host, shared by all crawl workers. Before, each worker slept on its own, so `--concurrency 4 --delay 500` still sent about six requests a second.
• The crawler reaches more pages. It runs a pool of workers instead of one depth level at a time, and follows links from the whole page, navigation included, not only from the extracted content.
• Text a browser never shows is dropped. `<template>`, the `hidden` attribute and inline `display:none` or `visibility:hidden` no longer reach the output, which keeps hidden prompt-injection text away from agents.
• Every scrape retries the same way. A 429 or 5xx response gets a second attempt that honors `Retry-After` when it fits the time budget, and the CLI prints a warning when it retries.
• `--raw-html` exits 1 on HTTP errors. It still prints the body.
• The MCP server refuses an unknown browser name instead of using Chrome, and lists the valid names.
• `--screenshot` and `--save-pdf` refuse output flags they would ignore, such as `-f`, `--query` and `--output-dir`.
• The MCP server has 14 tools with the new `select` and `screenshot`. The `webclaw-mcp` release binary grows from 23.9 MB to 26.6 MB.

Fixed

• The desktop Safari profile works again. It refused every HTTP/2 response, and `random` picked it one time in five.
• SOCKS proxies are refused with a clear message. They were advertised but never connected.
• Crawls keep pages whose links redirect to the same page with a trailing slash added or the scheme changed. They were dropped while robots.txt was respected.
• The MCP `map` tool finds URLs on sites without a sitemap, with a short crawl as the CLI does. It returned 0 URLs.
• The MCP `crawl` tool honors `WEBCLAW_PROXY`.
• Pages nested tens of thousands of elements deep no longer crash extraction.

Fixed

• Cloud fallback keeps the data later steps need. When a scrape runs through the cloud API and the command also asks for a diff, a summary or an extraction, the cloud request now asks for the full structured result instead of the display form…
• The cloud API base can be pointed elsewhere. `WEBCLAW_API_URL` now applies when a key is supplied directly, not only when the client is built with an explicit base, so the CLI and the MCP server can run against a different endpoint.

Added

• Content in `<noscript>` fallbacks is recovered. When the selected region has almost no visible text, readable text from a `<noscript>` block inside it is used instead. Exclude selectors still apply, and the text appears once.

Changed

• Unusable pages fail with a reason. When a target answers with an HTTP error status, returns no readable text, or serves a sign-in wall, an access check, or an error screen, extraction returns an error that names the cause instead of passing…
• Selectors and main-content mode are final. A short selected region no longer widens to the whole page, and text recovered from embedded page data, script state, or the Reddit and YouTube shortcuts no longer adds content from outside your se…
• Windows installation is documented and tested. The README walks through the prebuilt ZIP and a source build, and CI installs both on a hosted Windows runner and runs a smoke test.

Removed

• The MCP `lead` and `lead_batch` tools. Lead enrichment is retired, so the MCP server no longer advertises these tools or their input schemas, and `create-webclaw` no longer lists them.

Fixed

• Excluding one element keeps the page's hero. An exclude selector aimed elsewhere no longer drops the recovered hero title and tagline.
• Cookie banners, modals, and overlays stay out of recovered headings. A class name that only contains "modal", such as `free-modal-container`, no longer hides a real hero section.
• Translation tables and privacy settings embedded in scripts are not treated as page text.
• Plain text matches the markdown. The plain-text format is derived from the final content, so it includes the same recovered sections and leaves out the same excluded ones.
• JSON-LD with some non-ASCII characters parses whole. Structured data containing characters whose lowercase form has a different byte length, such as `İ` or `ẞ`, is no longer cut at the wrong place.
• The CLI cloud fallback prints the format you asked for. `--format html` requests raw HTML from the cloud API, and text, LLM, and HTML output read the matching fields of the cloud response.

Added

• An additional optional hosted model provider. Extraction and summarization can use another OpenAI-compatible routing service when it is configured, without changing provider priority for existing users.
• Cleaner managed-service model discovery. Hosted deployments can now avoid probing local inference services that are not part of their configuration. Local CLI and self-hosted behavior is unchanged.

Fixed

• A request the MCP server understands but cannot serve yet now says so accurately. A request arriving before the connection is set up was refused with "method not found", even when the server implements that method and the request was simply…
• Encoded characters survive document and sitemap parsing. DOCX text, metadata, sitemap URLs, and attributes no longer lose decoded entity references.
• Valid PDFs with a late header are accepted. Files whose PDF marker appears near the end of the permitted prefix are no longer rejected at the boundary.

Fixed

• A single unreadable line no longer ends an MCP session. One line the server could not parse, including a blank line, ended the connection and was reported as if the client had hung up — even when a perfectly valid request followed on the ne…

Changed

• The MCP server now speaks the current protocol revision (2026-07-28) with clients that ask for it. Clients that negotiate an older revision are unaffected and continue to work exactly as before.

Fixed

• The MCP server no longer exits when a client speaks before the handshake. A request sent ahead of `initialize` ended the process without a reply, so the client saw a closed pipe instead of an error.

Added

• Crawl without a page limit. `CrawlConfig.max_pages` is now `Option<usize>`; `None` crawls until the site is exhausted or you cancel. Expose it as `--max-pages 0` on the CLI, `max_pages: 0` on the MCP `crawl` tool, and `max_pages: 0` on the…
• Streaming batch. `POST /v1/batch` accepts `"stream": true` and answers with NDJSON — one result per line, written as each URL finishes — so a batch of ten and a batch of ten thousand cost the same in memory.
• Crawl streaming mode. `CrawlConfig.stream_only` drops each page after reporting it on `progress_tx` instead of collecting it, so peak memory stays flat regardless of page count.
• Chronopost parcel tracking extractor. `webclaw vertical chronopost <tracking-url>` — also available as the `vertical_scrape` MCP tool and `POST /v1/scrape/chronopost` — returns the current status, the milestone progress bar, and the full sc…

Fixed

• A fetch now honours one time budget instead of compounding several. The HTTP client applies `timeout` twice per attempt — once for the response head, then again for the body — and the retry loop ran two attempts with a pause between them, s…
• Healthy Cloudflare-fronted pages are no longer mistaken for challenge pages. Cloudflare adds a small JavaScript-detection snippet to ordinary, successfully served pages.

Changed

• Dropped a competitor's name from the project description. The MCP registry manifest was reworded in 0.6.17 (already live); the Chinese README still carried the same phrasing and is now aligned with the English one.

Added

• `@webclaw/mcp` — zero-install MCP launcher. `npx -y @webclaw/mcp` downloads the prebuilt `webclaw-mcp` binary once, verifies it against the release `SHA256SUMS`, caches it, and runs it as an MCP stdio server.
• Atlas Cloud provider in the LLM chain (opt-in). Set `ATLASCLOUD_API_KEY` to add [Atlas Cloud](https://atlascloud.ai) as a provider for the LLM features (extraction, summarization).

Fixed

• `--urls-file` with a single URL now works. A file containing exactly one URL (with no positional URL on the command line) was rejected with "no input provided": it fell between the batch path, which skips single-entry files, and the single-…

Added

• `lead` and `lead_batch` MCP tools. Turn a company URL into an outreach-ready lead — founders and leadership with their LinkedIn and X, plus company summary, socials, tech stack, pricing, and public emails — straight from any MCP client (Cla…

Fixed

• `webclaw --help` no longer prints secret environment values. Secret env vars are masked in help output instead of being shown in the clear.

Added

• Static musl Linux binaries (`x86_64-unknown-linux-musl`, `aarch64-unknown-linux-musl`) that run on any Linux regardless of glibc — Alpine, Amazon Linux 2023, RHEL 9, and older distros.

Changed

• gnu Linux binaries now run on older glibc. They are built on Ubuntu 22.04 (glibc 2.35) instead of the latest runner (glibc 2.39), so they start on Debian 12, Ubuntu 22.04, and other LTS distros that previously failed with `version 'GLIBC_2.…

Fixed

• `create-webclaw` install (`create-webclaw@0.1.5`) failed on every platform: it looked up the wrong release-asset name and used `unzip` (absent on Windows), silently falling back to `cargo install`.
• Self-hosting setup scripts wrote `WEBCLAW_AUTH_KEY` into the generated `.env`, but the server reads `WEBCLAW_API_KEY` — on a public bind this made the server refuse to start.

Added

• Standalone web search using your own [Serper.dev](https://serper.dev) key — no hosted webclaw account needed. Available across the CLI (`webclaw search "query" --num 5 --scrape`, key via `--serper-key` or `SERPER_API_KEY`), the MCP `search`…
• Layered URL discovery for `--map`: when a site has no sitemap or only a thin one, map now falls back to a bounded same-origin crawl and harvests links from every fetched page plus the unfetched frontier, returning far more URLs.

Fixed

• MCP tools now accept boolean arguments whether the client sends them as JSON booleans or as the strings `"true"`/`"false"` (case-insensitive). Some MCP clients (e.g.

Added

• New Google Gemini provider in the LLM provider chain. Set `GEMINI_API_KEY` (and optionally `GEMINI_MODEL`, default `gemini-2.5-flash`) to enable it; the chain tries Ollama → OpenAI → Gemini → Anthropic and uses the first available provider.

Fixed

• The Anthropic provider's default model pointed at a retired model id that now returns `404`, which could fail extraction/summarization when falling back to Anthropic.

Fixed

• MCP tools that take numeric arguments now accept those values whether the client sends them as numbers or as numeric strings. Some MCP clients (e.g.

Fixed

• The multi-arch Docker image (linux/amd64 + linux/arm64) now publishes reliably on each release. The build moved to Buildx so registry pushes no longer fail intermittently, and the Homebrew formula update that depends on it is no longer skip…

Fixed

• Pages with multibyte text (accented or CJK characters) no longer panic or get mangled during extraction. API-endpoint discovery now cuts oversized scripts on a character boundary instead of crashing mid-character, and structured-data parsin…
• LLM error messages from a provider no longer panic when the error body contains multibyte characters near the truncation point.
• LLM provider requests now have explicit connect and overall timeouts, so a stalled or unreachable provider fails fast instead of hanging.
• Batch extraction in the MCP server no longer aborts the whole batch when a single URL fails to resolve; bad URLs are reported as individual per-URL errors and the rest still run.
• CLI crawl and batch runs now wait for the completion webhook to actually send before exiting, replacing a fixed delay that could cut the request off or waste time.
• Homepage warm-up requests now include the port for hosts on a non-default port, so those sites are warmed correctly.

Changed

• Updated the HTTP/TLS engine (wreq 6.0.0-rc.29, wreq-util 3.0.0-rc.12). This pulls in upstream robustness fixes: no more panic on responses with non-UTF8 header values, a fix for short reads when decoding large compressed bodies, and the TCP…

Added

• Slow fetches now print a progress line to stderr every 10 seconds (`# webclaw: still fetching <url> (Ns)`) so a long request no longer looks like the CLI hung. Fast fetches stay silent and stdout is untouched.
• New `--url-encoded` flag plus a warning when a URL looks like the shell split it on `&` or `?`. The warning suggests quoting the URL; pass `--url-encoded` to silence it when the URL is intentional.

Changed

• Reddit threads extract reliably again. The old anonymous JSON endpoint is no longer available, so webclaw now reads old.reddit.com directly without an API key or JavaScript.

Added

• API surface discovery: a new module extracts the API endpoints embedded in a page's inline scripts and linked JavaScript bundles. It surfaces relative REST paths, absolute URLs, GraphQL operations, and WebSocket endpoints that a sitemap alo…

Fixed

• Hardened resource and path-safety limits across the CLI, MCP server, and self-hosted API: oversized or highly compressed responses are capped while streaming, deeply nested page data can no longer exhaust memory, output filenames stay insid…

Fixed

• Cleaned up `--format llm` output on noisy news and documentation pages. Comment-count links, bare page-number paragraphs, pagination leftovers such as `0 Next`, and duplicated JSON-LD article bodies are now removed before they reach the LLM…
• The CLI now recognizes common cookie-consent redirects and prints a clearer warning when a page returns a consent wall instead of usable content.
• The CLI keeps noisy parser warnings from real-world malformed HTML out of stderr by default. `WEBCLAW_LOG` still lets advanced users opt into deeper parser logs.

Fixed

• Hardened URL safety across the CLI, MCP server, and self-hosted API paths so local and private network targets are rejected more consistently, including after DNS resolution and redirects.
• Added a timeout around inline JavaScript data extraction so hostile pages cannot keep the extractor busy forever.
• Tightened Amazon and eBay URL recognition so deceptive hosts are rejected while common international marketplaces still work.
• Avoided unnecessary decoding work on large responses during bot-challenge detection.
• Reduced release workflow token permissions so build jobs run with narrower GitHub access.

Fixed

• Improved `--format llm` output quality on modern news and documentation pages. Framework hydration blobs and low-value page chrome structured-data records are now filtered out before they can flood the LLM context, while content-bearing Sch…
• Fixed element-to-text spacing so adjacent inline nodes no longer smash words together, while punctuation stays attached on real pages such as docs, forums, and reference sites.
• Removed common screen-reader-only link chrome such as "opens new tab" from LLM body text and link labels without stripping ordinary prose that happens to mention external links.

Fixed

• LLM providers now support `ANTHROPIC_BASE_URL` for Anthropic-compatible proxies, plus an `OPENAI_RESPONSE_FORMAT_TYPE` override for OpenAI-compatible backends such as LM Studio. Thanks to Toti (`@Toti330`) for the report.

Added

• GitHub Releases now include a Windows x86_64 `.zip` with `webclaw.exe`, `webclaw-mcp.exe`, and `webclaw-server.exe`. Thanks to Suryansh Mishra (`@notrealsuryansh`) for the contribution.

Fixed

• Improved brand extraction results for modern sites with large app shells. Brand colors, fonts, and logos are now less likely to be polluted by login widgets, customer-logo grids, icon fonts, or generated CSS noise.

Added

• `FetchClient::fetch_smart(url)` applies per-site rescue logic and returns the same `FetchResult` shape as `fetch()`. Reddit URLs route to the `.json` API with an identifiable bot `User-Agent`, and Akamai-style challenge pages trigger a home…

Fixed

• Regression introduced in 0.5.4 where the production server's `/v1/scrape` bypassed the Reddit `.json` shortcut and Akamai cookie warmup that `fetch_and_extract` had been providing.
• Panic in the markdown converter (`markdown.rs:925`) on single-pipe `|` lines. A `[1..len-1]` slice on a 1-char input triggered `begin <= end`. Guarded.

Added

• `webclaw --browser safari-ios` on the CLI. Pairs with `--proxy` for DataDome-fronted sites that reject desktop profiles.

Added

• New `BrowserProfile::SafariIos` for Safari iOS 26 fingerprinting. Pairs with a country-matched residential proxy for sites that reject non-mobile profiles.
• `accept_language_for_url(url)` and `accept_language_for_tld(tld)` helpers. Returns a locale-appropriate `Accept-Language` based on the URL's TLD, with `en-US` as the fallback.

Changed

• Chrome browser fingerprint refreshed for current Cloudflare bot management. Fixes 403 challenges on several e-commerce and jobs sites.
• Bumped `wreq-util` to `3.0.0-rc.10`.

Added

• `webclaw vertical <name> <url>` subcommand on the CLI. Runs a specific vertical extractor and prints typed JSON (pretty-printed by default, `--raw` for single-line).
• `webclaw extractors` subcommand on the CLI. Lists all 28 vertical extractors with name, label, and one URL pattern sample. `--json` emits the full catalog as JSON (same shape as `GET /v1/extractors`) for tooling.
• `vertical_scrape` and `list_extractors` tools on `webclaw-mcp`. Claude Desktop / Claude Code users can now call any of the 28 extractors by name from an MCP session. Tool count goes from 10 to 12.

Changed

• Server-info instruction string in `webclaw-mcp` now lists all 12 tools (previously hard-coded 10). Also `webclaw --help` on the CLI now shows the three subcommands: `bench`, `extractors`, `vertical`.

Added

• `webclaw_fetch::Fetcher` trait. Vertical extractors now consume `&dyn Fetcher` instead of `&FetchClient` directly. The trait exposes three methods (`fetch`, `fetch_with_headers`, `cloud`) covering everything extractors need.

Changed

• All 28 extractor `extract()` signatures migrated from `client: &FetchClient` to `client: &dyn Fetcher`. The dispatcher functions (`extractors::dispatch_by_url`, `extractors::dispatch_by_name`) and the cloud escalation helpers (`cloud::smart…

Added

• 28 vertical extractors that return typed JSON instead of generic markdown. New `webclaw_fetch::extractors` module with one extractor per site. Dev: reddit, hackernews, github_repo / github_pr / github_issue / github_release, crates_io, pypi…
• `POST /v1/scrape/{vertical}` on `webclaw-server` for explicit vertical routing. Picks the parser by name, validates the URL plausibly belongs to that vertical, returns the same shape as `POST /v1/scrape` but typed.
• `GET /v1/extractors` on `webclaw-server`. Returns the full catalog as `{"extractors": [{"name": "...", "label": "...", "description": "...", "url_patterns": [...]}, ...]}` so clients can build tooling / autocomplete / user-facing docs off a…
• Antibot cloud-escalation for 5 ecommerce + reviews verticals. Amazon, eBay, Etsy, Trustpilot, and Substack (as HTML fallback) go through `cloud::smart_fetch_html`: try local fetch first; on bot-protection detection (Cloudflare challenge, Da…
• `cloud::synthesize_html` for cloud-bypassed extraction. `api.webclaw.io/v1/scrape` deliberately does not return raw HTML; it returns a parsed bundle (`structured_data` JSON-LD blocks + `metadata` OG/meta tags + `markdown`).
• Trustpilot 2025 schema parser. Trustpilot replaced their single-Organization + aggregateRating shape with three separate JSON-LD blocks: a site-level Organization (Trustpilot itself), a Dataset with a csvw:Table `mainEntity` carrying the pe…
• OG-tag fallback in `ecommerce_product` for sites with no JSON-LD and sites with JSON-LD but empty offers. Three paths now: `jsonld` (Schema.org Product with offers), `jsonld+og` (Product JSON-LD plus OG product tags filling in missing price…
• URL-slug title fallback in `etsy_listing` for delisted / blocked pages. When Etsy serves a placeholder page (`"etsy.com"`, `"Etsy - Your place to buy..."`, `"This item is unavailable"`), humanise the URL slug (`/listing/123/personalized-sta…
• Force-cloud-escalation in `amazon_product` when local HTML lacks Product JSON-LD. Amazon A/B-tests JSON-LD presence. When local fetch succeeds but has no `Product` block and a cloud client is configured, the extractor force-escalates to the…
• AWS WAF "Verifying your connection" detector in `cloud::is_bot_protected`. Trustpilot serves a `~565` byte interstitial with an `interstitial-spinner` CSS class.

Changed

• `webclaw-fetch::FetchClient` gained an optional `cloud` field via `with_cloud(CloudClient)`. Extractors reach it through `client.cloud()` to decide whether to escalate.
• Consolidated `CloudClient` into `webclaw-fetch`. Previously duplicated between `webclaw-mcp/src/cloud.rs` (302 LOC) and `webclaw-cli/src/cloud.rs` (80 LOC).

Added

• `webclaw bench <url>` — per-URL extraction micro-benchmark. New subcommand. Fetches a URL once, runs the same extraction pipeline as `--format llm`, and prints a small ASCII table comparing raw-HTML tokens vs.
• `webclaw-server` — new OSS binary for self-hosting a REST API. Until now, `docs/self-hosting` promised a `webclaw-server` binary that only existed in the hosted-platform repo (closed source).
• `crawl` endpoint runs synchronously and hard-caps at 500 pages / 20 concurrency. No job queue, no background workers — a naive caller can't OOM the process. `batch` caps at 100 URLs / 20 concurrency for the same reason.

Changed

• Docker image now ships three binaries, not two. `Dockerfile` and `Dockerfile.ci` both add `webclaw-server` to `/usr/local/bin/` and `EXPOSE 3000` for documentation.

Fixed

• Dead-code warning on `cargo install webclaw-mcp`. `rmcp` 1.3.x changed how the `#[tool_handler]` macro reads the `tool_router` struct field — it now goes through a derived trait impl instead of referencing the field by name, so rustc's dead…

Fixed

• Docker image can be used as a FROM base again. v0.3.13 switched the Docker `CMD` to `ENTRYPOINT ["webclaw"]` so that `docker run IMAGE https://example.com` would pass the URL through as expected.

Fixed

• UTF-8 char boundary panic in `webclaw-core::extractor::find_content_position`. After rejecting a match that fell inside image syntax (``), the scan advanced `search_from` by a single byte.

Changed

• `webclaw-fetch::sitemap::parse_robots_txt` now does proper directive parsing. The previous `trimmed[..8].eq_ignore_ascii_case("sitemap:")` slice couldn't handle "Sitemap :" (space before colon) from bad generators, didn't strip inline `# ..…
• `webclaw-fetch::crawler::is_cancelled` uses `Ordering::Acquire` (was `Relaxed`). Technically equivalent on x86/arm64 for single-word loads, but the explicit ordering documents the synchronization intent for readers and the compiler.

Added

• `webclaw-mcp` caches the Firefox FetchClient lazily. Tool calls that repeatedly request the Firefox profile without cookies used to build a fresh reqwest pool + TLS stack per call; a single `OnceLock` keeps the client alive for the life of…

Fixed

• Batch/crawl no longer panics on semaphore close (P1). Three `permit.acquire().await.expect("semaphore closed")` call sites in `webclaw-fetch` (`client::fetch_batch`, `client::fetch_and_extract_batch_with_options`, `crawler` inner loop) now…

Fixed

• Docker CMD replaced with ENTRYPOINT: both `Dockerfile` and `Dockerfile.ci` now use `ENTRYPOINT ["webclaw"]` instead of `CMD ["webclaw"]`. CLI arguments (e.g.

Added

• Crawl scope control: new `allow_subdomains` and `allow_external_links` fields on `CrawlConfig`. By default crawls stay same-origin. Enable `allow_subdomains` to follow sibling/child subdomains (e.g.

Added

• Sitemap fallback paths: discovery now tries `/sitemap_index.xml`, `/wp-sitemap.xml`, and `/sitemap/sitemap-index.xml` in addition to the standard `/sitemap.xml`.

Changed

• Fetch timeout reduced from 30s to 12s: prevents cascading slowdowns when proxies are unresponsive. Worst-case per-URL drops from ~94s to ~25s.
• Retry attempts reduced from 3 to 2: combined with shorter timeout, total worst-case is 12s + 1s delay + 12s = 25s instead of 30s + 1s + 30s + 3s + 30s = 94s.

Fixed

• Layout tables rendered as sections: tables used for page layout (containing block elements like `<p>`, `<div>`, `<hr>`) are now rendered as standalone sections instead of pipe-delimited markdown tables.
• Stack overflow on deeply nested HTML: pages with 200+ DOM nesting levels (e.g., Express.co.uk live blogs) no longer overflow the stack. Two-layer fix: depth guard in markdown.rs falls back to iterator-based text collection at depth 200, and…
• Noise filter swallowing content in malformed HTML: `<form>` tags no longer unconditionally treated as noise — ASP.NET page-wrapping forms (>500 chars) are preserved.

Changed

• Bold/italic block passthrough: `<b>`/`<strong>`/`<em>`/`<i>` tags containing block-level children (e.g., Drudge wrapping columns in `<b>`) now act as transparent containers instead of collapsing everything into inline bold/italic.

Fixed

• MCP research token overflow: research results are now saved to `~/.webclaw/research/` and the MCP tool returns file paths + findings instead of the full report. Prevents "exceeds maximum allowed tokens" errors in Claude/Cursor.
• Research caching: same query returns cached result instantly without spending credits.
• Anthropic rate limit throttling: 60s delay between LLM calls in research to stay under Tier 1 limits (50K input tokens/min).

Added

• `dirs` dependency for `~/.webclaw/research/` path resolution.

Added

• `--research` CLI flag: run deep research via the cloud API. Prints report to stdout and saves full result (report + sources + findings) to a JSON file. Supports `--deep` for longer reports.
• MCP extract/summarize cloud fallback: when no local LLM is available, these tools now fall back to the cloud API instead of erroring. Set `WEBCLAW_API_KEY` for automatic fallback.
• MCP research structured output: the research tool now returns structured JSON (report + sources + findings + metadata) instead of raw text, so agents can reference individual findings and source URLs.

Added

• Structured data in markdown/LLM output: `__NEXT_DATA__`, SvelteKit, and JSON-LD data now appears as a `## Structured Data` section with a JSON code block at the end of `-f markdown` and `-f llm` output.

Fixed

• Homebrew CI: formula now updates all 4 platform checksums after Docker build completes, preventing SHA mismatch on Linux installs.

Added

• `__NEXT_DATA__` extraction: Next.js pages now have their `pageProps` JSON extracted into `structured_data`. Contains prices, product info, page state, and other data that isn't in the visible HTML.

Added

• SvelteKit data island extraction: extracts structured JSON from `kit.start()` data arrays. Handles unquoted JS object keys by converting to valid JSON before parsing. Data appears in the `structured_data` field.

Changed

• License changed from MIT to AGPL-3.0.

Changed

• Replaced custom TLS stack with wreq: migrated from webclaw-tls (patched rustls/h2/hyper/reqwest) to [wreq](https://github.com/0x676e67/wreq) by [@0x676e67](https://github.com/0x676e67).
• Removed all `[patch.crates-io]` entries: consumers no longer need to patch rustls, h2, hyper, hyper-util, or reqwest. Just depend on webclaw normally.
• Browser profiles rebuilt on wreq's Emulation API: Chrome 145, Firefox 135, Safari 18, Edge 145 with correct TLS options (cipher suites, curves, GREASE, ECH, PSK session resumption), HTTP/2 SETTINGS ordering, pseudo-header order, and header…
• Better TLS compatibility: BoringSSL handles more server configurations than patched rustls (e.g. servers that previously returned IllegalParameter alerts).

Removed

• webclaw-tls dependency and all 5 forked crates (webclaw-rustls, webclaw-h2, webclaw-hyper, webclaw-hyper-util, webclaw-reqwest).

Added

• `--cookie-file` flag: load cookies from JSON files exported by browser extensions (EditThisCookie, Cookie-Editor). Format: `[{name, value, domain, ...}]`.
• MCP `cookies` parameter: the `scrape` tool now accepts a `cookies` array for authenticated scraping.
• Combined cookies: `--cookie` and `--cookie-file` can be used together and merge automatically.

Added

• Cookie warmup fallback: when a fetch returns an Akamai challenge page, automatically visits the homepage first to collect `_abck`/`bm_sz` cookies, then retries the original URL. Enables extraction of Akamai-protected subpages (e.g.

Changed

• Fixed HTTP header wire order (accept/user-agent were in wrong positions) and added H2 PRIORITY flag in HEADERS frames.
• `FetchResult.headers` now uses `http::HeaderMap` instead of `HashMap<String, String>` — avoids per-response allocation, preserves multi-value headers.

Changed

• Replaced primp with webclaw-tls: switched to custom TLS fingerprinting stack.
• Browser profiles: Chrome 146 (Win/Mac), Firefox 135+, Safari 18, Edge 146 — captured from real browsers.
• HTTP/2 fingerprinting: SETTINGS frame ordering and pseudo-header ordering based on concepts pioneered by [@0x676e67](https://github.com/0x676e67).

Fixed

• HTTPS completely broken: primp's forked rustls rejected valid certificates (UnknownIssuer on cross-signed chains like example.com). Fixed by using native OS root CAs alongside Mozilla bundle.
• Unknown certificate extensions: servers returning SCT in certificate entries no longer cause TLS errors.

Added

• Native root CA support: uses OS trust store (macOS Keychain, Windows cert store) in addition to webpki-roots.
• HTTP/2 fingerprinting: SETTINGS frame ordering and pseudo-header ordering match real browsers.
• Per-browser header ordering: HTTP headers sent in browser-specific wire order.
• Bandwidth tracking: atomic byte counters shared across cloned clients.

Fixed

• `cargo install` broken with primp 1.2.0: added missing `reqwest` patch to `[patch.crates-io]`. primp moved to reqwest 0.13 which requires a patched fork.
• Weekly dependency check: CI now runs every Monday to catch primp patch drift before users hit it.

Added

• Docker image on GHCR: `docker run ghcr.io/0xmassi/webclaw` — auto-built on every release
• QuickJS data island extraction: inline `<script>` execution catches `window.__PRELOADED_STATE__`, Next.js hydration data, and other JS-embedded content

Fixed

• Docker CI now runs as part of the release workflow (was missing, image was never published)

Added

• DOCX extraction: auto-detected by Content-Type or URL extension, outputs markdown with headings
• XLSX/XLS extraction: spreadsheets converted to markdown tables, multi-sheet support via calamine
• CSV extraction: parsed with quoted field handling, output as markdown table
• HTML output format: `-f html` returns sanitized HTML from the extracted content
• Multi-URL watch: `--watch` now works with `--urls-file` to monitor multiple URLs in parallel
• Batch + LLM extraction: `--extract-prompt` and `--extract-json` now work with multiple URLs
• Scheduled batch watch: watch multiple URLs with aggregate change reports and per-URL diffs

Fixed

• `--only-main-content`, `--include`, and `--exclude` now work in batch mode

Added

• `--watch`: monitor a URL for changes at a configurable interval with diff output
• `--watch-interval`: seconds between checks (default: 300)
• `--on-change`: run a command when changes are detected (diff JSON piped to stdin)
• `--webhook`: POST JSON notifications on crawl/batch complete and watch changes. Auto-formats for Discord and Slack webhooks

Added

• `--output-dir`: save each page to a separate file instead of stdout. Works with single URL, crawl, and batch modes
• CSV input with custom filenames: `url,filename` format in `--urls-file`
• Root URLs use `hostname/index.ext` to avoid collisions in batch mode
• Subdirectories created automatically from URL path structure

Added

• QuickJS integration for extracting data from inline JavaScript (NYTimes +168%, Wired +580% more content)
• Executes inline `<script>` tags in a sandboxed runtime to capture `window.__*` data blobs
• Parses Next.js RSC flight data (`self.__next_f`) for App Router sites
• Smart text filtering rejects CSS, base64, file paths, and code — only keeps readable prose
• Feature-gated with `quickjs` feature flag (enabled by default, disable for WASM builds)

Added

• Crawl streaming: real-time progress on stderr as pages complete (`[2/50] OK https://... (234ms, 1523 words)`)
• Crawl resume/cancel: `--crawl-state <path>` saves progress on Ctrl+C and resumes from where it left off
• MCP server proxy support via `WEBCLAW_PROXY` and `WEBCLAW_PROXY_FILE` env vars

Changed

• Crawl results now expose visited set and remaining frontier for accurate state persistence

Changed

• Default TLS profile switched from Chrome145/Win to Safari26/Mac (highest pass rate across CF-protected sites)
• Plain client fallback: when impersonated TLS gets connection error or 403, automatically retries without impersonation (fixes ycombinator.com, producthunt.com, and similar sites)

Fixed

• Reddit scraping: use plain HTTP client for `.json` endpoint (TLS fingerprinting was getting blocked)

Added

• YouTube transcript extraction infrastructure in webclaw-core (caption track parsing, timed text XML parser) — wired up when cloud API launches

Fixed

• MCP server now identifies as `webclaw-mcp` instead of `rmcp` in the MCP handshake
• Research tool polling caps at 200 iterations (~10 min) instead of looping forever
• CLI returns non-zero exit codes on errors (invalid format, fetch failures, missing LLM)
• Text format output strips markdown table syntax (`| --- |` pipes)
• All MCP tools validate URLs before network calls with clear error messages
• Cloud API HTTP client has 60s timeout instead of no timeout
• Local fetch calls timeout after 30s to prevent hanging on slow servers
• Diff cloud fallback computes actual diff instead of returning raw scrape JSON
• FetchClient startup failure logs and exits gracefully instead of panicking

Added

• Upper bounds: batch capped at 100 URLs, crawl capped at 500 pages