Open Source Web Crawler Guide for 2026

You crawl a documentation site, save a folder of pages, and open the first result. Half the file contains navigation. The code example you needed is missing. Your script completed, but you still have to fix the input before you can build a search index.

Choose an open source web crawler around the output you need. For product records, you need consistent fields. For a documentation search engine, you need readable pages with headings, code blocks, and source URLs. Count those results when you evaluate a tool.

I build Webclaw, which I include below. I link to each project's documentation for capability claims. The recommendations reflect how I would shortlist tools for each workload; I have not run a head-to-head benchmark of these four crawlers.

Choose a crawler for your workload

Start with these four candidates if you are building a scraper or an AI ingestion pipeline. This shortlist does not cover web archiving or distributed search-engine crawling.

Language matters here. A Python team can often maintain a Scrapy spider with less effort than a separate browser service. A TypeScript team may prefer to keep request handling and extraction in Crawlee. Keep your existing deployment setup in mind before adding another runtime.

Scrapy: Python spiders and structured records

Pick Scrapy when you can describe the output as records: a product name and price, a job title and location, or an article title and publication date.

Scrapy provides CSS and XPath selectors, asynchronous request scheduling, item pipelines, and exports such as JSON and CSV. Its documentation includes a spider that extracts quotes, then follows pagination links. You write the parsing rules and decide which requests to schedule. See the Scrapy overview and example spider.

Start with one page type. Define the fields you require and reject records that miss them. A crawl that exports 1,000 rows with empty prices has not completed the job you intended.

For pages whose content requires browser interaction, account for that extra work before choosing your implementation. Check the page's response and the visible browser content; a selector cannot extract a field that the response does not contain.

Crawlee: HTTP and browser crawling in one JavaScript project

Consider Crawlee when your targets include both pages you can fetch over HTTP and pages that require a browser. Its JavaScript documentation describes a shared interface for HTTP and headless-browser crawling, a persistent request queue, storage, routing, and retry handling. See the Crawlee introduction.

You still write the extraction handler. That gives you control over what to keep, but you should budget time for selectors and page-specific behavior.

Before starting a large crawl, save the failed URL and error alongside successful output. Check that a retry produces the requested content rather than another empty result. For a closer look at browser automation versus a content-oriented workflow, read our Crawl4AI and Playwright comparison.

Crawl4AI: Markdown extraction in Python

Consider Crawl4AI when you want to fetch pages and process their Markdown in the same Python application. The project documents an asynchronous crawler, Markdown output, browser configuration, and structured extraction using CSS, XPath, or an LLM. See the Crawl4AI documentation.

Test your content filters on several page layouts. A rule that removes a documentation sidebar may also remove an example or a useful note on another template.

For a retrieval pipeline, inspect the content before splitting it into chunks. Keep the source URL with each document, and check whether code blocks and section headings survive extraction. Our guide to building a RAG pipeline with web data covers the next steps after collection.

Webclaw: a CLI crawl you can inspect on disk

Use the Webclaw CLI if you want to try a small crawl from your terminal and inspect Markdown files before writing an ingestion script. Follow the CLI installation and usage guide; the open-source page links to the project.

With the CLI installed, run:

webclaw https://docs.scrapy.org/en/latest/intro/overview.html \
  --crawl \
  --depth 1 \
  --max-pages 3 \
  --concurrency 1 \
  --delay 1000 \
  --only-main-content \
  --format markdown \
  --output-dir ./crawler-sample

I ran this command on September 21, 2026, with a temporary output directory. The crawl saved three Markdown files: Scrapy's overview, command-line tools, and architecture pages. The overview output contained the example spider, including its pagination code. The architecture output contained the description of requests passing through Scrapy's components.

That check establishes that this command produced useful content for this small sample. It does not establish a success rate for other sites or a speed advantage over another crawler. The pages discovered may also change when the source site changes its links.

Open the saved files before increasing the page limit. Check a paragraph and a code block against the source page. Use the same checks for the other candidates.

Evaluate two candidates on the same pages

Choose ten representative pages from a site you are allowed to crawl. Include a long page, a page with code or a table, and any layout that has caused extraction problems before.

For an extraction comparison, give both tools the same URL list. For a discovery comparison, give both the same starting URL and scope, then compare which pages they found. Mixing those tests makes it harder to tell whether you have a discovery problem or an extraction problem.

If you do not yet have a URL list, our site-mapping tools guide covers discovery options.

Record the tool version, configuration, and test date. Save the output files and errors so you can inspect disagreements rather than rerun from memory.

Define acceptance before measuring speed. For product data, you might require a name, price, currency, and source URL on each record. For documentation, require the selected paragraph and code sample to survive extraction.

Then measure elapsed time and resource use for the output you accepted. Keep setup effort separate from ongoing repair work. A tool you can configure in an afternoon may still need frequent selector changes on your target site.

Questions to settle before a larger crawl

Do you need crawling, scraping, or URL discovery?

Use crawling when you need to follow links and collect multiple pages. Use scraping when you already know the URLs and need their content. Use a mapping workflow when you need a URL inventory before selecting pages to fetch.

For a known list of documentation URLs, begin with that list. You can test extraction without also debugging link discovery.

Does open source mean the crawl costs nothing?

Budget for compute, storage, and the time you spend maintaining the crawler. Browser-driven jobs may require different resources from HTTP-only jobs. Check each project's current license and dependencies before adopting it.

A small local sample can tell you whether the output fits your application. It cannot tell you the monthly operating cost of a larger deployment without measurements from that workload.

Which open source crawler should you try for RAG?

Start with Crawl4AI if you want to work inside Python, or Webclaw if you want a terminal-based Markdown collection step. Include Scrapy when you need precise records that you will turn into documents yourself. Include Crawlee when browser interaction and JavaScript request handlers are central to the job.

Try two candidates on your target pages. Keep the files, inspect the missing content, and choose the one you can maintain while meeting your acceptance checks.