SDK integration
Web scraping for LlamaIndex
Ingest web content into LlamaIndex nodes with clean markdown.
LlamaIndex is a data framework for LLM applications specializing in retrieval and context augmentation. webclaw provides clean markdown and structured data that loads directly into LlamaIndex Documents, with automatic bot bypass for protected sites.
Setup
LlamaIndex Python: native Document bridge
python
from llama_index.core import Document, VectorStoreIndex
from webclaw import Webclaw
# Scrape a site with LLM-optimized output
urls = ["https://example.com/docs", "https://example.com/api"]
with Webclaw("wc_...") as wc:
results = wc.batch(urls=urls, formats=["llm"])
# Convert to LlamaIndex Documents
docs = [
Document(
text=r.llm or r.markdown or r.text,
metadata={"source": r.url},
)
for r in results.results
if r.llm or r.markdown or r.text
]
# Build the index
index = VectorStoreIndex.from_documents(docs)
query_engine = index.as_query_engine()Why webclaw for LlamaIndex
- Clean markdown output ideal for semantic chunking
- Batch endpoint for parallel multi-URL ingestion
- ~90% token reduction cuts embedding costs
- Content diff for incremental index updates
Common use cases
- Native LlamaIndex Document ingestion for web content
- Vector indexes over fresh web documentation
- Hybrid search with structured + unstructured data
- Query engines over competitor or market data
Scrape URLs with the webclaw SDK, convert each result into a native LlamaIndex Document, and pass them to VectorStoreIndex.from_documents. The SDK does not ship a dedicated reader module.