SDK integration
Web scraping for LangChain
Turn clean webclaw output into native LangChain Documents.
LangChain is a framework for building LLM applications in Python and TypeScript. webclaw does not ship a dedicated LangChain loader; use the Python SDK and convert each response into LangChain's native Document type.
Setup
LangChain Python: native Document bridge
python
from langchain_core.documents import Document
from webclaw import Webclaw
with Webclaw("wc_...") as wc:
result = wc.scrape("https://example.com", formats=["llm"])
content = result.llm or result.markdown or result.text or ""
if not content:
raise RuntimeError("webclaw returned no document content")
docs = [Document(
page_content=content,
metadata={"source": result.url},
)]
# Feed into your vector store
vectorstore.add_documents(docs)Why webclaw for LangChain
- One explicit SDK-to-Document conversion step
- 118ms on static pages, faster than browser-based loaders
- Handles bot-protected sites automatically
- LLM-optimized markdown cuts token costs ~90%
Common use cases
- RAG pipelines with fresh web content
- Native Document ingestion with bot protection bypass
- LangChain agents with real-time web tools
- Multi-source research chains
Yes, at the ingestion boundary. Fetch with the webclaw Python SDK and wrap the returned content in langchain_core.documents.Document. The rest of your splitter, vector store, and chain code stays unchanged.