Go SDK

The Go SDK provides typed clients for supported webclaw operations. Zero dependencies beyond the standard library, context.Context on every method, and functional options for configuration.

Installation

go get

go get github.com/0xMassi/webclaw-go

Note

Requires Go 1.23 or later. No external dependencies. Code fragments below assume a client, context, and standard-library imports in your Go program.

Configuration

Create a client with your API key and optional functional options.

Basic

import webclaw "github.com/0xMassi/webclaw-go"

client := webclaw.NewClient("wc_your_api_key")

Options

Function	Default	Description
`WithBaseURL(url)`	`https://api.webclaw.io`	Override for self-hosted instances.
`WithTimeout(d)`	`30s`	HTTP client timeout.
`WithHTTPClient(hc)`	default http.Client	Replace the underlying HTTP client entirely.

All options

client := webclaw.NewClient(
    "wc_your_api_key",
    webclaw.WithBaseURL("https://api.webclaw.io"),
    webclaw.WithTimeout(60 * time.Second),
    webclaw.WithHTTPClient(&http.Client{
        Transport: customTransport,
    }),
)

Scrape

result, err := client.Scrape(ctx, &webclaw.ScrapeRequest{
    URL:              "https://example.com",
    Formats:          []webclaw.Format{webclaw.FormatMarkdown, webclaw.FormatLLM},
    IncludeSelectors: []string{"article", ".content"},
    ExcludeSelectors: []string{"nav", "footer"},
    OnlyMainContent:  true,
    NoCache:          true,
})
if err != nil {
    log.Fatal(err)
}

fmt.Println(result.URL)
fmt.Println(result.Markdown)
fmt.Println(result.LLM)
fmt.Println(result.Cache.Status)  // "hit" | "miss" | "bypass"

Crawl

Start an async crawl, then poll or wait for completion.

job, err := client.Crawl(ctx, &webclaw.CrawlRequest{
    URL:        "https://example.com",
    MaxDepth:   3,
    MaxPages:   100,
    UseSitemap: true,
})
if err != nil {
    log.Fatal(err)
}

// Block until complete (polls every 2s, 5min timeout)
status, err := client.WaitForCompletion(ctx, job.ID, &webclaw.CrawlPollOptions{
    Interval: 2 * time.Second,
    Timeout:  5 * time.Minute,
})
if err != nil {
    log.Fatal(err)
}

if status.Status != webclaw.CrawlStatusCompleted {
    log.Fatal("Crawl failed")
}
fmt.Println(status.Status)
fmt.Println(status.Total)     // pages discovered
fmt.Println(status.Completed) // pages crawled

for _, page := range status.Pages {
    fmt.Printf("%s: %d bytes\n", page.URL, len(page.Markdown))
}

Tip

WaitForCompletion respects the context deadline -- pass a context with timeout for upper-bound control.

Map

result, err := client.Map(ctx, &webclaw.MapRequest{
    URL: "https://example.com",
})
if err != nil {
    log.Fatal(err)
}

fmt.Println(result.Count)
for _, u := range result.URLs {
    fmt.Println(u)
}

Batch

result, err := client.Batch(ctx, &webclaw.BatchRequest{
    URLs:        []string{"https://a.com", "https://b.com", "https://c.com"},
    Formats:     []webclaw.Format{webclaw.FormatMarkdown},
    Concurrency: 5,
})
if err != nil {
    log.Fatal(err)
}

for _, item := range result.Results {
    if item.Error != "" {
        fmt.Printf("FAIL %s: %s\n", item.URL, item.Error)
    } else {
        fmt.Printf("OK   %s: %d chars\n", item.URL, len(item.Markdown))
    }
}

Extract

LLM-powered structured extraction. Pass a JSON schema or a plain-text prompt.

Schema-based

result, err := client.Extract(ctx, &webclaw.ExtractRequest{
    URL:    "https://example.com/pricing",
    Schema: json.RawMessage(`{
        "type": "object",
        "properties": {
            "plans": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "name": {"type": "string"},
                        "price": {"type": "string"}
                    }
                }
            }
        }
    }`),
})
if err != nil {
    log.Fatal(err)
}

fmt.Println(string(result.Data))

Prompt-based

result, err := client.Extract(ctx, &webclaw.ExtractRequest{
    URL:    "https://example.com",
    Prompt: "Extract all pricing tiers with names and monthly prices",
})
if err != nil {
    log.Fatal(err)
}

fmt.Println(string(result.Data))

Summarize

result, err := client.Summarize(ctx, &webclaw.SummarizeRequest{
    URL:          "https://example.com",
    MaxSentences: 3,
})
if err != nil {
    log.Fatal(err)
}

fmt.Println(result.Summary)

Brand

result, err := client.Brand(ctx, &webclaw.BrandRequest{
    URL: "https://example.com",
})
if err != nil {
    log.Fatal(err)
}

fmt.Println(string(result.Data))

Error handling

API errors are returned as *APIError with status code and message. The SDK ships matching predicates — IsAuthError, IsRateLimited, IsNotFound and IsForbidden — which unwrap the chain for you, so a wrapped error still matches.

Field	Type	Description
`StatusCode`	`int`	HTTP status code (401, 404, 429, etc.)
`Message`	`string`	Human-readable error message

result, err := client.Scrape(ctx, req)
if err != nil {
    switch {
    case webclaw.IsAuthError(err):
        fmt.Println("Check your API key")
    case webclaw.IsRateLimited(err):
        fmt.Println("Rate limited, slow down")
    case webclaw.IsNotFound(err):
        fmt.Println("No such resource")
    default:
        fmt.Println(err)
    }
    return err
}

Types and constants

The SDK exports typed constants for formats and crawl statuses.

// Output formats
webclaw.FormatMarkdown  // "markdown"
webclaw.FormatText      // "text"
webclaw.FormatLLM       // "llm"
webclaw.FormatJSON      // "json"

// Crawl statuses
webclaw.CrawlStatusRunning    // "running"
webclaw.CrawlStatusCompleted  // "completed"
webclaw.CrawlStatusFailed     // "failed"

// Cache statuses
webclaw.CacheHit     // "hit"
webclaw.CacheMiss    // "miss"
webclaw.CacheBypass  // "bypass"

Source

github.com/0xMassi/webclaw-go