Capture the API calls behind a web page

How it works

1. Render the page. Set js_render: true, because capture only runs in a browser.
2. Choose the calls. Set network_capture to a urls list of patterns, such as /api/search.
3. Read the bodies. Each matching call comes back in the network array with its status, type and body.

curl -sS https://api.spicrawl.com/v1/scrape \
  -H "Authorization: Bearer $SPICRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/search?q=desk",
    "js_render": true,
    "network_capture": { "urls": ["/api/search"] }
  }'

spicrawl scrape "https://example.com/search?q=desk" --render \
  --network-capture '{"urls":["/api/search"]}'

Keep your key in SPICRAWL_API_KEY.

What comes back

An excerpt of the documented response. The page contents are an example.

{
  "status": 200,
  "credits": 3,
  "network": [
    {
      "url": "https://example.com/api/search?q=desk&page=1",
      "status": 200,
      "resource_type": "fetch",
      "content_type": "application/json",
      "body": { "total": 38, "items": [{ "id": 42, "name": "Walnut Desk", "price": 349.0 }] }
    }
  ]
}

When to use it

• The page fills itself from an API. The JSON is usually cleaner and steadier than the markup drawn from it.
• Search, listings and feeds. Results often load as one JSON call you can read whole.
• Otherwise, use structured extraction with CSS selectors or a JSON Schema.

Good to know

• Needs js_render. It adds no credits on top of the 3 for JavaScript rendering. Failed requests cost 0.
• Be respectful. Only fetch data you are allowed to use, and follow the site’s terms.
• Pricing: free during the beta, with 1,000 credits a month.