Scrape JavaScript websites

How it works

1. Send the page URL to POST /v1/scrape with js_render: true.
2. Wait for the content. Set wait_for to a CSS selector, such as .price.
3. Get the rendered page in the format you choose. The examples below return Markdown.

curl -sS -D - https://api.spicrawl.com/v1/scrape \
  -H "Authorization: Bearer $SPICRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/products/42", "js_render": true, "wait_for": ".price", "response_format": "markdown"}'

import os, requests

r = requests.post(
    "https://api.spicrawl.com/v1/scrape",
    headers={"Authorization": f"Bearer {os.environ['SPICRAWL_API_KEY']}"},
    json={"url": "https://example.com/products/42", "js_render": True, "wait_for": ".price", "response_format": "markdown"},
    timeout=180,
)
print("site status:", r.headers.get("X-Target-Status"))
print(r.text)

const r = await fetch("https://api.spicrawl.com/v1/scrape", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.SPICRAWL_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://example.com/products/42", js_render: true, wait_for: ".price", response_format: "markdown" }),
});
console.log("site status:", r.headers.get("X-Target-Status"));
console.log(await r.text());

spicrawl scrape https://example.com/products/42 --render --wait-for .price --format markdown --meta

Keep your key in SPICRAWL_API_KEY. Never put it in browser code or a file you commit.

When to render JavaScript

A plain request downloads HTML without running scripts. If the result is an empty shell or a “Loading…” message, the page may need JavaScript. Set js_render: true to run it in a browser before reading it.

Use a plain request for content already in the HTML. Rendering is also needed for browser actions, resource blocking and network capture.

Wait for the right content

An explicit resource list replaces the default. Use ["none"] to block nothing. Do not block scripts when the page needs them to build its content.

Check the result

HTTP/2 200
content-type: text/markdown; charset=utf-8
x-target-status: 200
x-engine: obscura
x-credits-charged: 3

X-Engine tells you which engine ran. X-Target-Status is the site's status, while X-Credits-Charged shows the charge.

If wait_for never matches, the page may come back as it stood with X-Warning: RENDER_DEGRADED. Check the content and warning before using it.

Choose an engine and check the cost

• Free during beta. Credits show the cost of each request.
• Browser rendering: js_render: true normally uses Obscura, at 3 credits per successful request.
• Chromium: costs 8 credits. If Obscura is not deployed, an unpinned render may use Chromium and return X-Warning: ENGINE_SUBSTITUTED.
• Pin an engine with engine: "obscura" to avoid substitution. Your plan must allow it; a pinned engine never escalates.
• Failures cost 0. Waiting and resource blocking add no extra charge. Set max_cost to cap a request.

Good to know

• Choose rendering or auto mode. mode: "auto" cannot be combined with js_render or engine. It starts with fetch and may move to Obscura; it never reaches Chromium.
• Browser flags need a browser. Sending wait_for, actions or resource blocking with the fetch engine returns an error.
• No crawling. Send the URLs you want; Spicrawl does not find pages or follow links.
• Coming soon: managed proxies, stealth mode, a remote browser and extraction from a plain-language prompt.
• Your responsibility: only fetch pages you have the right to use. Read our terms.