Scrape websites behind a login

How it works

1. Create a session. Keep the ID from the response.
2. Log in with browser actions. Send the session ID with the login request.
3. Read account pages. Reuse the same ID, one request at a time.
4. Release the session when you are done.

Your API key needs the sessions scope to create and manage sessions, and scrape to use one. Keep it in SPICRAWL_API_KEY.

1. Create a session

curl https://api.spicrawl.com/v1/sessions \
  -H "Authorization: Bearer $SPICRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"engine": "chromium", "ttl_seconds": 7200}'

import os, requests

API = "https://api.spicrawl.com"
headers = {"Authorization": f"Bearer {os.environ['SPICRAWL_API_KEY']}"}
s = requests.post(f"{API}/v1/sessions", headers=headers,
    json={"engine": "chromium", "ttl_seconds": 7200}, timeout=30)
s.raise_for_status()
session_id = s.json()["id"]

const API = "https://api.spicrawl.com";
const headers = {
  Authorization: `Bearer ${process.env.SPICRAWL_API_KEY}`,
  "Content-Type": "application/json",
};
const s = await fetch(`${API}/v1/sessions`, {
  method: "POST", headers,
  body: JSON.stringify({ engine: "chromium", ttl_seconds: 7200 }),
});
if (!s.ok) throw new Error(await s.text());
const { id: sessionId } = await s.json();

spicrawl sessions create --engine chromium --ttl 7200

The response includes id, engine, expires_at and hard_expires_at. It never includes cookies. The example below uses a sample ID; replace it with yours.

2. Log in with actions

Send this body to POST /v1/scrape with your bearer token. Change the URL and selectors to match the site. Use credentials for an account you own or may use.

{
  "url": "https://example.com/login",
  "session_id": "01J9ZQ4M7R3T8VX2K5N6P0B1CD",
  "js_render": true,
  "actions": [
    {
      "fill": {
        "selector": "input[name=email]",
        "value": "YOUR_EMAIL"
      }
    },
    {
      "fill": {
        "selector": "input[name=password]",
        "value": "YOUR_PASSWORD",
        "secret": true
      }
    },
    {
      "click": "button[type=submit]"
    },
    {
      "wait_for": ".account-menu"
    }
  ]
}

Replace the email and password placeholders securely at runtime. The password action uses secret: true. Keep js_render: true: a session ID alone does not allow browser-only fields.

3. Read pages with the same login

After the login succeeds, pass session_id on each scrape. Cookies and storage carry over. Python and TypeScript reuse the variables from the create step.

curl https://api.spicrawl.com/v1/scrape \
  -H "Authorization: Bearer $SPICRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/account/orders", "session_id": "01J9ZQ4M7R3T8VX2K5N6P0B1CD", "js_render": true, "response_format": "markdown"}'

r = requests.post(f"{API}/v1/scrape", headers=headers, json={
    "url": "https://example.com/account/orders", "session_id": session_id,
    "js_render": True, "response_format": "markdown",
}, timeout=120)
r.raise_for_status()
print(r.text)

const r = await fetch(`${API}/v1/scrape`, {
  method: "POST", headers,
  body: JSON.stringify({
    url: "https://example.com/account/orders", session_id: sessionId,
    js_render: true, response_format: "markdown",
  }),
});
if (!r.ok) throw new Error(await r.text());
console.log(await r.text());

spicrawl scrape https://example.com/account/orders --session 01J9ZQ4M7R3T8VX2K5N6P0B1CD --render --format markdown

Session requests always bypass the result cache. Check Cache-State: bypass; the page is fetched fresh.

4. Release the session

curl -X POST https://api.spicrawl.com/v1/sessions/01J9ZQ4M7R3T8VX2K5N6P0B1CD/release \
  -H "Authorization: Bearer $SPICRAWL_API_KEY"

Release deletes cookies and storage. It does not keep the login. Later requests on that ID return 410 ERR::SESSION::RELEASED.

Session options

To save a login before release, read GET /v1/sessions/{id}/context. Store that state safely, then pass it as session_context when creating a new session.

What it costs

Free during beta. Creating, reading and releasing sessions costs nothing. Each scrape uses the session engine's price: fetch 1 credit, Obscura 3, Chromium 8. Failures, including a busy or expired session, cost 0.

Good to know

• One request per session at a time. A second render returns 409 ERR::SESSION::BUSY with Retry-After. Use one session per worker for parallel work.
• Expired sessions need a new login. An unknown, deleted or malformed ID returns an error; the request never runs without its session.
• Your own proxy: use a fixed exit for sites that challenge a login when its IP changes. Managed session exits are coming soon.
• No crawling. You supply the URLs. Managed proxies, stealth, a remote browser and extraction from a prompt are coming soon.
• Your responsibility: only use accounts and pages you have the right to access. Read our terms.