Skip to content

Click, type and scroll before you scrape

Open menus, fill forms and load more. Then read the page.

Send a request
actions.sh
curl -X POST https://api.spicrawl.com/v1/scrape \  -H "Authorization: Bearer spicrawl_live_…" \  -H "Content-Type: application/json" \  -d '{    "url": "https://example.com",    "js_render": true,    "response_format": "markdown",    "actions": [      { "click": "#accept" },      { "fill": { "selector": "#search", "value": "shoes" } },      { "scroll": { "y": 800 } }    ]  }'

How it works

  1. Turn on rendering with js_render: true. Actions need a browser.
  2. List your steps in actions. They run in order after the page loads.
  3. Get the page as it looks after the last step, in the format you chose.
curl https://api.spicrawl.com/v1/scrape \
  -H "Authorization: Bearer $SPICRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products?category=lamps",
    "js_render": true,
    "response_format": "markdown",
    "actions": [
      {"click": "button.load-more"},
      {"wait_for": ".product-card:nth-child(40)"}
    ]
  }'

Steps you can use

StepWhat it does
wait_forWaits for an element, or for a set time
clickClicks a button or link
fillTypes into an input. Add secret: true to hide the value from logs
selectPicks an option in a list
scrollScrolls the page, for feeds that load more as you go
evaluateRuns your own JavaScript on the page
screenshotSaves an image of the page at that step

Good to know

  • Up to 50 steps per request. The whole list is checked before anything is spent.
  • Browser only. Without js_render or a browser engine, the request is refused with a 400.
  • Not on batch. Actions run on single scrapes only.
  • Logins: only log in to accounts you own or may use. To stay logged in, use a session.
  • Pricing: free during the beta. A rendered request uses 3 credits, or 8 with full Chromium.

Frequently asked questions

What can browser actions do?

Wait for an element, wait for navigation, click, fill an input, select an option, scroll, run JavaScript and take a screenshot. The steps run in order after the page loads.

How many steps can one request have?

Up to 50. The whole list is checked before the request runs, so a mistake fails early and costs nothing.

Do browser actions need JavaScript rendering?

Yes. Actions run in a browser, so set js_render to true or choose a browser engine. Without one, the request is refused with a 400.

Can I fill in a password safely?

Yes. Mark the fill step with secret set to true, and the value is hidden from logs. Only log in to accounts you own or may use.

Can I get a screenshot after the steps?

Yes. Add a screenshot step. The response then becomes the JSON envelope, with the images under screenshots.