Click, type and scroll before you scrape
Open menus, fill forms and load more. Then read the page.
Send a request
actions.sh
curl -X POST https://api.spicrawl.com/v1/scrape \ -H "Authorization: Bearer spicrawl_live_…" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com", "js_render": true, "response_format": "markdown", "actions": [ { "click": "#accept" }, { "fill": { "selector": "#search", "value": "shoes" } }, { "scroll": { "y": 800 } } ] }'How it works
- Turn on rendering with
js_render: true. Actions need a browser. - List your steps in
actions. They run in order after the page loads. - Get the page as it looks after the last step, in the format you chose.
curl https://api.spicrawl.com/v1/scrape \
-H "Authorization: Bearer $SPICRAWL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/products?category=lamps",
"js_render": true,
"response_format": "markdown",
"actions": [
{"click": "button.load-more"},
{"wait_for": ".product-card:nth-child(40)"}
]
}'import os, requests
r = requests.post(
"https://api.spicrawl.com/v1/scrape",
headers={"Authorization": f"Bearer {os.environ['SPICRAWL_API_KEY']}"},
json={
"url": "https://example.com/products?category=lamps",
"js_render": True,
"response_format": "markdown",
"actions": [
{"click": "button.load-more"},
{"wait_for": ".product-card:nth-child(40)"},
],
},
timeout=180,
)
print(r.text)spicrawl scrape "https://example.com/products?category=lamps" --render --format markdown \
--actions '[{"click":"button.load-more"},{"wait_for":".product-card:nth-child(40)"}]'Steps you can use
| Step | What it does |
|---|---|
wait_for | Waits for an element, or for a set time |
click | Clicks a button or link |
fill | Types into an input. Add secret: true to hide the value from logs |
select | Picks an option in a list |
scroll | Scrolls the page, for feeds that load more as you go |
evaluate | Runs your own JavaScript on the page |
screenshot | Saves an image of the page at that step |
Good to know
- Up to 50 steps per request. The whole list is checked before anything is spent.
- Browser only. Without
js_renderor a browser engine, the request is refused with a 400. - Not on batch. Actions run on single scrapes only.
- Logins: only log in to accounts you own or may use. To stay logged in, use a session.
- Pricing: free during the beta. A rendered request uses 3 credits, or 8 with full Chromium.
Frequently asked questions
What can browser actions do?
Wait for an element, wait for navigation, click, fill an input, select an option, scroll, run JavaScript and take a screenshot. The steps run in order after the page loads.
How many steps can one request have?
Up to 50. The whole list is checked before the request runs, so a mistake fails early and costs nothing.
Do browser actions need JavaScript rendering?
Yes. Actions run in a browser, so set js_render to true or choose a browser engine. Without one, the request is refused with a 400.
Can I fill in a password safely?
Yes. Mark the fill step with secret set to true, and the value is hidden from logs. Only log in to accounts you own or may use.
Can I get a screenshot after the steps?
Yes. Add a screenshot step. The response then becomes the JSON envelope, with the images under screenshots.