Frequently asked questions
Quick answers about Spicrawl, from pricing to proxies.
All questions
General
What is Spicrawl?
Spicrawl is a web scraping API that turns any URL into clean Markdown, HTML, a PDF or structured JSON for your code or AI agent, through an API, CLI or MCP server.
Is Spicrawl free?
Yes. It is free during the beta, with 1,000 credits a month and no credit card.
What does Spicrawl mean?
The name joins "spider" and "crawl", which is why our logo is a spider. It is spelled S-P-I-C-R-A-W-L. Spicrawl is a web scraping API: you send the URLs you want, and it returns clean data.
Why not use my AI assistant’s built-in web fetch?
Built-in fetch tools are fine for a quick look, but they often return raw or cut-off text and may not run JavaScript. Spicrawl returns clean Markdown or JSON, renders JavaScript when a page needs it, keeps login sessions, runs clicks and form inputs, and batches up to 10,000 URLs in one job. Your agent gets the content of the page, not its menus and scripts.
Is web scraping legal?
It depends on what you collect, where you are and how you use it. Scraping publicly available pages is common, but you are responsible for respecting each site’s terms, copyright and data protection laws such as GDPR, especially for personal data. If you are unsure, ask a lawyer about your specific case.
Scraping and data
Can I scrape JavaScript websites?
Yes. Set js_render to true to load the page in a real browser. With mode=auto, Spicrawl tries a plain request first and uses a browser only if needed.
Can Spicrawl get past bot protection like Cloudflare?
Often, but not always. Use a real browser fingerprint, browser rendering or your own proxy. A bot challenge returns a clear error and costs nothing. Stealth mode and managed proxies are not in the beta.
How do I turn a website into Markdown for an LLM?
Send the URL with response_format set to markdown. Spicrawl returns the main content of the page as Markdown, with menus, footers and asides removed, which usually takes a fraction of the tokens of raw HTML. It parses PDFs too, and Markdown conversion costs nothing extra.
Can I extract structured data as JSON?
Yes, in three ways: autoparse reads the page’s own metadata, a selector map pulls exact fields, and a JSON Schema with selectors returns typed, validated output. All three are free on top of the engine price. Extraction from a plain-language description, using a model, is coming soon.
Can I scrape many URLs at once?
Yes. A batch job accepts up to 10,000 URLs per call and runs them in parallel. You check its progress, then download the results as JSON Lines. Results are kept for 72 hours.
Can I keep a login session?
Yes. Create a session and pass its id on each request to reuse the same cookies, browser storage and exit IP, so a login carries over between pages. An organization can hold up to 100 live sessions at a time.
Can I automate clicks and forms?
Yes. Browser actions can click, type, select, scroll, wait and run JavaScript on a rendered page before it is captured, with up to 50 steps per request.
Can I use proxies?
Yes. Bring your own proxy with no proxy surcharge. Managed proxies and built-in country targeting are coming soon and are not available during beta.
AI agents and developers
Can I use Spicrawl with AI agents?
Yes. The hosted MCP server lets Claude Code, Cursor and Codex scrape pages, run batches, reuse sessions and run browser actions. Run spicrawl init to set it up.
Which programming languages can I use?
Any language that can send an HTTP request. The docs include examples in cURL, Python and TypeScript, and there is a JavaScript and TypeScript SDK on npm (@spicrawl/sdk) for Node.js 18 and later. The spicrawl CLI works from any shell or script.
Is there a CLI or a dashboard?
Both. A CLI covers scraping, batches, sessions, logs and usage, and the dashboard has a playground, activity logs, usage and key management.
Credits and limits
Do failed requests cost credits?
No. Only successful requests cost credits: 1 for plain, 3 for JavaScript rendering and 8 for full browser. Errors, bot challenges and timeouts are free.
Are there rate limits?
Yes. Each API key has a request rate, each organization has a limit on requests running at the same time, and there is a monthly credit allowance. Every response reports your current limits in its headers, and a request refused by a limit costs nothing.
Still stuck? Read the docs or email support@spicrawl.com.