# Spicrawl

> Spicrawl is a web scraping API that turns web pages into clean data for apps, AI agents and automation workflows. Send a URL and get Markdown, JSON, HTML, text or PDF back. Free during beta, no credit card required.

Spicrawl can fetch a page with a plain HTTP request or render JavaScript in a real browser. With `mode=auto` it tries a plain request first and moves to a browser only when needed. You pay credits only for successful requests (1 for a plain request, 3 for browser rendering, 8 for full Chromium); errors, bot challenges and cached results cost nothing.

## Features

- Scrape one URL, or submit up to 10,000 URLs per call as a batch job with JSON Lines results
- JavaScript rendering in a real browser
- Output as Markdown (main content only), JSON, HTML, text or PDF
- Structured data extraction with autoparse, CSS selectors or a JSON Schema (extraction with a model prompt is coming soon)
- Sessions that keep cookies and browser storage between requests, so a login carries over
- Browser actions: click, type, select, scroll and run JavaScript before the page is read (up to 50 steps)
- Bring your own proxy with no proxy surcharge (managed proxies are coming soon)
- Resource blocking, network capture and caching for up to 48 hours
- Hosted MCP server so AI agents can scrape pages, run batches, reuse sessions and check usage
- CLI (`spicrawl`), a JavaScript/TypeScript SDK (`@spicrawl/sdk`), and a dashboard with a playground, activity logs, usage and API key management

## Links

- [Homepage](https://spicrawl.com/)
- [Documentation](https://docs.spicrawl.com/): guides, API reference and OpenAPI spec
- [Docs index for LLMs](https://docs.spicrawl.com/llms.txt)
- [Sign up for free beta access](https://app.spicrawl.com/signup)
- API base URL: https://api.spicrawl.com
- MCP server: https://mcp.spicrawl.com/mcp
- Support: support@spicrawl.com
- X (Twitter): https://x.com/Spicrawl

## Product pages

- [Convert any website to clean Markdown for LLMs](https://spicrawl.com/website-to-markdown): Convert any web page to clean Markdown for LLMs and RAG with one API call. Main content only, JavaScript pages, PDFs and batches. Free during beta.
- [A hosted web scraping MCP server for AI agents](https://spicrawl.com/web-scraping-mcp-server): Give Claude Code, Cursor, Codex or any MCP client live web data. Hosted Spicrawl MCP server with 25 tools: scrape, batch, sessions and docs. Free in beta.

## Comparisons

- [Spicrawl vs Firecrawl](https://spicrawl.com/vs/firecrawl): Choose Spicrawl to stop paying for cache hits and blocked pages. Choose Firecrawl if you need site crawling, web search or self-hosting.
- [Spicrawl vs ScrapingBee](https://spicrawl.com/vs/scrapingbee): ScrapingBee and Spicrawl are close. Choose Spicrawl for sessions that keep logins; choose ScrapingBee for ready-made site APIs and managed proxies today.
- [Spicrawl vs Apify](https://spicrawl.com/vs/apify): Choose Spicrawl for one-call page-to-Markdown at a predictable per-request cost. Choose Apify for ready-made scrapers, site crawling or running your own code.
- [Spicrawl vs ZenRows](https://spicrawl.com/vs/zenrows): Both bill only successful requests. Choose Spicrawl for free cache hits and bring-your-own proxy; choose ZenRows for a remote browser, bigger batches and managed proxies.
- [Spicrawl vs Context.dev](https://spicrawl.com/vs/context-dev): Choose Spicrawl for free cache hits and richer browser actions. Choose Context.dev for site crawling, page monitoring, brand data and larger batches.
- [Spicrawl vs Geonode](https://spicrawl.com/vs/geonode): Choose Spicrawl for more output formats and larger batches. Choose Geonode for its proxy pools, site crawling and a flat price per request.
- [Spicrawl vs Bright Data](https://spicrawl.com/vs/bright-data): Choose Spicrawl for one simple scrape API with sessions for logged-in pages; choose Bright Data for proxies, site crawling, remote browsers and ready-made scrapers.
- [Spicrawl vs Crawl4AI](https://spicrawl.com/vs/crawl4ai): Choose Spicrawl for a hosted API with free cache hits and nothing to run. Choose Crawl4AI for open source, self-hosting, site crawling and stealth.
- [Spicrawl vs Jina Reader](https://spicrawl.com/vs/jina-reader): Choose Spicrawl for big batches and logged-in pages. Choose Jina Reader for no-key use, web search, image and document reading or self-hosting.
- [Spicrawl vs Browserbase](https://spicrawl.com/vs/browserbase): Choose Spicrawl to turn known URLs into clean content in one call. Choose Browserbase when an agent must drive a real browser, with long sessions and recordings.

## Blog

- [Cloudflare AI crawler blocking in 2026: what changed and what it means for your agent](https://spicrawl.com/blog/cloudflare-blocking-ai-crawlers): Cloudflare lets sites block Search, Agent and Training bots separately, with new defaults from 15 September 2026. What changed and how to scrape responsibly.
- [Web scraping 403 Forbidden: causes and fixes](https://spicrawl.com/blog/web-scraping-403-forbidden): Why a scraper gets a 403 Forbidden, how to tell the causes apart, and Python fixes that are fair to the site: headers, pacing, robots.txt and sessions.
- [What is llms.txt? Examples and how to write one](https://spicrawl.com/blog/llms-txt-explained): llms.txt is a plain file that points AI tools to your best pages. Learn the format, see a real example, and read what Google says about it.
- [The best web scraping tools for AI agents in 2026](https://spicrawl.com/blog/best-web-scraping-tools-for-ai-agents): An honest comparison of 10 web scraping tools for AI agents, from scraping APIs and MCP servers to open-source crawlers and cloud browsers.
- [How to scrape a website with Claude Code](https://spicrawl.com/blog/scrape-websites-with-claude-code): Connect Claude Code to Spicrawl's hosted MCP server, read live pages as Markdown, and handle JavaScript, logins, costs and errors.
- [HTML vs Markdown for LLMs: which uses fewer tokens?](https://spicrawl.com/blog/html-vs-markdown-for-llms): We measured six public pages with tiktoken: HTML and Spicrawl Markdown token counts, the method, what conversion keeps and loses, and when HTML wins.