A hosted web scraping MCP server for AI agents

What the Spicrawl MCP server does

The server gives an AI agent Spicrawl as native tools. It is hosted at https://mcp.spicrawl.com/mcp, speaks MCP Streamable HTTP, and authenticates with your own API key as a bearer token. Every tool call runs under that key exactly as if you had called the API yourself: the same limits and the same credits.

Set it up in one command

The Spicrawl CLI adds the MCP server and the Spicrawl agent skill to your client, and shows the changes before applying them:

npm install -g @spicrawl/cli
spicrawl init --client claude   # or: --client cursor, --client codex

Or add it by hand. Keep the key in the SPICRAWL_API_KEY environment variable and reference it from the config, so the key never lands in a committed file:

claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
  --header "Authorization: Bearer $SPICRAWL_API_KEY"

# Or, for a team, commit .mcp.json (each person's own key is used):
{
  "mcpServers": {
    "spicrawl": {
      "type": "http",
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${SPICRAWL_API_KEY}" }
    }
  }
}

// .cursor/mcp.json (or ~/.cursor/mcp.json for every project)
{
  "mcpServers": {
    "spicrawl": {
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${env:SPICRAWL_API_KEY}" }
    }
  }
}

// .vscode/mcp.json: VS Code asks for the key once and stores it securely
{
  "inputs": [
    { "type": "promptString", "id": "spicrawl-api-key", "description": "Spicrawl API key", "password": true }
  ],
  "servers": {
    "spicrawl": {
      "type": "http",
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${input:spicrawl-api-key}" }
    }
  }
}

codex mcp add spicrawl --url https://mcp.spicrawl.com/mcp --bearer-token-env-var SPICRAWL_API_KEY

# Equivalent ~/.codex/config.toml:
[mcp_servers.spicrawl]
url = "https://mcp.spicrawl.com/mcp"
bearer_token_env_var = "SPICRAWL_API_KEY"

// claude_desktop_config.json: bridges to the hosted server with mcp-remote (needs Node.js)
{
  "mcpServers": {
    "spicrawl": {
      "command": "npx",
      "args": ["-y", "mcp-remote", "https://mcp.spicrawl.com/mcp", "--header", "Authorization:${SPICRAWL_AUTH}"],
      "env": { "SPICRAWL_AUTH": "Bearer spicrawl_live_..." }
    }
  }
}

Check the connection with claude mcp list, codex mcp list or your client’s MCP settings. You should see 25 spicrawl_* tools.

The tools, grouped by job

The tool for connecting to a remote browser over CDP is listed but coming soon.

A first task to try

Use Spicrawl to read https://example.com/pricing as markdown.
List each plan with its monthly price. If the page comes back empty,
retry with rendering.

The agent calls spicrawl_scrape with the URL and format: "markdown", and adds render: true only if the first result is empty. A full walkthrough is in how to scrape a website with Claude Code.

Keep agents cheap and safe

• Cheapest first. A plain request is 1 credit, browser rendering 3, full Chromium 8. Agents should escalate one step at a time and stop at the first that works.
• Cap every call. Set max_cost and a request that would cost more is refused before it runs.
• Failures are free. Errors and blocked pages cost 0 credits.
• Test without spending. spicrawl_test_ keys run against a sandbox and never spend live credits.
• Limit what a key can do. Keys get the scrape, batch and sessions scopes by default; grant read only if the agent should see usage.

What it does not do

• No crawling. Spicrawl does not follow links or discover URLs. Give the agent the URLs, or a sitemap’s list, and use a batch job for many pages.
• Coming soon: managed proxies, stealth mode, a remote browser connection, and extraction from a plain-language prompt. Today you can bring your own proxy at no extra cost, and extract data with CSS selectors or a JSON Schema.

How it compares with other scraping MCP servers

Most scraping APIs now ship an MCP server. What each offers, from our comparison pages, which cite each vendor’s own docs:

Choose Spicrawl’s server when your agent mostly reads pages it already has URLs for, and you want clean Markdown, sessions for logged-in pages and failures that cost nothing. Choose one that crawls or searches when the agent has to find pages first.