Web scraping for RAG and LLM pipelines

Why Markdown for RAG

• Structure survives. Headings, lists, tables and code stay, so you can split on real sections instead of arbitrary character counts.
• Far fewer tokens. With menus, footers and ads removed, the main content is typically 3% to 24% of the raw HTML’s tokens.
• Main content by default. Narrow it further with include_tags and exclude_tags.

How it works

1. Send a URL to POST /v1/scrape with response_format: "markdown".
2. Chunk and embed the Markdown in your own pipeline. Spicrawl does not do this step.
3. Refresh on your schedule by sending the same URLs again. For many pages, use one batch job of up to 10,000 URLs.

curl -sS https://api.spicrawl.com/v1/scrape \
  -H "Authorization: Bearer $SPICRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/docs/getting-started", "response_format": "markdown"}'

import os, requests

H = {"Authorization": f"Bearer {os.environ['SPICRAWL_API_KEY']}"}

page = requests.post("https://api.spicrawl.com/v1/scrape", headers=H, json={
    "url": "https://example.com/docs/getting-started",
    "response_format": "markdown",
}, timeout=120)

markdown = page.text
# Split on headings, embed each chunk and store it: your pipeline, your tools.
chunks = markdown.split("\n## ")

curl https://api.spicrawl.com/v1/batch \
  -H "Authorization: Bearer $SPICRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "docs-refresh",
    "urls": ["https://example.com/docs/a", "https://example.com/docs/b"],
    "response_format": "markdown"
  }'

Keep your key in SPICRAWL_API_KEY.

Sources that need more

• JavaScript pages: set js_render: true. See scraping JavaScript websites.
• PDFs: parsed to plain text when you ask for markdown or text. Scanned or encrypted PDFs return no text.
• Pages behind a login: use a session, for accounts you own or may use.

Good to know

• No crawling. Spicrawl does not find URLs or follow links. Send the list, for example from a sitemap.
• Caching: results are cached for 48 hours by default. Set cache_ttl: 0 for a fresh fetch. A cache hit is billed like the fetch that stored it.
• Pricing: free during the beta. A plain page costs 1 credit, JavaScript 3, a full browser 8. Failed requests cost 0.