https://spicrawl.com/blog Guides, tutorials and comparisons for scraping the web with AI agents. en https://spicrawl.com/blog/cloudflare-blocking-ai-crawlers https://spicrawl.com/blog/cloudflare-blocking-ai-crawlers Mon, 05 Oct 2026 00:00:00 GMT Guides Cloudflare lets sites block Search, Agent and Training bots separately, with new defaults from 15 September 2026. What changed and how to scrape responsibly. https://spicrawl.com/blog/web-scraping-403-forbidden https://spicrawl.com/blog/web-scraping-403-forbidden Mon, 05 Oct 2026 00:00:00 GMT Tutorials Why a scraper gets a 403 Forbidden, how to tell the causes apart, and Python fixes that are fair to the site: headers, pacing, robots.txt and sessions. https://spicrawl.com/blog/llms-txt-explained https://spicrawl.com/blog/llms-txt-explained Mon, 05 Oct 2026 00:00:00 GMT Guides llms.txt is a plain file that points AI tools to your best pages. Learn the format, see a real example, and read what Google says about it. https://spicrawl.com/blog/best-web-scraping-tools-for-ai-agents https://spicrawl.com/blog/best-web-scraping-tools-for-ai-agents Thu, 01 Oct 2026 00:00:00 GMT Comparisons An honest comparison of 10 web scraping tools for AI agents, from scraping APIs and MCP servers to open-source crawlers and cloud browsers. https://spicrawl.com/blog/scrape-websites-with-claude-code https://spicrawl.com/blog/scrape-websites-with-claude-code Thu, 01 Oct 2026 00:00:00 GMT Tutorials Connect Claude Code to Spicrawl's hosted MCP server, read live pages as Markdown, and handle JavaScript, logins, costs and errors. https://spicrawl.com/blog/html-vs-markdown-for-llms https://spicrawl.com/blog/html-vs-markdown-for-llms Thu, 01 Oct 2026 00:00:00 GMT Guides We measured six public pages with tiktoken: HTML and Spicrawl Markdown token counts, the method, what conversion keeps and loses, and when HTML wins.