crawler.sh

crawler.sh is a desktop crawler that converts websites into clean Markdown for RAG pipelines, fine-tuning datasets and agent context. It runs on your own machine with no per-page fees and renders JavaScript-heavy sites without headless Chrome.

Each page is extracted with word count, author, language and an excerpt, and a whole site can be exported as a Markdown archive. A custom rendering engine handles React, Vue and Next.js apps and keeps cookies so pages behind a session render correctly.

It respects robots.txt rules and slows down automatically when sites push back. The same crawl also runs SEO checks for missing titles, duplicate descriptions, broken links and thin content, with CSV or text export.