Crawler for the-frameworks
Async and sync crawler for json object
Agnostic tree traversal library.
The fastest directory crawler & globbing alternative to glob, fast-glob, & tiny-glob. Crawls 1m files in < 1s
A specification compliant robots.txt parser with wildcard (*) matching support.
Official Firecrawl nodes for n8n - scrape, crawl, map, search, and extract data from websites. Supports AI Agent tool usage.
JavaScript SDK for Firecrawl API
A community node for n8n to integrate Tavily API for web search and content extraction.
JavaScript SDK for Firecrawl API
Schema-driven relayfile adapter generator and runtime
Mdream Crawl generates comprehensive llms.txt artifacts from a single URL, using mdream to convert HTML to Markdown.
CLI tool for Genspark Tool API - search, crawl, analyze images, generate media
Extra Cypress query commands for v12+
[](https://www.npmjs.com/package/recrawl) [](https://github.com/aleclarson/recrawl/actions/workflows/release.yml) [![codeco
Cypress command for flexible test data setup
Shared types for the crawlee projects
FireCrawl nodes for n8n
Agent-first open-source AEO operating platform - track how answer engines cite your domain
A set of shared utilities that can be used by crawlers
No description provided.
The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.
A simple in-memory storage implementation of the Apify API
Rotate multiple browsers using popular automation libraries such as Playwright or Puppeteer.
The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.
Ruby based client for the ProxyCrawl API that helps developers crawl or scrape thousands of web pages anonymously
Crawls Twitter
Crawl websites
Crawling framework
Ruby utilities for web crawling.
Fassbinder crawls book offers on Amazon.
Easilly crawl a website
Crawls public LinkedIn profiles via Google
Crawls Indeed resumes
The SimpleCrawler module is a library for crawling web sites. The crawler provides comprehensive data from the page crawled which can be used for page analysis, indexing, accessibility checks etc. Restrictions can be specified to limit crawling of binary files.
Web crawling framework based on ActiveJob
Vessel is a high-level web crawling framework, used to crawl websites and extract structured data from their pages