Simple web crawler with basic capability to crawl next page based on callback
React component to inject misleading meta tags and hidden instructions to deter AI/webcrawler indexing.
a minimal puppeteer crawler api
Crawler to get CVM info by company's CNPJ
Walk URLs
crawler request tool
simple analytics middleware for express
A standards-compliant generator for producing robots.txt files
A user agent parser that determines whether a request is from a robot, crawler, spider, etc.
Web crawler in JavaScript
Verify that a request is from Twitter crawlers using DNS verification steps
ESM module - crawls a website, validating that all the links on the site which point to the same orgin can be fetched.
A configuration - based crawler framework
A easy to use NodeJS http/https web-crawler.
A frontend test and audit framework. Extensible with WebPipes.
CLI toolkit that transforms SEO-optimized websites into AI-search-ready content
A fast crawler cli with pyppteer, this crawler can crawl SPA(single page application)
structure any website
Bloom filter implemenation for Node.js applications.
Crawler for the-frameworks
GraphQL based web scraper / spider
puppeteer crawler
Fast & Easy crawling framework deeply integrated with TypeORM. Built on Crawlee.
Official Orion bot tracking middleware for Next.js, Nuxt, and Node.js applications