A specification compliant robots.txt parser with wildcard (*) matching support.
Tame the robots crawling and indexing your Nuxt site with ease.
Gatsby plugin that automatically creates robots.txt for your site
Generate a robots.txt for Astro
Express middleware to serve and generate robots.txt
A simple, accessible, tagging component ready to drop into your React projects.
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
Validate urls against robots.txt rules.
Get the name of a dependency from a module file path.
Streaming parser for robots.txt files
parse arguments with recursive contexts
jquery plugin for sortable, nestable lists
parse arguments with recursive contexts
TypeScript definitions for react-tag-autocomplete
A reliable robots.txt generator for Astro projects, offering zero-config setup and Verified Bots support.
Vite plugin for generating robots.txt for production and development builds
EasyZoom is an elegant, highly optimised jQuery image zoom and panning plugin based on the original work by Alen Grakalic.
Express middleware for blocking ai-robots
HTML assertions for Chai.
Parse robot directives within HTML meta and/or HTTP headers.
Hapi plugin to serve up robots.txt
A simple rate limited queue for asynchronous operations. Restricts the number of operations executed per time interval.
Vite plugin to generate robots.txt. Supports AI blocking and works in dev mode.
Hardened Fetch is a tiny wrapper for `global.fetch` adding request timeouts, throttling, rate limit awareness, retries, and descriptive errors. It makes working with APIs without SDKs and building robust web scrapers easier.