a robotstxt parser for node.js
Awesome generator robots.txt
Fork from leecrossley / hexo-generator-robotstxt
A webpack plugin to output a robots.txt file
A lightweight, Open Source robots.txt parser written in JavaScript
Basic robots.txt generator plugin for Hexo
Pure JS robots.txt parser and guard built with TypeScript and ESM support (tree-shaking).
Google's robots.txt parser ported to TypeScript - RFC 9309 compliant
`sqlite-robotstxt` is distributed on `npm` for Node.js developers. To install on [supported platforms](#supported-platforms), simply run:
google/robotstxt library wasm warpper
Robots.txt bounce middleware for koa
RFC 9309 spec compliant robots.txt builder and parser. 🦾
An Eleventy plugin to generate a robots.txt file for your static site
Validate urls against robots.txt rules.
robots.txt middleware for Kerouac.
A plugin to output a robots.txt file with options per environment
generate robots.txt based on env content for static websites
MCP server for robots.txt parsing, analysis, and generation with AI bot detection
React Robots.txt file Generator
Gatsby plugin that automatically creates robots.txt for your site
Express middleware to handle different robots.txt responses on different domains
Vite plugin for generating robots.txt for production and development builds
<!--- Generated with the npm_generate_platform_packages.sh script, don't edit by hand -->
<!--- Generated with the npm_generate_platform_packages.sh script, don't edit by hand -->
A native Rust port of Google's robots.txt parser and matcher C++ library.
An intuitive Rust library for aquiring, parsing and querying robots.txt files.
A native Rust port of Google's robots.txt parser and matcher C++ library.
A zero-copy robots.txt parser with SIMD-accelerated byte scanning.
A SQLite extension for parsing robots.txt files
Easily crawl multiple sitemaps and list URLs
Async BFS web crawler with rate limiting and robots.txt support for CRW
A polite and user-friendly downloader for Common Crawl data.
A polite and user-friendly downloader for Common Crawl data.
A simple Gemini server that serves a read-only view of public repositories from a Git forge.
Robotstxt Parser allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap: rules.
Robotstxt-Parser allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap: rules.
Parse robots.txt files from Ruby.
Parse robots.txt files from Ruby.
Automatically generate a robots.txt file for your Jekyll site.
This is a unofficial Ruby gem that provides a wrapper around [Google Robotstxt Parser C++ library]
Enables fast, standards-compliant robots.txt parsing and URL access checking directly from Ruby.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.