A native Rust port of Google's robots.txt parser and matcher C++ library.
A zero-copy robots.txt parser with SIMD-accelerated byte scanning.
This is a unofficial Ruby gem that provides a wrapper around [Google Robotstxt Parser C++ library]