A short summary of what your crate does
Rust Web Crawler
A simple Rust web crawler
a simple web crawler in rust
A fast async Rust crawler that discovers and filters URLs by pattern without scraping content.
A lightweight async Web crawler in Rust, optimized for concurrent scraping while respecting `robots.txt` rules.
Concert web crawler in Rust
Common things i use to build Rust CLI tools for web crawlers.
A web crawler framework for Rust
Core functionality for the spider-lib web scraping framework.
A Rust-based web scraping framework inspired by Scrapy (Python).
Finds every page, image, and script on a website (and downloads it)