BETAmodules.com is in beta — open to partnerships & joint ventures.Build with us
modules.com
⌘K

crawlr

v0.2.3RubyGems· Ruby

Crawlr is a modern Ruby web scraping framework built for respectful and efficient data extraction. Features async HTTP requests, robots.txt compliance, cookie management, proxy rotation, flexible CSS/XPath selectors, and comprehensive error handling. Designed for both simple scraping tasks and complex, large-scale data extraction projects.

The verdict
Aging — last published 8 months ago — check before adopting. Check the repo for activity before adopting.
Check the repo for activity before adopting.
Live from the RubyGems registry · derived rules, not AI
How it scores
MaintenanceAging
PopularityNiche
SecurityClean
LicensePermissive
DepsZero deps
Maintenance
Last published 8 months ago — check before adopting.
Popularity
11 downloads / week
Security
No known advisories for this version (OSV).
License
MIT
Dependencies
No runtime dependencies
Recent releases
  • 0.2.38 months ago
  • 0.2.28 months ago
  • 0.2.18 months ago
  • 0.2.08 months ago
  • 0.1.08 months ago
crawlr — Crawlr is a modern Ruby web scraping framework built for respectful and efficient data extraction. Features async HTTP requests, robots.txt compliance, cookie management, proxy rotation, flexible CSS/XPath selectors, and comprehensive error handling. Designed for both simple scraping tasks and complex, large-scale data extraction projects. (Ruby / RubyGems) · Modules