BETAmodules.com is in beta — open to partnerships & joint ventures.Build with us
modules.com
⌘K

cobweb

v1.2.1RubyGems· Ruby

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

The verdict
Abandoned. Last published 5 years ago. No recent activity — look for a maintained alternative.
No recent activity — look for a maintained alternative.
Live from the RubyGems registry · derived rules, not AI
How it scores
MaintenanceAbandoned
PopularityNiche
SecurityClean
LicensePermissive
DepsZero deps
Maintenance
Last published 5 years ago.
Popularity
13 downloads / week
Security
No known advisories for this version (OSV).
License
MIT
Dependencies
No runtime dependencies
Recent releases
  • 1.2.15 years ago
  • 1.2.07 years ago
  • 1.1.010 years ago
  • 1.0.2910 years ago
  • 1.0.2811 years ago
  • 1.0.2711 years ago
  • 1.0.2611 years ago
  • 1.0.2511 years ago
cobweb — Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls. (Ruby / RubyGems) · Modules