Floodesh is a distributed web spider/crawler written with Nodejs.
daemon for floodesh/flowesh process manager
a cli tool for floodesh
core library for floodesh, which contains core request, response and context class, also used for flowesh
mof is short for Middleware of Floodesh
A middleware wrapped Bottleneckp module for floodesh
cookie obtain middleware for floodesh
proxy middleware for floodesh
A middleware wrapped request.js module for floodesh
cheerio wrapper for middleware of floodesh
A middleware wrapper of JSON parsing for floodesh
the record checker for middleware of floodesh
seenreq wrapper for middleware of floodesh
the response body checker for middleware of floodesh
middleware of floodesh, which rotates User-Agent automatically
middleware of floodesh, which normalizes queue options
A middleware wrapped statsd client module for floodesh
Flowesh is the non-cluster version of floodesh. It's a middleware based web spider which is lightweight and easy to maintain
middleware of floodesh, which provides a statsd client that allows you to send data to statsd server and automatically has some statistics sent that generally apply to all requests.
A middleware wrapped whacko module for floodesh
the charset parser for middleware of floodesh
middleware of floodesh, prints gene and url of a task, along with # of new tasks and # of records
middleware of floodesh, which adapts queued options to request options
Convert the response body to String from one encoding to another encoding