simple polite crawling of the web.
transform stream for mrspider
mrspider jsdom middleware
cheerio middleware for mrspider
mrspider middleware mongodb persistence
data extractor middleware for mrspider
mrspider middleware to validate data that has been extracted.
mrspider middleware to extract data using regular expressions.
middleware for mrspider to extract links using css rules
mrspider middleware to extract image urls from a webpage.