Normalizes the API for cookies, and localStorage/sessionStorage
Express-based webserver utilities with proxy, storage, and security helpers.
proxy storage
An HTTP(s) proxy `http.Agent` implementation for HTTP
Offers getProxyForUrl to get the proxy URL for a URL, respecting the *_PROXY (e.g. HTTP_PROXY) and NO_PROXY environment variables.
Maps proxy protocols to `http.Agent` implementations
Turn a function into an `http.Agent` instance
An HTTP(s) proxy `http.Agent` implementation for HTTPS
Determine address of proxied request
The one-liner node.js proxy middleware for connect, express, next.js and more
A SOCKS proxy `http.Agent` implementation for HTTP and HTTPS
HTTP proxying for the masses
A PAC file proxy `http.Agent` implementation for HTTP
ES5 shim for ES6 (ECMAScript 6) Reflect and Proxy objects
Global HTTP/HTTPS proxy configurable using environment variables.
A continuation-local storage module compatible with NestJS's dependency injection.
TypeScript definitions for http-proxy
Google APIs Client Library for Node.js
Get configured proxy
Compare two objects using accessed properties with Proxy
A path proxy object constructor
Proxies React components without unmounting or losing their state.
Determine the address of a proxied request
A TypeScript SSE proxy for MCP servers that use stdio transport.
Cloud storage to image gallery + image proxy + file api
Proxie is a HTTP proxy server with sqlite-powered storage and web interface for debugging.
== Medusa: a ruby crawler framework {rdoc-image:https://badge.fury.io/rb/medusa-crawler.svg}[https://rubygems.org/gems/medusa-crawler] rdoc-image:https://github.com/brutuscat/medusa-crawler/workflows/Ruby/badge.svg?event=push Medusa is a framework for the ruby language to crawl and collect useful information about the pages it visits. It is versatile, allowing you to write your own specialized tasks quickly and easily. === Features * Choose the links to follow on each page with +focus_crawl+ * Multi-threaded design for high performance * Tracks +301+ HTTP redirects * Allows exclusion of URLs based on regular expressions * Records response time for each page * Obey _robots.txt_ directives (optional, but recommended) * In-memory or persistent storage of pages during crawl, provided by Moneta[https://github.com/moneta-rb/moneta] * Inherits OpenURI behavior (redirects, automatic charset and encoding detection, proxy configuration options). <b>Do you have an idea or a suggestion? {Open an issue and talk about it}[https://github.com/brutuscat/medusa-crawler/issues/new]</b> === Examples Medusa is versatile and to be used programatically, you can start with one or multiple URIs: require 'medusa' Medusa.crawl('https://www.example.com', depth_limit: 2) Or you can pass a block and it will yield the crawler back, to manage configuration or drive its crawling focus: require 'medusa' Medusa.crawl('https://www.example.com', depth_limit: 2) do |crawler| crawler.discard_page_bodies = some_flag # Persist all the pages state across crawl-runs. crawler.clear_on_startup = false crawler.storage = Medusa::Storage.Moneta(:Redis, 'redis://redis.host.name:6379/0') crawler.skip_links_like(/private/) crawler.on_pages_like(/public/) do |page| logger.debug "[public page] #{page.url} took #{page.response_time} found #{page.links.count}" end # Use an arbitrary logic, page by page, to continue customize the crawling. crawler.focus_crawl(/public/) do |page| page.links.first end end
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.