http pipe proxy
minimal implementation of a PassThrough stream
pipe streams together and close all of them if one of them closes
A small fast zlib stream built on [minipass](http://npm.im/minipass) and Node.js's zlib binding.
Gulp plugin for mapping sources of a sourcemap.
A PAC file proxy `http.Agent` implementation for HTTP
Gulp plugin for generating an identity sourcemap for a file.
Transform stream that gunzips its input if it is gzipped and just echoes it if not
Use websockets with the node streams API. Works in browser and node, with all current WS versions
An HTTP(s) proxy `http.Agent` implementation for HTTP
Low-Level COPY TO and COPY FROM streams for PostgreSQL in JavaScript using
Filesystem-buffered, passthrough stream that buffers indefinitely rather than propagate backpressure from downstream consumers.
Like JSONStream, but using Minipass streams
An HTTP(s) proxy `http.Agent` implementation for HTTPS
Fast and powerful CSV parser for the browser that supports web workers and streaming large files. Converts CSV to JSON and JSON to CSV.
Library to make working with Iterators/AsyncIterators easier.
An iteration of the Node.js core streams with a series of improvements
a streaming CRC32 checksumer
Rename files
This package provides utilities for encoding and formatting URL query parameters.
Compose promise-returning & async functions into a reusable pipeline
pipe streams with centralized error handling
Returns a `stream.Readable` from a URI string
A response-like object for mocking a Node.js HTTP response stream
# Rack HTTP Pipe Use to pipe directly a remote HTTP file without buffering it. > /!\ Do not work with WebBrick, tested with puma ## Use case * Given a file named #HASH#.pdf on S3 * You want a clean URL and handling the authentication in front of it ``` GET http:/example.com/download Content-Disposition: attachment;filename=name-fetched-from-db.pdf Content-Length Content-Type etc. ``` ## Usage ```ruby get "/" do http_pipe "http://example.com/iso-ubuntu-1404-64bits", { status: 200, headers: { "Content-Type: application/octet-stream", "Content-Disposition: attachment;filename=ubuntu.iso", } } end ``` See the example directory for an example app using sinatra
This gem provides a base class and helpers for writing remotebackend servers for pipe/unix/. It is intended to make using remotebackend easier. For http support, see pdns-remotebackend-http.
scbi_blast is a ruby gem to handle blast+ executions using pipes when possible to read data without the need of temporary files, it has been developed at [SCBI](http://www.scbi.uma.es) by Almudena Bocinos & Dario Guerrero.
A small library dealing with the obscurities of shell commands, piping and timeouts. Most of the stuff comes from bad experience in http://beanstalkapp.com.
A Magic Pipe to send data in arbitrary formats to configurable backends, with topics. It features a modular design that allows to configure and extend how the payloads are fetched, serialized, encoded, submitted, and most importantly where: SQS, HTTPS endpoints, etc.
Prepare to pack or unpack piles of files with the pastiepacker. To pack a folder: pastiepacker To pack some files ending with "txt": find * | grep "txt$" | pastiepacker - It outputs the url of the prepared pastie, so you can pipe it to xargs: - pastiepacker | xargs open To unpack a packed pastie: pastiepacker http://pastie.caboo.se/175886 - This unpacks the files into a subfolder 175886/
The splice family of Linux system calls can transfer data between file descriptors without the need to copy data into userspace. Instead of a userspace buffer, they rely on an ordinary Unix pipe as a kernel-level buffer. The `splice' and `tee' syscalls are also provided by the {sleepy_penguin}[https://yhbt.net/sleepy_penguin/] library. "io_splice" remains maintained for old applications or users experimenting with the vmsplice syscalls
This plugin get the feed rss content (being able to use keywords to get the feed) , the params are: 1) multi_feed => [array] URI parent with more rss links inside , something like this: http://rss.elmundo.es/rss/ 2) one_feed => [array] (optionally) childs URIS with XML content inside , something like this: http://estaticos.elmundo.es/elmundo/rss/portada.xml 3) blacklist => [array] (optionally) strings , links, text ... what you dont want explored 4) Interval => [int] Set the Stoppable_sleep interval for the pipe 5) keywords => [array] if you use this parameter will only compile those news that contain in any of its attributes a word from this array
Mingle 3.3 introduced a new Events API in the form of an "Atom feed":http://www.thoughtworks-studios.com/mingle/3.3/help/mingle_api_events.html. The Mingle team and ThoughtWorks Studios are big believers in the use of Atom for exposing events. Atom is a widely used standard, and this event API style puts the issue of robust event delivery in the hands of the consumer, where it belongs. In fact, we'd argue this is the only feasible means of robust, scalable event delivery, short of spending hundreds of thousands or millions of dollars on enterprise buses and such. Atom-delivered events are cheap, scalable, standards-based, and robust. However, we do accept that asking integrators wishing to consume events to implement polling is not ideal. Writing polling consumers can be tedious. And this tedium gets in the way of writing sweet Mingle integrations. We are addressing this by publishing libraries such as this, which if effective, fully hide the mechanics of event polling from the consumer. The consumer only need worry about the processing of events. Said processing is modeled in the style of 'pipes and filters.'
Log2json lets you read, filter and send logs as JSON objects via Unix pipes. It is inspired by Logstash, and is meant to be compatible with it at the JSON event/record level so that it can easily work with Kibana. Reading logs is done via a shell script(eg, `tail`) running in its own process. You then configure(see the `syslog2json` or the `nginxlog2json` script for examples) and run your filters in Ruby using the `Log2Json` module and its contained helper classes. `Log2Json` reads from stdin the logs(one log record per line), parses the log lines into JSON records, and then serializes and writes the records to stdout, which then can be piped to another process for processing or sending it to somewhere else. Currently, Log2json ships with a `tail-log` script that can be run as the input process. It's the same as using the Linux `tail` utility with the `-v -F` options except that it also tracks the positions(as the numbers of lines read from the beginning of the files) in a few files in the file system so that if the input process is interrupted, it can continue reading from where it left off next time if the files had been followed. This feature is similar to the sincedb feature in Logstash's file input. Note: If you don't need the tracking feature(ie, you are fine with always tailling from the end of file with `-v -F -n0`), then you can just use the `tail` utility that comes with your Linux distribution.(Or more specifically, the `tail` from the GNU coreutils). Other versions of the `tail` utility may also work, but are not tested. The input protocol expected by Log2json is very simple and documented in the source code. ** The `tail-log` script uses a patched version of `tail` from the GNU coreutils package. A binary of the `tail` utility compiled for Ubuntu 12.04 LTS is included with the Log2json gem. If the binary doesn't work for your distribution, then you'll need to get GNU coreutils-8.13, apply the patch(it can be found in the src/ directory of the installed gem), and then replace the bin/tail binary in the directory of the installed gem with your version of the binary. ** P.S. If you know of a way to configure and compile ONLY the tail program in coreutils, please let me know! The reason I'm not building tail post gem installation is that it takes too long to configure && make because that actually builds every utilties in coreutils. For shipping logs to Redis, there's the `lines2redis` script that can be used as the output process in the pipe. For shipping logs from Redis to ElasticSearch, Log2json provides a `redis2es` script. Finally here's an example of Log2json in action: From a client machine: tail-log /var/log/{sys,mail}log /var/log/{kern,auth}.log | syslog2json | queue=jsonlogs \ flush_size=20 \ flush_interval=30 \ lines2redis host.to.redis.server 6379 0 # use redis DB 0 On the Redis server: redis_queue=jsonlogs redis2es host.to.es.server Resources that help writing log2json filters: - look at log2json.rb source and example filters - http://grokdebug.herokuapp.com/ - http://www.ruby-doc.org/stdlib-1.9.3/libdoc/date/rdoc/DateTime.html#method-i-strftime
Log2json lets you read, filter and send logs as JSON objects via Unix pipes. It is inspired by Logstash, and is meant to be compatible with it at the JSON event/record level so that it can easily work with Kibana. Reading logs is done via a shell script(eg, `tail`) running in its own process. You then configure(see the `syslog2json` or the `nginxlog2json` script for examples) and run your filters in Ruby using the `Log2Json` module and its contained helper classes. `Log2Json` reads from stdin the logs(one log record per line), parses the log lines into JSON records, and then serializes and writes the records to stdout, which then can be piped to another process for processing or sending it to somewhere else. Currently, Log2json ships with a `tail-log` script that can be run as the input process. It's the same as using the Linux `tail` utility with the `-v -F` options except that it also tracks the positions(as the numbers of lines read from the beginning of the files) in a few files in the file system so that if the input process is interrupted, it can continue reading from where it left off next time if the files had been followed. This feature is similar to the sincedb feature in Logstash's file input. Note: If you don't need the tracking feature(ie, you are fine with always tailling from the end of file with `-v -F -n0`), then you can just use the `tail` utility that comes with your Linux distribution.(Or more specifically, the `tail` from the GNU coreutils). Other versions of the `tail` utility may also work, but are not tested. The input protocol expected by Log2json is very simple and documented in the source code. ** The `tail-log` script uses a patched version of `tail` from the GNU coreutils package. A binary of the `tail` utility compiled for Ubuntu 12.04 LTS is included with the Log2json gem. If the binary doesn't work for your distribution, then you'll need to get GNU coreutils-8.13, apply the patch(it can be found in the src/ directory of the installed gem), and then replace the bin/tail binary in the directory of the installed gem with your version of the binary. ** P.S. If you know of a way to configure and compile ONLY the tail program in coreutils, please let me know! The reason I'm not building tail post gem installation is that it takes too long to configure && make because that actually builds every utilties in coreutils. For shipping logs to Redis, there's the `lines2redis` script that can be used as the output process in the pipe. For shipping logs from Redis to ElasticSearch, Log2json provides a `redis2es` script. Finally here's an example of Log2json in action: From a client machine: tail-log /var/log/{sys,mail}log /var/log/{kern,auth}.log | syslog2json | queue=jsonlogs \ flush_size=20 \ flush_interval=30 \ lines2redis host.to.redis.server 6379 0 # use redis DB 0 On the Redis server: redis_queue=jsonlogs redis2es host.to.es.server Resources that help writing log2json filters: - look at log2json.rb source and example filters - http://grokdebug.herokuapp.com/ - http://www.ruby-doc.org/stdlib-1.9.3/libdoc/date/rdoc/DateTime.html#method-i-strftime
# Single File Delivery Method for Mail gem ## Summary This gem is a delivery-method plug-in for [mail](https://github.com/mikel/mail) that delivers all mail to a single file for testing. The Mail gem already provides a file delivery-method that appends a copy of each message to a file named after each message recipient, but I want them to all go to a single file so that I can monitor them from another window with `tail -f my-file`, or `cat my-named-pipe` while I hand-test the web interface from a browser. Of course this is _in addition to_ running automated tests with Rspec and Cucumber. At some point in development, I want to actually see the pages and enter my own inputs and perhaps display the mail messages in an HTML reader. ## Synopsis Mail.defaults do delivery_method SingleFileDelivery => '/tmp/my-file.txt' end
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.