A tool to auto-install npm packages that you have 'import'ed or 'require'd in your code FOR LINUX
Another JSON Schema Validator
Traverse JSON Schema passing each schema object to callback
return the github url from a package.json file
A set of efficient utilities that extend the use of JSON (streaming, estimate size, NDJSON/JSONL, etc.)
JSON.parse with bigints support
Scanner and parser for JSON with comments.
Programmatic API to update package.json
Easily read/write JSON files.
Parse and stringify JSON with comments. It will retain comments even after saved!
Load the local package.json from either src or dist folder
JSON for Humans
deterministic `JSON.stringify()` - a faster version of substack's json-stable-strigify without jsonify
Normalizes data that can be found in package.json files.
Language service for JSON
Another JSON Schema Validator
JSON parse & stringify that supports binary via bops & base64
Build and manage the fast-json-stringify instances for the fastify framework
Big-friendly JSON. Asynchronous streaming functions for large JSON data sets.
A convertor between XML text and Javascript object / JSON text.
Higher-level content negotiation
Sort an Object or package.json based on the well-known package.json keys
JSON viewer component for React focused on performance for large volume input while still providing few customiziation features
A system for sharing tool configurations between projects without duplicating config files.
Log2json lets you read, filter and send logs as JSON objects via Unix pipes. It is inspired by Logstash, and is meant to be compatible with it at the JSON event/record level so that it can easily work with Kibana. Reading logs is done via a shell script(eg, `tail`) running in its own process. You then configure(see the `syslog2json` or the `nginxlog2json` script for examples) and run your filters in Ruby using the `Log2Json` module and its contained helper classes. `Log2Json` reads from stdin the logs(one log record per line), parses the log lines into JSON records, and then serializes and writes the records to stdout, which then can be piped to another process for processing or sending it to somewhere else. Currently, Log2json ships with a `tail-log` script that can be run as the input process. It's the same as using the Linux `tail` utility with the `-v -F` options except that it also tracks the positions(as the numbers of lines read from the beginning of the files) in a few files in the file system so that if the input process is interrupted, it can continue reading from where it left off next time if the files had been followed. This feature is similar to the sincedb feature in Logstash's file input. Note: If you don't need the tracking feature(ie, you are fine with always tailling from the end of file with `-v -F -n0`), then you can just use the `tail` utility that comes with your Linux distribution.(Or more specifically, the `tail` from the GNU coreutils). Other versions of the `tail` utility may also work, but are not tested. The input protocol expected by Log2json is very simple and documented in the source code. ** The `tail-log` script uses a patched version of `tail` from the GNU coreutils package. A binary of the `tail` utility compiled for Ubuntu 12.04 LTS is included with the Log2json gem. If the binary doesn't work for your distribution, then you'll need to get GNU coreutils-8.13, apply the patch(it can be found in the src/ directory of the installed gem), and then replace the bin/tail binary in the directory of the installed gem with your version of the binary. ** P.S. If you know of a way to configure and compile ONLY the tail program in coreutils, please let me know! The reason I'm not building tail post gem installation is that it takes too long to configure && make because that actually builds every utilties in coreutils. For shipping logs to Redis, there's the `lines2redis` script that can be used as the output process in the pipe. For shipping logs from Redis to ElasticSearch, Log2json provides a `redis2es` script. Finally here's an example of Log2json in action: From a client machine: tail-log /var/log/{sys,mail}log /var/log/{kern,auth}.log | syslog2json | queue=jsonlogs \ flush_size=20 \ flush_interval=30 \ lines2redis host.to.redis.server 6379 0 # use redis DB 0 On the Redis server: redis_queue=jsonlogs redis2es host.to.es.server Resources that help writing log2json filters: - look at log2json.rb source and example filters - http://grokdebug.herokuapp.com/ - http://www.ruby-doc.org/stdlib-1.9.3/libdoc/date/rdoc/DateTime.html#method-i-strftime
Log2json lets you read, filter and send logs as JSON objects via Unix pipes. It is inspired by Logstash, and is meant to be compatible with it at the JSON event/record level so that it can easily work with Kibana. Reading logs is done via a shell script(eg, `tail`) running in its own process. You then configure(see the `syslog2json` or the `nginxlog2json` script for examples) and run your filters in Ruby using the `Log2Json` module and its contained helper classes. `Log2Json` reads from stdin the logs(one log record per line), parses the log lines into JSON records, and then serializes and writes the records to stdout, which then can be piped to another process for processing or sending it to somewhere else. Currently, Log2json ships with a `tail-log` script that can be run as the input process. It's the same as using the Linux `tail` utility with the `-v -F` options except that it also tracks the positions(as the numbers of lines read from the beginning of the files) in a few files in the file system so that if the input process is interrupted, it can continue reading from where it left off next time if the files had been followed. This feature is similar to the sincedb feature in Logstash's file input. Note: If you don't need the tracking feature(ie, you are fine with always tailling from the end of file with `-v -F -n0`), then you can just use the `tail` utility that comes with your Linux distribution.(Or more specifically, the `tail` from the GNU coreutils). Other versions of the `tail` utility may also work, but are not tested. The input protocol expected by Log2json is very simple and documented in the source code. ** The `tail-log` script uses a patched version of `tail` from the GNU coreutils package. A binary of the `tail` utility compiled for Ubuntu 12.04 LTS is included with the Log2json gem. If the binary doesn't work for your distribution, then you'll need to get GNU coreutils-8.13, apply the patch(it can be found in the src/ directory of the installed gem), and then replace the bin/tail binary in the directory of the installed gem with your version of the binary. ** P.S. If you know of a way to configure and compile ONLY the tail program in coreutils, please let me know! The reason I'm not building tail post gem installation is that it takes too long to configure && make because that actually builds every utilties in coreutils. For shipping logs to Redis, there's the `lines2redis` script that can be used as the output process in the pipe. For shipping logs from Redis to ElasticSearch, Log2json provides a `redis2es` script. Finally here's an example of Log2json in action: From a client machine: tail-log /var/log/{sys,mail}log /var/log/{kern,auth}.log | syslog2json | queue=jsonlogs \ flush_size=20 \ flush_interval=30 \ lines2redis host.to.redis.server 6379 0 # use redis DB 0 On the Redis server: redis_queue=jsonlogs redis2es host.to.es.server Resources that help writing log2json filters: - look at log2json.rb source and example filters - http://grokdebug.herokuapp.com/ - http://www.ruby-doc.org/stdlib-1.9.3/libdoc/date/rdoc/DateTime.html#method-i-strftime
$Id: README.txt 204 2010-11-30 02:20:04Z pwilkins $ sm-transcript reads results of SLS processing and produces transcripts for the SpokenMedia browser. For each file in the source folder whose extension matches the source type, a file of destination type is created in the destination folder. All of these parameters have default values. Note: Examples of the commands you enter in the terminal are for *nix. The command prompt in the examples is: felix$ <command line> If you are a Windows user, make the usual adjustments. Requirements: sm-transcript is written in Ruby and packaged as a RubyGem. Since Ruby is not a compiled language, you will need to have Ruby installed on your machine to run sm-transcript. You can determine if Ruby is installed by typing "ruby -v" at a terminal prompt. It should return the version of Ruby that is installed. If Ruby is not installed on your machine, navigate to http://www.ruby-lang.org/ and follow the installation instructions. sm-transcript was developed using Ruby 1.8. Other Ruby versions have not been tested as of this release. Installation: You can get sm-transcript as either a RubyGem or as source from svn. The preferred way to install this package is as a Rubygem. You can download and install the gem with this command: felix$ sudo gem install [--verbose] sm-transcript This command downloads the most recent version of the gem from rubygems.org and makes it active. Previous versions of the gem remain installed, but are deactivated. You must use "sudo" to properly install the gem. If you execute "gem install" (omitting the "sudo") the gem is installed in your home gem repository and it isn't in your path without additional configuration. Note: You need sudo privileges to run the command as written. If you can't sudo, then you can install it locally and will need some additional configuration. Contact me (or your local Ruby wizard) for assistance. The executable is now in your path. You can cleanly uninstall the gem with this command: felix$ sudo gem uninstall sm-transcript If you have access to our svn repository, you are welcome to check out the code. Be warned that the trunk tip is not necessarily stable. It changes frequently as enhancements (and bug fixes) are added. (note that the 'smb_transcript' in the command line below is not a typo.) svn co svn+ssh://svn.mit.edu/oeit-tsa/SMB/smb_transcript/trunk sm_transcript build the gem by running this command from the directory you installed the source. This is what it looks like on my machine: felix$ rake gem The gem will be built and put in ./pkg You can now use the gem installation instructions above. Using the App: Run with no command line parameters, the app reads *.wrd files out of ./results and writes *.t1.html files to ./transcripts. These directories are relative to where sm_transcript is called. Note: destination files are overwritten without a warning prompt. If you want to preserve an existing output file, rename it before running the app again. For example, run the app by navigating to the bin folder and enter projects/sm_transcript/bin felix$ sm_transcript This command run from this folder will read *.wrd files from bin/results and write *-t1.html to bin/transcripts. Usage: sm_transcript [options] --srcdir PATH Read files from this folder (Default: ./results) --destdir PATH Write files to this folder (Default: ./transcripts) --srctype wrd | seg | txt | ttml | srt Kind of file to process (Default: wrd) --desttype html | ttml | datajs | json Kind of file to output (Default: html) -h, --help Show this message There is a serious gotch'a in specifying the srctype parameter: it must match the case of the file extension that you're processing. This means that if the srt files that you are processing have the extension .SRT, then you must specify the srctype as "SRT". Pretty lame, I know. I will update the gem with a fix shortly. My apologies until then. Troubleshooting: sm-transcript requires additional gems to operate. The RubyGem installation should install dependencies automatically, but when it doesn't, you get an error that includes ... no such file to load -- builder (LoadError) in the first few lines when you run sm-transcript, the problem is a missing dependent gem. (the error above indicates that the Builder gem is missing.) Try installing the missing gem. For the error above, the command looks like this on my computer: felix$ sudo gem install builder See "Required Gems" below for more information. A warning message such as: "WARNING: Nokogiri was built against LibXML version 2.7.6, but has dynamically loaded 2.7.7"" may be safely ignored. If you continue to have trouble, feel free to contact me. Upgrading: You can easily upgrade by simply executing the same command you used to install the gem. Running install again will add the newer version and make it active. By default the most recent version is used, but older versions are still available, simply inactive. If are using svn, you should already know what to do. Required Gems: builder - create structured data, such as XML extensions - added for the 'require_relative' command. (To get this command in Ruby 1.8 you need to install this gem, for Ruby 1.9 the command is already part of the core.) htmlentities - html parsing json - create JSON structured data nokogiri - xml parsing library optparse - option parsing of command line ostruct - open data structures ppcommand - pp is a pretty printer. It is used only for debugging rake - make for Ruby rubygems - support for gems (shouldn't be needed for Ruby 1.9) shoulda - enhancement for Test::Unit This command installs gems on OSX and Linux: felix$ sudo gem install <gem name> I recommend running the following command to update to latest version of rubygems before loading new gems. felix$ sudo gem update --system Unit Tests: You may run all unit tests by navigating to the test folder and running rake with no parameters (the default rake task runs all tests). On my computer, it looks like this: projects/sm_transcript/test felix$ rake Release Notes: Initial Version - runs under Ruby 1.8.x. version 0.0.4 - fixes bug when processing .WRD files with CRLF line endings. version 0.0.5 - removed due to posting error version 0.0.6 - added srctype of ttml and desttype of json, fixed bug where beginning time of word was actually for previous word. version 0.0.7 - added srt as srctype version 0.0.8 - fixed bug that dropped last phrase from transcripts version 1.0.0 - declared this version 1.0.0 to conform more closely with gem numbering conventions. All tests run successfully. To Do: - specify individual files for processing rather than folders - fix bug in srt processing: can't read Creole srt content. - allow user to modify the "t1" file extension for addition languages of the same transcript. - update code to run under Ruby 1.9
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.
No description provided.