Results for loading-indicator

Provides binary and rake tasks to dump, load and optionally rename indices. Implements live-reindex with hot-swap of old code/index with new code/index.

Provides binary and rake tasks to dump, load and optionally rename indices. Implements live-reindex with hot-swap of old code/index with new code/index.

A simple loading indicator to show you're doing ajax requests

A Sass mixin and Compass extension to generate pure CSS3 loading/busy indicators

Pathological provides a way to manage a project's require paths by using a small config file that indicates all directories to include in the load path.

Automatic page load progress bar

Extension of Pathological Gem to support Ruby > 1.9 version indicates all directories to include in the load path.

This gem provides a set of Backbone.js extensions commonly used in Coroutine projects. These extensions include simple collection views, paginated collection views, searching, and loading indicators.

This is a framework for creating handlers for serf, by Hashicorp. The handlers are modular, loading themselves according to a configuration file that indicates what commands are supported, and where to load the code for those commands. The framework will also support executing the handler on the command line, with the serf payload provided on the command line. This is useful for testing.

README ====== This is a simple API to evaluate information retrieval results. It allows you to load ranked and unranked query results and calculate various evaluation metrics (precision, recall, MAP, kappa) against a previously loaded gold standard. Start this program from the command line with: retreval -l <gold-standard-file> -q <query-results> -f <format> -o <output-prefix> The options are outlined when you pass no arguments and just call retreval You will find further information in the RDOC documentation and the HOWTO section below. If you want to see an example, use this command: retreval -l example/gold_standard.yml -q example/query_results.yml -f yaml -v INSTALLATION ============ If you have RubyGems, just run gem install retreval You can manually download the sources and build the Gem from there by `cd`ing to the folder where this README is saved and calling gem build retreval.gemspec This will create a gem file called which you just have to install with `gem install <file>` and you're done. HOWTO ===== This API supports the following evaluation tasks: - Loading a Gold Standard that takes a set of documents, queries and corresponding judgements of relevancy (i.e. "Is this document relevant for this query?") - Calculation of the _kappa measure_ for the given gold standard - Loading ranked or unranked query results for a certain query - Calculation of _precision_ and _recall_ for each result - Calculation of the _F-measure_ for weighing precision and recall - Calculation of _mean average precision_ for multiple query results - Calculation of the _11-point precision_ and _average precision_ for ranked query results - Printing of summary tables and results Typically, you will want to use this Gem either standalone or within another application's context. Standalone Usage ================ Call parameters --------------- After installing the Gem (see INSTALLATION), you can always call `retreval` from the commandline. The typical call is: retreval -l <gold-standard-file> -q <query-results> -f <format> -o <output-prefix> Where you have to define the following options: - `gold-standard-file` is a file in a specified format that includes all the judgements - `query-results` is a file in a specified format that includes all the query results in a single file - `format` is the format that the files will use (either "yaml" or "plain") - `output-prefix` is the prefix of output files that will be created Formats ------- Right now, we focus on the formats you can use to load data into the API. Currently, we support YAML files that must adhere to a special syntax. So, in order to load a gold standard, we need a file in the following format: * "query" denotes the query * "documents" these are the documents judged for this query * "id" the ID of the document (e.g. its filename, etc.) * "judgements" an array of judgements, each one with: * "relevant" a boolean value of the judgment (relevant or not) * "user" an optional identifier of the user Example file, with one query, two documents, and one judgement: - query: 12th air force germany 1957 documents: - id: g5701s.ict21311 judgements: [] - id: g5701s.ict21313 judgements: - relevant: false user: 2 So, when calling the program, specify the format as `yaml`. For the query results, a similar format is used. Note that it is necessary to specify whether the result sets are ranked or not, as this will heavily influence the calculations. You can specify the score for a document. By "score" we mean the score that your retrieval algorithm has given the document. But this is not necessary. The documents will always be ranked in the order of their appearance, regardless of their score. Thus in the following example, the document with "07" at the end is the first and "25" is the last, regardless of the score. --- query: 12th air force germany 1957 ranked: true documents: - score: 0.44034874 document: g5701s.ict21307 - score: 0.44034874 document: g5701s.ict21309 - score: 0.44034874 document: g5701s.ict21311 - score: 0.44034874 document: g5701s.ict21313 - score: 0.44034874 document: g5701s.ict21315 - score: 0.44034874 document: g5701s.ict21317 - score: 0.44034874 document: g5701s.ict21319 - score: 0.44034874 document: g5701s.ict21321 - score: 0.44034874 document: g5701s.ict21323 - score: 0.44034874 document: g5701s.ict21325 --- query: 1612 ranked: true documents: - score: 1.0174774 document: g3290.np000144 - score: 0.763108 document: g3201b.ct000726 - score: 0.763108 document: g3400.ct000886 - score: 0.6359234 document: g3201s.ct000130 --- **Note**: You can also use the `plain` format, which will load the gold standard in a different way (but not the results): my_query my_document_1 false my_query my_document_2 true See that every query/document/relevancy pair is separated by a tabulator? You can also add the user's ID in the fourth column if necessary. Running the evaluation ----------------------- After you have specified the input files and the format, you can run the program. If needed, the `-v` switch will turn on verbose messages, such as information on how many judgements, documents and users there are, but this shouldn't be necessary. The program will first load the gold standard and then calculate the statistics for each result set. The output files are automatically created and contain a YAML representation of the results. Calculations may take a while depending on the amount of judgements and documents. If there are a thousand judgements, always consider a few seconds for each result set. Interpreting the output files ------------------------------ Two output files will be created: - `output_avg_precision.yml` - `output_statistics.yml` The first lists the average precision for each query in the query result file. The second file lists all supported statistics for each query in the query results file. For example, for a ranked evaluation, the first two entries of such a query result statistic look like this: --- 12th air force germany 1957: - :precision: 0.0 :recall: 0.0 :false_negatives: 1 :false_positives: 1 :true_negatives: 2516 :true_positives: 0 :document: g5701s.ict21313 :relevant: false - :precision: 0.0 :recall: 0.0 :false_negatives: 1 :false_positives: 2 :true_negatives: 2515 :true_positives: 0 :document: g5701s.ict21317 :relevant: false You can see the precision and recall for that specific point and also the number of documents for the contingency table (true/false positives/negatives). Also, the document identifier is given. API Usage ========= Using this API in another ruby application is probably the more common use case. All you have to do is include the Gem in your Ruby or Ruby on Rails application. For details about available methods, please refer to the API documentation generated by RDoc. **Important**: For this implementation, we use the document ID, the query and the user ID as the primary keys for matching objects. This means that your documents and queries are identified by a string and thus the strings should be sanitized first. Loading the Gold Standard ------------------------- Once you have loaded the Gem, you will probably start by creating a new gold standard. gold_standard = GoldStandard.new Then, you can load judgements into this standard, either from a file, or manually: gold_standard.load_from_yaml_file "my-file.yml" gold_standard.add_judgement :document => doc_id, :query => query_string, :relevant => boolean, :user => John There is a nice shortcut for the `add_judgement` method. Both lines are essentially the same: gold_standard.add_judgement :document => doc_id, :query => query_string, :relevant => boolean, :user => John gold_standard << :document => doc_id, :query => query_string, :relevant => boolean, :user => John Note the usage of typical Rails hashes for better readability (also, this Gem was developed to be used in a Rails webapp). Now that you have loaded the gold standard, you can do things like: gold_standard.contains_judgement? :document => "a document", :query => "the query" gold_standard.relevant? :document => "a document", :query => "the query" Loading the Query Results ------------------------- Now we want to create a new `QueryResultSet`. A query result set can contain more than one result, which is what we normally want. It is important that you specify the gold standard it belongs to. query_result_set = QueryResultSet.new :gold_standard => gold_standard Just like the Gold Standard, you can read a query result set from a file: query_result_set.load_from_yaml_file "my-results-file.yml" Alternatively, you can load the query results one by one. To do this, you have to create the results (either ranked or unranked) and then add documents: my_result = RankedQueryResult.new :query => "the query" my_result.add_document :document => "test_document 1", :score => 13 my_result.add_document :document => "test_document 2", :score => 11 my_result.add_document :document => "test_document 3", :score => 3 This result would be ranked, obviously, and contain three documents. Documents can have a score, but this is optional. You can also create an Array of documents first and add them altogether: documents = Array.new documents << ResultDocument.new :id => "test_document 1", :score => 20 documents << ResultDocument.new :id => "test_document 2", :score => 21 my_result = RankedQueryResult.new :query => "the query", :documents => documents The same applies to `UnrankedQueryResult`s, obviously. The order of ranked documents is the same as the order in which they were added to the result. The `QueryResultSet` will now contain all the results. They are stored in an array called `query_results`, which you can access. So, to iterate over each result, you might want to use the following code: query_result_set.query_results.each_with_index do |result, index| # ... end Or, more simply: for result in query_result_set.query_results # ... end Calculating statistics ---------------------- Now to the interesting part: Calculating statistics. As mentioned before, there is a conceptual difference between ranked and unranked results. Unranked results are much easier to calculate and thus take less CPU time. No matter if unranked or ranked, you can get the most important statistics by just calling the `statistics` method. statistics = my_result.statistics In the simple case of an unranked result, you will receive a hash with the following information: * `precision` - the precision of the results * `recall` - the recall of the results * `false_negatives` - number of not retrieved but relevant items * `false_positives` - number of retrieved but nonrelevant * `true_negatives` - number of not retrieved and nonrelevantv items * `true_positives` - number of retrieved and relevant items In case of a ranked result, you will receive an Array that consists of _n_ such Hashes, depending on the number of documents. Each Hash will give you the information at a certain rank, e.g. the following to lines return the recall at the fourth rank. statistics = my_ranked_result.statistics statistics[3][:recall] In addition to the information mentioned above, you can also get for each rank: * `document` - the ID of the document that was returned at this rank * `relevant` - whether the document was relevant or not Calculating statistics with missing judgements ---------------------------------------------- Sometimes, you don't have judgements for all document/query pairs in the gold standard. If this happens, the results will be cleaned up first. This means that every document in the results that doesn't appear to have a judgement will be removed temporarily. As an example, take the following results: * A * B * C * D Our gold standard only contains judgements for A and C. The results will be cleaned up first, thus leading to: * A * C With this approach, we can still provide meaningful results (for precision and recall). Other statistics ---------------- There are several other statistics that can be calculated, for example the **F measure**. The F measure weighs precision and recall and has one parameter, either "alpha" or "beta". Get the F measure like so: my_result.f_measure :beta => 1 If you don't specify either alpha or beta, we will assume that beta = 1. Another interesting measure is **Cohen's Kappa**, which tells us about the inter-agreement of assessors. Get the kappa statistic like this: gold_standard.kappa This will calculate the average kappa for each pairwise combination of users in the gold standard. For ranked results one might also want to calculate an **11-point precision**. Just call the following: my_ranked_result.eleven_point_precision This will return a Hash that has indices at the 11 recall levels from 0 to 1 (with steps of 0.1) and the corresponding precision at that recall level.

$Id: README.txt 204 2010-11-30 02:20:04Z pwilkins $ sm-transcript reads results of SLS processing and produces transcripts for the SpokenMedia browser. For each file in the source folder whose extension matches the source type, a file of destination type is created in the destination folder. All of these parameters have default values. Note: Examples of the commands you enter in the terminal are for *nix. The command prompt in the examples is: felix$ <command line> If you are a Windows user, make the usual adjustments. Requirements: sm-transcript is written in Ruby and packaged as a RubyGem. Since Ruby is not a compiled language, you will need to have Ruby installed on your machine to run sm-transcript. You can determine if Ruby is installed by typing "ruby -v" at a terminal prompt. It should return the version of Ruby that is installed. If Ruby is not installed on your machine, navigate to http://www.ruby-lang.org/ and follow the installation instructions. sm-transcript was developed using Ruby 1.8. Other Ruby versions have not been tested as of this release. Installation: You can get sm-transcript as either a RubyGem or as source from svn. The preferred way to install this package is as a Rubygem. You can download and install the gem with this command: felix$ sudo gem install [--verbose] sm-transcript This command downloads the most recent version of the gem from rubygems.org and makes it active. Previous versions of the gem remain installed, but are deactivated. You must use "sudo" to properly install the gem. If you execute "gem install" (omitting the "sudo") the gem is installed in your home gem repository and it isn't in your path without additional configuration. Note: You need sudo privileges to run the command as written. If you can't sudo, then you can install it locally and will need some additional configuration. Contact me (or your local Ruby wizard) for assistance. The executable is now in your path. You can cleanly uninstall the gem with this command: felix$ sudo gem uninstall sm-transcript If you have access to our svn repository, you are welcome to check out the code. Be warned that the trunk tip is not necessarily stable. It changes frequently as enhancements (and bug fixes) are added. (note that the 'smb_transcript' in the command line below is not a typo.) svn co svn+ssh://svn.mit.edu/oeit-tsa/SMB/smb_transcript/trunk sm_transcript build the gem by running this command from the directory you installed the source. This is what it looks like on my machine: felix$ rake gem The gem will be built and put in ./pkg You can now use the gem installation instructions above. Using the App: Run with no command line parameters, the app reads *.wrd files out of ./results and writes *.t1.html files to ./transcripts. These directories are relative to where sm_transcript is called. Note: destination files are overwritten without a warning prompt. If you want to preserve an existing output file, rename it before running the app again. For example, run the app by navigating to the bin folder and enter projects/sm_transcript/bin felix$ sm_transcript This command run from this folder will read *.wrd files from bin/results and write *-t1.html to bin/transcripts. Usage: sm_transcript [options] --srcdir PATH Read files from this folder (Default: ./results) --destdir PATH Write files to this folder (Default: ./transcripts) --srctype wrd | seg | txt | ttml | srt Kind of file to process (Default: wrd) --desttype html | ttml | datajs | json Kind of file to output (Default: html) -h, --help Show this message There is a serious gotch'a in specifying the srctype parameter: it must match the case of the file extension that you're processing. This means that if the srt files that you are processing have the extension .SRT, then you must specify the srctype as "SRT". Pretty lame, I know. I will update the gem with a fix shortly. My apologies until then. Troubleshooting: sm-transcript requires additional gems to operate. The RubyGem installation should install dependencies automatically, but when it doesn't, you get an error that includes ... no such file to load -- builder (LoadError) in the first few lines when you run sm-transcript, the problem is a missing dependent gem. (the error above indicates that the Builder gem is missing.) Try installing the missing gem. For the error above, the command looks like this on my computer: felix$ sudo gem install builder See "Required Gems" below for more information. A warning message such as: "WARNING: Nokogiri was built against LibXML version 2.7.6, but has dynamically loaded 2.7.7"" may be safely ignored. If you continue to have trouble, feel free to contact me. Upgrading: You can easily upgrade by simply executing the same command you used to install the gem. Running install again will add the newer version and make it active. By default the most recent version is used, but older versions are still available, simply inactive. If are using svn, you should already know what to do. Required Gems: builder - create structured data, such as XML extensions - added for the 'require_relative' command. (To get this command in Ruby 1.8 you need to install this gem, for Ruby 1.9 the command is already part of the core.) htmlentities - html parsing json - create JSON structured data nokogiri - xml parsing library optparse - option parsing of command line ostruct - open data structures ppcommand - pp is a pretty printer. It is used only for debugging rake - make for Ruby rubygems - support for gems (shouldn't be needed for Ruby 1.9) shoulda - enhancement for Test::Unit This command installs gems on OSX and Linux: felix$ sudo gem install <gem name> I recommend running the following command to update to latest version of rubygems before loading new gems. felix$ sudo gem update --system Unit Tests: You may run all unit tests by navigating to the test folder and running rake with no parameters (the default rake task runs all tests). On my computer, it looks like this: projects/sm_transcript/test felix$ rake Release Notes: Initial Version - runs under Ruby 1.8.x. version 0.0.4 - fixes bug when processing .WRD files with CRLF line endings. version 0.0.5 - removed due to posting error version 0.0.6 - added srctype of ttml and desttype of json, fixed bug where beginning time of word was actually for previous word. version 0.0.7 - added srt as srctype version 0.0.8 - fixed bug that dropped last phrase from transcripts version 1.0.0 - declared this version 1.0.0 to conform more closely with gem numbering conventions. All tests run successfully. To Do: - specify individual files for processing rather than folders - fix bug in srt processing: can't read Creole srt content. - allow user to modify the "t1" file extension for addition languages of the same transcript. - update code to run under Ruby 1.9

No description provided.

No description provided.

No description provided.

No description provided.

No description provided.

No description provided.

No description provided.

No description provided.

No description provided.

No description provided.

No description provided.

No description provided.