map an object, predicate updates values, receives (value, key, object)
Persistent ordered mapping from strings
Map and Set with automatic key interning
Fixes stack traces for files with source maps
Returns an object with values at all depths mapped according to the provided function
A thing that is a lot like ES6 `Map`, but without iterators, for use in environments where `for..of` syntax and `Map` are not available.
Map object keys and values into a new object
Runtime to be use with the Flow Enums transform.
Simple “Least Recently Used” (LRU) cache
A Map with weakly referenced values.
Map over promises concurrently
A WeakMap shim for Node.js and browsers
Retrieves a name:pathname Map for a given workspaces config
A function similar to lodash.mapValues or Ramda.mapObjIndexed, but for async functions
A tiny (304B to 489B) utility for check for deep equality
Immutable Data Collections
Maps the values yielded by an async iterator
Bigmap for Keyv
JS library that allows you to easily serialize and deserialize data with BigInt values
Generates and consumes source maps
Curated collection of data structures for the JavaScript/TypeScript.
Generates and consumes source maps
Map object values returning a new object
Transforms nested values of complex objects
==== Ruby Topic Maps (RTM) RTM is a Topic Maps engine written in Ruby. See http://rtm.topicmapslab.de/ for instructions. Several backends and extensions are available as separate gems. ==== Overview From a developer's perspective, RTM is a schema-less database management system. The Topic Maps standard (described below) on which RTM is based provides a way of creating a self-describing schema just by using it. ==== Quickstart require 'rtm' connection = RTM.connect # uses the default Ontopia in-memory backend topic_map = connection.create "http://example.org/my_topic_map/" some_topic = topicmap.get!("identifier_of_the_topic") some_topic["-"] = "default name for the topic" topic_map.to_xtm("my_xtm_file.xtm") ==== Topic Maps Topic Maps is an international industry standard (ISO13250) for interchangeably representing information about the structure of information resources used to define topics, and the relationships between topics. A set of one or more interrelated documents that employs the notation defined by this International Standard is called a topic map. A topic map defines a multidimensional topic space - a space in which the locations are topics, and in which the distances between topics are measurable in terms of the number of intervening topics which must be visited in order to get from one topic to another, and the kinds of relationships that define the path from one topic to another, if any, through the intervening topics, if any. In addition, information objects can have properties, as well as values for those properties, assigned to them. The Topic Maps Data Model which is used in this implementation can be found on http://www.isotopicmaps.org/sam/sam-model/. ==== Backends * rtm-ontopia: JRuby only, recommended, uses Ontopia: http://code.google.com/p/ontopia/ * rtm-tinytim: JRuby only, uses TinyTiM: http://tinytim.sourceforge.net/ * rtm-activerecord: uses a custom ActiveRecord schema ==== Extensions * rtm-tmql: Adds support for the Topic Maps Query Language (TMQL), http://isotopicmaps.org/tmql/ * rtm-tmcl: Adds support for the Topic Maps Constraint Language (TMCL), http://isotopicmaps.org/tmcl/ ==== License Copyright 2009 Topic Maps Lab, University of Leipzig. Apache License, Version 2.0
Sinatra-rraoute provides `gget'/`ppost'/`ddelete'/... methods which work just like Sinatra's built-in `get'/`post'/`delete'/... methods, but which map named routes to functions so that they can be referenced in redirects etc. The `path' helper will return a route for a certain route name and the given values for this route and comes in handy in both, the controller/model component of the application, and the view where you can use it to render links, assets URLs, AJAX calls... The nestable `nnamespace' method is useful for API versioning and does not interfere with other namespace extensions for Sinatra.
==== Topic Maps for Rails (rtm-rails) RTM-Rails is the Rails-Adapter for Ruby Topic Maps. It allows simple configuration of topicmaps in config/topicmaps.yml. ==== Overview From a developer's perspective, RTM is a schema-less database management system. The Topic Maps standard (described below) on which RTM is based provides a way of creating a self-describing schema just by using it. You can use RTM as a complement data storage to ActiveRecord in your Rails apps. ==== Quickstart - existing Rails project jruby script/generate topicmaps Run the command above after installing rtm-rails. This will create * a minimal default configuration: config/topicmaps.yml and * a file with more examples and explanations config/topicmaps.example.yml * a file README.topicmaps.txt which contains more information how to use it and where to find more information * an initializer to load the topicmaps at startup * a rake task to migrate the topic maps backends in your rails application. ==== Quickstart - new Rails project For a new Rails application these are the complete initial steps: jruby -S rails my_topicmaps_app cd my_topicmaps_app jruby -S script/generate jdbc jruby -S script/generate topicmaps # The following lines are necessary because Rails does not have a template # for the H2 database and Ontopia does not support the Rails default SQLite3. sed -e "s/sqlite3/h2/" config/database.yml > config/database.yml.h2 mv config/database.yml.h2 config/database.yml # Prepare the database and then check if all is OK jruby -S rake topicmaps:migrate_backends jruby -S rake topicmaps:check ==== Usage inside the application When everything is fine, let's create our first topic: jruby -S script/console TM[:example].get!("http://example.org/my/first/topic") # and save the topic map TM[:example].commit Access the configured topic maps anywhere in your application like this: TM[:example] To retrieve all topics, you can do TM[:example].topics To retrieve a specific topic by its subject identifier: TM[:example].get("http://example.org/my/topic") Commit the changes to the database permanently: TM[:example].commit ... or abort the transaction: TM[:example].abort More information can be found on http://rtm.topicmapslab.de/ ==== Minimal configuration default: topicmaps: example: http://rtm.topicmapslab.de/example1/ The minimal configuration creates a single topic map, named :example with the locator given. This topic map will be persisted in the same database as your ActiveRecord connection if not specified otherwise. The default backend is OntopiaRDBMS (from the rtm-ontopia gem). A more complete configuration can be found in config/topicmaps.example.yml after running "jruby script/generate topicmaps". It also includes how to specifiy multiple connections to different data stores and so on. ==== Topic Maps Topic Maps is an international industry standard (ISO13250) for interchangeably representing information about the structure of information resources used to define topics, and the relationships between topics. A set of one or more interrelated documents that employs the notation defined by this International Standard is called a topic map. A topic map defines a multidimensional topic space - a space in which the locations are topics, and in which the distances between topics are measurable in terms of the number of intervening topics which must be visited in order to get from one topic to another, and the kinds of relationships that define the path from one topic to another, if any, through the intervening topics, if any. In addition, information objects can have properties, as well as values for those properties, assigned to them. The Topic Maps Data Model which is used in this implementation can be found on http://www.isotopicmaps.org/sam/sam-model/. ==== License Copyright 2009 Topic Maps Lab, University of Leipzig. Apache License, Version 2.0
The Tripletex API is a **RESTful API**, which does not implement PATCH, but uses a PUT with optional fields. **Actions** or commands are represented in our RESTful path with a prefixed `:`. Example: `/v2/hours/123/:approve`. **Summaries** or aggregated results are represented in our RESTful path with a prefixed <code>></code>. Example: <code>/v2/hours/>thisWeeksBillables</code>. **"requestID"** is a key found in all validation and error responses. If additional log information is absolutely necessary, our support division can locate the key value. **Download** the [swagger.json](/v2/swagger.json) file [OpenAPI Specification](https://github.com/OAI/OpenAPI-Specification) to [generate code](https://github.com/sveredyuk/tripletex_ruby). This document was generated from the Swagger JSON file. **version:** This is a versioning number found on all DB records. If included, it will prevent your PUT/POST from overriding any updates to the record since your GET. **Date & DateTime** follows the **ISO 8601** standard. Date: `YYYY-MM-DD`. DateTime: `YYYY-MM-DDThh:mm:ssZ` **Sorting** is done by specifying a comma separated list, where a `-` prefix denotes descending. You can sort by sub object with the following format: `project.name, -date`. **Searching:** is done by entering values in the optional fields for each API call. The values fall into the following categories: range, in, exact and like. **Missing fields or even no response data** can occur because result objects and fields are filtered on authorization. **See [FAQ](https://tripletex.no/execute/docViewer?articleId=906&language=0) for more additional information.** ## Authentication: - **Tokens:** The Tripletex API uses 3 different tokens - **consumerToken**, **employeeToken** and **sessionToken**. - **consumerToken** is a token provided to the consumer by Tripletex after the API 2.0 registration is completed. - **employeeToken** is a token created by an administrator in your Tripletex account via the user settings and the tab "API access". Each employee token must be given a set of entitlements. [Read more here.](https://tripletex.no/execute/docViewer?articleId=853&language=0) - **sessionToken** is the token from `/token/session/:create` which requires a consumerToken and an employeeToken created with the same consumer token, but not an authentication header. See how to create a sessionToken [here](https://tripletex.no/execute/docViewer?articleId=855&language=0). - The session token is used as the password in "Basic Authentication Header" for API calls. - Use blank or `0` as username for accessing the account with regular employee token, or if a company owned employee token accesses <code>/company/>withLoginAccess</code> or <code>/token/session/>whoAmI</code>. - For company owned employee tokens (accounting offices) the ID from <code>/company/>withLoginAccess</code> can be used as username for accessing client accounts. - If you need to create the header yourself use <code>Authorization: Basic <base64encode('0:sessionToken')></code>. ## Tags: - <div class="tag-icon-beta"></div> **[BETA]** This is a beta endpoint and can be subject to change. - <div class="tag-icon-deprecated"></div> **[DEPRECATED]** Deprecated means that we intend to remove/change this feature or capability in a future "major" API release. We therefore discourage all use of this feature/capability. ## Fields: Use the `fields` parameter to specify which fields should be returned. This also supports fields from sub elements. Example values: - `project,activity,hours` returns `{project:..., activity:...., hours:...}`. - just `project` returns `"project" : { "id": 12345, "url": "tripletex.no/v2/projects/12345" }`. - `project(*)` returns `"project" : { "id": 12345 "name":"ProjectName" "number.....startDate": "2013-01-07" }`. - `project(name)` returns `"project" : { "name":"ProjectName" }`. - All elements and some subElements : `*,activity(name),employee(*)`. ## Changes: To get the changes for a resource, `changes` have to be explicitly specified as part of the `fields` parameter, e.g. `*,changes`. There are currently two types of change available: - `CREATE` for when the resource was created - `UPDATE` for when the resource was updated NOTE: For objects created prior to October 24th 2018 the list may be incomplete, but will always contain the CREATE and the last change (if the object has been changed after creation). ## Rate limiting in each response header: Rate limiting is performed on the API calls for an employee for each API consumer. Status regarding the rate limit is returned as headers: - `X-Rate-Limit-Limit` - The number of allowed requests in the current period. - `X-Rate-Limit-Remaining` - The number of remaining requests. - `X-Rate-Limit-Reset` - The number of seconds left in the current period. Once the rate limit is hit, all requests will return HTTP status code `429` for the remainder of the current period. ## Response envelope: ``` { "fullResultSize": ###, "from": ###, // Paging starting from "count": ###, // Paging count "versionDigest": "Hash of full result", "values": [...list of objects...] } { "value": {...single object...} } ``` ## WebHook envelope: ``` { "subscriptionId": ###, "event": "object.verb", // As listed from /v2/event/ "id": ###, // Object id "value": {... single object, null if object.deleted ...} } ``` ## Error/warning envelope: ``` { "status": ###, // HTTP status code "code": #####, // internal status code of event "message": "Basic feedback message in your language", "link": "Link to doc", "developerMessage": "More technical message", "validationMessages": [ // Will be null if Error { "field": "Name of field", "message": "Validation failure information" } ], "requestId": "UUID used in any logs" } ``` ## Status codes / Error codes: - **200 OK** - **201 Created** - From POSTs that create something new. - **204 No Content** - When there is no answer, ex: "/:anAction" or DELETE. - **400 Bad request** - - **4000** Bad Request Exception - **11000** Illegal Filter Exception - **12000** Path Param Exception - **24000** Cryptography Exception - **401 Unauthorized** - When authentication is required and has failed or has not yet been provided - **3000** Authentication Exception - **9000** Security Exception - **403 Forbidden** - When AuthorisationManager says no. - **404 Not Found** - For content/IDs that does not exist. - **6000** Not Found Exception - **409 Conflict** - Such as an edit conflict between multiple simultaneous updates - **7000** Object Exists Exception - **8000** Revision Exception - **10000** Locked Exception - **14000** Duplicate entry - **422 Bad Request** - For Required fields or things like malformed payload. - **15000** Value Validation Exception - **16000** Mapping Exception - **17000** Sorting Exception - **18000** Validation Exception - **21000** Param Exception - **22000** Invalid JSON Exception - **23000** Result Set Too Large Exception - **429 Too Many Requests** - Request rate limit hit - **500 Internal Error** - Unexpected condition was encountered and no more specific message is suitable - **1000** Exception
The Tripletex API is a **RESTful API**, which does not implement PATCH, but uses a PUT with optional fields. **Actions** or commands are represented in our RESTful path with a prefixed `:`. Example: `/v2/hours/123/:approve`. **Summaries** or aggregated results are represented in our RESTful path with a prefixed <code>></code>. Example: <code>/v2/hours/>thisWeeksBillables</code>. **"requestID"** is a key found in all validation and error responses. If additional log information is absolutely necessary, our support division can locate the key value. **Download** the [swagger.json](/v2/swagger.json) file [OpenAPI Specification](https://github.com/OAI/OpenAPI-Specification) to [generate code](https://github.com/swagger-api/swagger-codegen). This document was generated from the Swagger JSON file. **version:** This is a versioning number found on all DB records. If included, it will prevent your PUT/POST from overriding any updates to the record since your GET. **Date & DateTime** follows the **ISO 8601** standard. Date: `YYYY-MM-DD`. DateTime: `YYYY-MM-DDThh:mm:ssZ` **Sorting** is done by specifying a comma separated list, where a `-` prefix denotes descending. You can sort by sub object with the following format: `project.name, -date`. **Searching:** is done by entering values in the optional fields for each API call. The values fall into the following categories: range, in, exact and like. **Missing fields or even no response data** can occur because result objects and fields are filtered on authorization. **See [FAQ](https://tripletex.no/execute/docViewer?articleId=906&language=0) for more additional information.** ## Authentication: - **Tokens:** The Tripletex API uses 3 different tokens - **consumerToken**, **employeeToken** and **sessionToken**. - **consumerToken** is a token provided to the consumer by Tripletex after the API 2.0 registration is completed. - **employeeToken** is a token created by an administrator in your Tripletex account via the user settings and the tab "API access". Each employee token must be given a set of entitlements. [Read more here.](https://tripletex.no/execute/docViewer?articleId=853&language=0) - **sessionToken** is the token from `/token/session/:create` which requires a consumerToken and an employeeToken created with the same consumer token, but not an authentication header. See how to create a sessionToken [here](https://tripletex.no/execute/docViewer?articleId=855&language=0). - The session token is used as the password in "Basic Authentication Header" for API calls. - Use blank or `0` as username for accessing the account with regular employee token, or if a company owned employee token accesses <code>/company/>withLoginAccess</code> or <code>/token/session/>whoAmI</code>. - For company owned employee tokens (accounting offices) the ID from <code>/company/>withLoginAccess</code> can be used as username for accessing client accounts. - If you need to create the header yourself use <code>Authorization: Basic <base64encode('0:sessionToken')></code>. ## Tags: - <div class="tag-icon-beta"></div> **[BETA]** This is a beta endpoint and can be subject to change. - <div class="tag-icon-deprecated"></div> **[DEPRECATED]** Deprecated means that we intend to remove/change this feature or capability in a future "major" API release. We therefore discourage all use of this feature/capability. ## Fields: Use the `fields` parameter to specify which fields should be returned. This also supports fields from sub elements. Example values: - `project,activity,hours` returns `{project:..., activity:...., hours:...}`. - just `project` returns `"project" : { "id": 12345, "url": "tripletex.no/v2/projects/12345" }`. - `project(*)` returns `"project" : { "id": 12345 "name":"ProjectName" "number.....startDate": "2013-01-07" }`. - `project(name)` returns `"project" : { "name":"ProjectName" }`. - All elements and some subElements : `*,activity(name),employee(*)`. ## Changes: To get the changes for a resource, `changes` have to be explicitly specified as part of the `fields` parameter, e.g. `*,changes`. There are currently two types of change available: - `CREATE` for when the resource was created - `UPDATE` for when the resource was updated NOTE: For objects created prior to October 24th 2018 the list may be incomplete, but will always contain the CREATE and the last change (if the object has been changed after creation). ## Rate limiting in each response header: Rate limiting is performed on the API calls for an employee for each API consumer. Status regarding the rate limit is returned as headers: - `X-Rate-Limit-Limit` - The number of allowed requests in the current period. - `X-Rate-Limit-Remaining` - The number of remaining requests. - `X-Rate-Limit-Reset` - The number of seconds left in the current period. Once the rate limit is hit, all requests will return HTTP status code `429` for the remainder of the current period. ## Response envelope: ``` { "fullResultSize": ###, "from": ###, // Paging starting from "count": ###, // Paging count "versionDigest": "Hash of full result", "values": [...list of objects...] } { "value": {...single object...} } ``` ## WebHook envelope: ``` { "subscriptionId": ###, "event": "object.verb", // As listed from /v2/event/ "id": ###, // Object id "value": {... single object, null if object.deleted ...} } ``` ## Error/warning envelope: ``` { "status": ###, // HTTP status code "code": #####, // internal status code of event "message": "Basic feedback message in your language", "link": "Link to doc", "developerMessage": "More technical message", "validationMessages": [ // Will be null if Error { "field": "Name of field", "message": "Validation failure information" } ], "requestId": "UUID used in any logs" } ``` ## Status codes / Error codes: - **200 OK** - **201 Created** - From POSTs that create something new. - **204 No Content** - When there is no answer, ex: "/:anAction" or DELETE. - **400 Bad request** - - **4000** Bad Request Exception - **11000** Illegal Filter Exception - **12000** Path Param Exception - **24000** Cryptography Exception - **401 Unauthorized** - When authentication is required and has failed or has not yet been provided - **3000** Authentication Exception - **9000** Security Exception - **403 Forbidden** - When AuthorisationManager says no. - **404 Not Found** - For content/IDs that does not exist. - **6000** Not Found Exception - **409 Conflict** - Such as an edit conflict between multiple simultaneous updates - **7000** Object Exists Exception - **8000** Revision Exception - **10000** Locked Exception - **14000** Duplicate entry - **422 Bad Request** - For Required fields or things like malformed payload. - **15000** Value Validation Exception - **16000** Mapping Exception - **17000** Sorting Exception - **18000** Validation Exception - **21000** Param Exception - **22000** Invalid JSON Exception - **23000** Result Set Too Large Exception - **429 Too Many Requests** - Request rate limit hit - **500 Internal Error** - Unexpected condition was encountered and no more specific message is suitable - **1000** Exception
The Tripletex API is a **RESTful API**, which does not implement PATCH, but uses a PUT with optional fields. **Actions** or commands are represented in our RESTful path with a prefixed `:`. Example: `/v2/hours/123/:approve`. **Summaries** or aggregated results are represented in our RESTful path with a prefixed <code>></code>. Example: <code>/v2/hours/>thisWeeksBillables</code>. **"requestID"** is a key found in all validation and error responses. If additional log information is absolutely necessary, our support division can locate the key value. **Download** the [swagger.json](/v2/swagger.json) file [OpenAPI Specification](https://github.com/OAI/OpenAPI-Specification) to [generate code](https://github.com/sveredyuk/tripletex_ruby). This document was generated from the Swagger JSON file. **version:** This is a versioning number found on all DB records. If included, it will prevent your PUT/POST from overriding any updates to the record since your GET. **Date & DateTime** follows the **ISO 8601** standard. Date: `YYYY-MM-DD`. DateTime: `YYYY-MM-DDThh:mm:ssZ` **Sorting** is done by specifying a comma separated list, where a `-` prefix denotes descending. You can sort by sub object with the following format: `project.name, -date`. **Searching:** is done by entering values in the optional fields for each API call. The values fall into the following categories: range, in, exact and like. **Missing fields or even no response data** can occur because result objects and fields are filtered on authorization. **See [FAQ](https://tripletex.no/execute/docViewer?articleId=906&language=0) for more additional information.** ## Authentication: - **Tokens:** The Tripletex API uses 3 different tokens - **consumerToken**, **employeeToken** and **sessionToken**. - **consumerToken** is a token provided to the consumer by Tripletex after the API 2.0 registration is completed. - **employeeToken** is a token created by an administrator in your Tripletex account via the user settings and the tab "API access". Each employee token must be given a set of entitlements. [Read more here.](https://tripletex.no/execute/docViewer?articleId=853&language=0) - **sessionToken** is the token from `/token/session/:create` which requires a consumerToken and an employeeToken created with the same consumer token, but not an authentication header. See how to create a sessionToken [here](https://tripletex.no/execute/docViewer?articleId=855&language=0). - The session token is used as the password in "Basic Authentication Header" for API calls. - Use blank or `0` as username for accessing the account with regular employee token, or if a company owned employee token accesses <code>/company/>withLoginAccess</code> or <code>/token/session/>whoAmI</code>. - For company owned employee tokens (accounting offices) the ID from <code>/company/>withLoginAccess</code> can be used as username for accessing client accounts. - If you need to create the header yourself use <code>Authorization: Basic <base64encode('0:sessionToken')></code>. ## Tags: - <div class="tag-icon-beta"></div> **[BETA]** This is a beta endpoint and can be subject to change. - <div class="tag-icon-deprecated"></div> **[DEPRECATED]** Deprecated means that we intend to remove/change this feature or capability in a future "major" API release. We therefore discourage all use of this feature/capability. ## Fields: Use the `fields` parameter to specify which fields should be returned. This also supports fields from sub elements. Example values: - `project,activity,hours` returns `{project:..., activity:...., hours:...}`. - just `project` returns `"project" : { "id": 12345, "url": "tripletex.no/v2/projects/12345" }`. - `project(*)` returns `"project" : { "id": 12345 "name":"ProjectName" "number.....startDate": "2013-01-07" }`. - `project(name)` returns `"project" : { "name":"ProjectName" }`. - All elements and some subElements : `*,activity(name),employee(*)`. ## Changes: To get the changes for a resource, `changes` have to be explicitly specified as part of the `fields` parameter, e.g. `*,changes`. There are currently two types of change available: - `CREATE` for when the resource was created - `UPDATE` for when the resource was updated NOTE: For objects created prior to October 24th 2018 the list may be incomplete, but will always contain the CREATE and the last change (if the object has been changed after creation). ## Rate limiting in each response header: Rate limiting is performed on the API calls for an employee for each API consumer. Status regarding the rate limit is returned as headers: - `X-Rate-Limit-Limit` - The number of allowed requests in the current period. - `X-Rate-Limit-Remaining` - The number of remaining requests. - `X-Rate-Limit-Reset` - The number of seconds left in the current period. Once the rate limit is hit, all requests will return HTTP status code `429` for the remainder of the current period. ## Response envelope: ``` { "fullResultSize": ###, "from": ###, // Paging starting from "count": ###, // Paging count "versionDigest": "Hash of full result", "values": [...list of objects...] } { "value": {...single object...} } ``` ## WebHook envelope: ``` { "subscriptionId": ###, "event": "object.verb", // As listed from /v2/event/ "id": ###, // Object id "value": {... single object, null if object.deleted ...} } ``` ## Error/warning envelope: ``` { "status": ###, // HTTP status code "code": #####, // internal status code of event "message": "Basic feedback message in your language", "link": "Link to doc", "developerMessage": "More technical message", "validationMessages": [ // Will be null if Error { "field": "Name of field", "message": "Validation failure information" } ], "requestId": "UUID used in any logs" } ``` ## Status codes / Error codes: - **200 OK** - **201 Created** - From POSTs that create something new. - **204 No Content** - When there is no answer, ex: "/:anAction" or DELETE. - **400 Bad request** - - **4000** Bad Request Exception - **11000** Illegal Filter Exception - **12000** Path Param Exception - **24000** Cryptography Exception - **401 Unauthorized** - When authentication is required and has failed or has not yet been provided - **3000** Authentication Exception - **9000** Security Exception - **403 Forbidden** - When AuthorisationManager says no. - **404 Not Found** - For content/IDs that does not exist. - **6000** Not Found Exception - **409 Conflict** - Such as an edit conflict between multiple simultaneous updates - **7000** Object Exists Exception - **8000** Revision Exception - **10000** Locked Exception - **14000** Duplicate entry - **422 Bad Request** - For Required fields or things like malformed payload. - **15000** Value Validation Exception - **16000** Mapping Exception - **17000** Sorting Exception - **18000** Validation Exception - **21000** Param Exception - **22000** Invalid JSON Exception - **23000** Result Set Too Large Exception - **429 Too Many Requests** - Request rate limit hit - **500 Internal Error** - Unexpected condition was encountered and no more specific message is suitable - **1000** Exception
Lookout Lookout is a unit testing framework for Ruby¹ that puts your results in focus. Tests (expectations) are written as follows expect 2 do 1 + 1 end expect ArgumentError do Integer('1 + 1') end expect Array do [1, 2, 3].select{ |i| i % 2 == 0 } end expect [2, 4, 6] do [1, 2, 3].map{ |i| i * 2 } end Lookout is designed to encourage – force, even – unit testing best practices such as • Setting up only one expectation per test • Not setting expectations on non-public APIs • Test isolation This is done by • Only allowing one expectation to be set per test • Providing no (additional) way of accessing private state • Providing no setup and tear-down methods, nor a method of providing test helpers Other important points are • Putting the expected outcome of a test in focus with the steps of the calculation of the actual result only as a secondary concern • A focus on code readability by providing no mechanism for describing an expectation other than the code in the expectation itself • A unified syntax for setting up both state-based and behavior-based expectations The way Lookout works has been heavily influenced by expectations², by {Jay Fields}³. The code base was once also heavily based on expectations, based at Subversion {revision 76}⁴. A lot has happened since then and all of the work past that revision are due to {Nikolai Weibull}⁵. ¹ Ruby: http://ruby-lang.org/ ² Expectations: http://expectations.rubyforge.org/ ³ Jay Fields’s blog: http://blog.jayfields.com/ ⁴ Lookout revision 76: https://github.com/now/lookout/commit/537bedf3e5b3eb4b31c066b3266f42964ac35ebe ⁵ Nikolai Weibull’s home page: http://disu.se/ § Installation Install Lookout with % gem install lookout § Usage Lookout allows you to set expectations on an object’s state or behavior. We’ll begin by looking at state expectations and then take a look at expectations on behavior. § Expectations on State: Literals An expectation can be made on the result of a computation: expect 2 do 1 + 1 end Most objects, in fact, have their state expectations checked by invoking ‹#==› on the expected value with the result as its argument. Checking that a result is within a given range is also simple: expect 0.099..0.101 do 0.4 - 0.3 end Here, the more general ‹#===› is being used on the ‹Range›. § Regexps ‹Strings› of course match against ‹Strings›: expect 'ab' do 'abc'[0..1] end but we can also match a ‹String› against a ‹Regexp›: expect %r{a substring} do 'a string with a substring' end (Note the use of ‹%r{…}› to avoid warnings that will be generated when Ruby parses ‹expect /…/›.) § Modules Checking that the result includes a certain module is done by expecting the ‹Module›. expect Enumerable do [] end This, due to the nature of Ruby, of course also works for classes (as they are also modules): expect String do 'a string' end This doesn’t hinder us from expecting the actual ‹Module› itself: expect Enumerable do Enumerable end or the ‹Class›: expect String do String end for obvious reasons. As you may have figured out yourself, this is accomplished by first trying ‹#==› and, if it returns ‹false›, then trying ‹#===› on the expected ‹Module›. This is also true of ‹Ranges› and ‹Regexps›. § Booleans Truthfulness is expected with ‹true› and ‹false›: expect true do 1 end expect false do nil end Results equaling ‹true› or ‹false› are slightly different: expect TrueClass do true end expect FalseClass do false end The rationale for this is that you should only care if the result of a computation evaluates to a value that Ruby considers to be either true or false, not the exact literals ‹true› or ‹false›. § IO Expecting output on an IO object is also common: expect output("abc\ndef\n") do |io| io.puts 'abc', 'def' end This can be used to capture the output of a formatter that takes an output object as a parameter. § Warnings Expecting warnings from code isn’t very common, but should be done: expect warning('this is your final one!') do warn 'this is your final one!' end expect warning('this is your final one!') do warn '%s:%d: warning: this is your final one!' % [__FILE__, __LINE__] end ‹$VERBOSE› is set to ‹true› during the execution of the block, so you don’t need to do so yourself. If you have other code that depends on the value of $VERBOSE, that can be done with ‹#with_verbose› expect nil do with_verbose nil do $VERBOSE end end § Errors You should always be expecting errors from – and in, but that’s a different story – your code: expect ArgumentError do Integer('1 + 1') end Often, not only the type of the error, but its description, is important to check: expect StandardError.new('message') do raise StandardError.new('message') end As with ‹Strings›, ‹Regexps› can be used to check the error description: expect StandardError.new(/mess/) do raise StandardError.new('message') end § Queries Through Symbols Symbols are generally matched against symbols, but as a special case, symbols ending with ‹?› are seen as expectations on the result of query methods on the result of the block, given that the method is of zero arity and that the result isn’t a Symbol itself. Simply expect a symbol ending with ‹?›: expect :empty? do [] end To expect it’s negation, expect the same symbol beginning with ‹not_›: expect :not_nil? do [1, 2, 3] end This is the same as expect true do [].empty? end and expect false do [1, 2, 3].empty? end but provides much clearer failure messages. It also makes the expectation’s intent a lot clearer. § Queries By Proxy There’s also a way to make the expectations of query methods explicit by invoking methods on the result of the block. For example, to check that the even elements of the Array ‹[1, 2, 3]› include ‹1› you could write expect result.to.include? 1 do [1, 2, 3].reject{ |e| e.even? } end You could likewise check that the result doesn’t include 2: expect result.not.to.include? 2 do [1, 2, 3].reject{ |e| e.even? } end This is the same as (and executes a little bit slower than) writing expect false do [1, 2, 3].reject{ |e| e.even? }.include? 2 end but provides much clearer failure messages. Given that these two last examples would fail, you’d get a message saying “[1, 2, 3]#include?(2)” instead of the terser “true≠false”. It also clearly separates the actual expectation from the set-up. The keyword for this kind of expectations is ‹result›. This may be followed by any of the methods • ‹#not› • ‹#to› • ‹#be› • ‹#have› or any other method you will want to call on the result. The methods ‹#to›, ‹#be›, and ‹#have› do nothing except improve readability. The ‹#not› method inverts the expectation. § Literal Literals If you need to literally check against any of the types of objects otherwise treated specially, that is, any instances of • ‹Module› • ‹Range› • ‹Regexp› • ‹Exception› • ‹Symbol›, given that it ends with ‹?› you can do so by wrapping it in ‹literal(…)›: expect literal(:empty?) do :empty? end You almost never need to do this, as, for all but symbols, instances will match accordingly as well. § Expectations on Behavior We expect our objects to be on their best behavior. Lookout allows you to make sure that they are. Reception expectations let us verify that a method is called in the way that we expect it to be: expect mock.to.receive.to_str(without_arguments){ '123' } do |o| o.to_str end Here, ‹#mock› creates a mock object, an object that doesn’t respond to anything unless you tell it to. We tell it to expect to receive a call to ‹#to_str› without arguments and have ‹#to_str› return ‹'123'› when called. The mock object is then passed in to the block so that the expectations placed upon it can be fulfilled. Sometimes we only want to make sure that a method is called in the way that we expect it to be, but we don’t care if any other methods are called on the object. A stub object, created with ‹#stub›, expects any method and returns a stub object that, again, expects any method, and thus fits the bill. expect stub.to.receive.to_str(without_arguments){ '123' } do |o| o.to_str if o.convertable? end You don’t have to use a mock object to verify that a method is called: expect Object.to.receive.name do Object.name end As you have figured out by now, the expected method call is set up by calling ‹#receive› after ‹#to›. ‹#Receive› is followed by a call to the method to expect with any expected arguments. The body of the expected method can be given as the block to the method. Finally, an expected invocation count may follow the method. Let’s look at this formal specification in more detail. The expected method arguments may be given in a variety of ways. Let’s introduce them by giving some examples: expect mock.to.receive.a do |m| m.a end Here, the method ‹#a› must be called with any number of arguments. It may be called any number of times, but it must be called at least once. If a method must receive exactly one argument, you can use ‹Object›, as the same matching rules apply for arguments as they do for state expectations: expect mock.to.receive.a(Object) do |m| m.a 0 end If a method must receive a specific argument, you can use that argument: expect mock.to.receive.a(1..2) do |m| m.a 1 end Again, the same matching rules apply for arguments as they do for state expectations, so the previous example expects a call to ‹#a› with 1, 2, or the Range 1..2 as an argument on ‹m›. If a method must be invoked without any arguments you can use ‹without_arguments›: expect mock.to.receive.a(without_arguments) do |m| m.a end You can of course use both ‹Object› and actual arguments: expect mock.to.receive.a(Object, 2, Object) do |m| m.a nil, 2, '3' end The body of the expected method may be given as the block. Here, calling ‹#a› on ‹m› will give the result ‹1›: expect mock.to.receive.a{ 1 } do |m| raise 'not 1' unless m.a == 1 end If no body has been given, the result will be a stub object. To take a block, grab a block parameter and ‹#call› it: expect mock.to.receive.a{ |&b| b.call(1) } do |m| j = 0 m.a{ |i| j = i } raise 'not 1' unless j == 1 end To simulate an ‹#each›-like method, ‹#call› the block several times. Invocation count expectations can be set if the default expectation of “at least once” isn’t good enough. The following expectations are possible • ‹#at_most_once› • ‹#once› • ‹#at_least_once› • ‹#twice› And, for a given ‹N›, • ‹#at_most(N)› • ‹#exactly(N)› • ‹#at_least(N)› § Utilities: Stubs Method stubs are another useful thing to have in a unit testing framework. Sometimes you need to override a method that does something a test shouldn’t do, like access and alter bank accounts. We can override – stub out – a method by using the ‹#stub› method. Let’s assume that we have an ‹Account› class that has two methods, ‹#slips› and ‹#total›. ‹#Slips› retrieves the bank slips that keep track of your deposits to the ‹Account› from a database. ‹#Total› sums the ‹#slips›. In the following test we want to make sure that ‹#total› does what it should do without accessing the database. We therefore stub out ‹#slips› and make it return something that we can easily control. expect 6 do |m| stub(Class.new{ def slips raise 'database not available' end def total slips.reduce(0){ |m, n| m.to_i + n.to_i } end }.new, :slips => [1, 2, 3]){ |account| account.total } end To make it easy to create objects with a set of stubbed methods there’s also a convenience method: expect 3 do s = stub(:a => 1, :b => 2) s.a + s.b end This short-hand notation can also be used for the expected value: expect stub(:a => 1, :b => 2).to.receive.a do |o| o.a + o.b end and also works for mock objects: expect mock(:a => 2, :b => 2).to.receive.a do |o| o.a + o.b end Blocks are also allowed when defining stub methods: expect 3 do s = stub(:a => proc{ |a, b| a + b }) s.a(1, 2) end If need be, we can stub out a specific method on an object: expect 'def' do stub('abc', :to_str => 'def'){ |a| a.to_str } end The stub is active during the execution of the block. § Overriding Constants Sometimes you need to override the value of a constant during the execution of some code. Use ‹#with_const› to do just that: expect 'hello' do with_const 'A::B::C', 'hello' do A::B::C end end Here, the constant ‹A::B::C› is set to ‹'hello'› during the execution of the block. None of the constants ‹A›, ‹B›, and ‹C› need to exist for this to work. If a constant doesn’t exist it’s created and set to a new, empty, ‹Module›. The value of ‹A::B::C›, if any, is restored after the block returns and any constants that didn’t previously exist are removed. § Overriding Environment Variables Another thing you often need to control in your tests is the value of environment variables. Depending on such global values is, of course, not a good practice, but is often unavoidable when working with external libraries. ‹#With_env› allows you to override the value of environment variables during the execution of a block by giving it a ‹Hash› of key/value pairs where the key is the name of the environment variable and the value is the value that it should have during the execution of that block: expect 'hello' do with_env 'INTRO' => 'hello' do ENV['INTRO'] end end Any overridden values are restored and any keys that weren’t previously a part of the environment are removed when the block returns. § Overriding Globals You may also want to override the value of a global temporarily: expect 'hello' do with_global :$stdout, StringIO.new do print 'hello' $stdout.string end end You thus provide the name of the global and a value that it should take during the execution of a block of code. The block gets passed the overridden value, should you need it: expect true do with_global :$stdout, StringIO.new do |overridden| $stdout != overridden end end § Integration Lookout can be used from Rake¹. Simply install Lookout-Rake²: % gem install lookout-rake and add the following code to your Rakefile require 'lookout-rake-3.0' Lookout::Rake::Tasks::Test.new Make sure to read up on using Lookout-Rake for further benefits and customization. ¹ Read more about Rake at http://rake.rubyforge.org/ ² Get information on Lookout-Rake at http://disu.se/software/lookout-rake/ § API Lookout comes with an API¹ that let’s you create things such as new expected values, difference reports for your types, and so on. ¹ See http://disu.se/software/lookout/api/ § Interface Design The default output of Lookout can Spartanly be described as Spartan. If no errors or failures occur, no output is generated. This is unconventional, as unit testing frameworks tend to dump a lot of information on the user, concerning things such as progress, test count summaries, and flamboyantly colored text telling you that your tests passed. None of this output is needed. Your tests should run fast enough to not require progress reports. The lack of output provides you with the same amount of information as reporting success. Test count summaries are only useful if you’re worried that your tests aren’t being run, but if you worry about that, then providing such output doesn’t really help. Testing your tests requires something beyond reporting some arbitrary count that you would have to verify by hand anyway. When errors or failures do occur, however, the relevant information is output in a format that can easily be parsed by an ‹'errorformat'› for Vim or with {Compilation Mode}¹ for Emacs². Diffs are generated for Strings, Arrays, Hashes, and I/O. ¹ Read up on Compilation mode for Emacs at http://www.emacswiki.org/emacs/CompilationMode ² Visit The GNU Foundation’s Emacs’ software page at http://www.gnu.org/software/emacs/ § External Design Let’s now look at some of the points made in the introduction in greater detail. Lookout only allows you to set one expectation per test. If you’re testing behavior with a reception expectation, then only one method-invocation expectation can be set. If you’re testing state, then only one result can be verified. It may seem like this would cause unnecessary duplication between tests. While this is certainly a possibility, when you actually begin to try to avoid such duplication you find that you often do so by improving your interfaces. This kind of restriction tends to encourage the use of value objects, which are easy to test, and more focused objects, which require simpler tests, as they have less behavior to test, per method. By keeping your interfaces focused you’re also keeping your tests focused. Keeping your tests focused improves, in itself, test isolation, but let’s look at something that hinders it: setup and tear-down methods. Most unit testing frameworks encourage test fragmentation by providing setup and tear-down methods. Setup methods create objects and, perhaps, just their behavior for a set of tests. This means that you have to look in two places to figure out what’s being done in a test. This may work fine for few methods with simple set-ups, but makes things complicated when the number of tests increases and the set-up is complex. Often, each test further adjusts the previously set-up object before performing any verifications, further complicating the process of figuring out what state an object has in a given test. Tear-down methods clean up after tests, perhaps by removing records from a database or deleting files from the file-system. The duplication that setup methods and tear-down methods hope to remove is better avoided by improving your interfaces. This can be done by providing better set-up methods for your objects and using idioms such as {Resource Acquisition Is Initialization}¹ for guaranteed clean-up, test or no test. By not using setup and tear-down methods we keep everything pertinent to a test in the test itself, thus improving test isolation. (You also won’t {slow down your tests}² by keeping unnecessary state.) Most unit test frameworks also allow you to create arbitrary test helper methods. Lookout doesn’t. The same rationale as that that has been crystallized in the preceding paragraphs applies. If you need helpers you’re interface isn’t good enough. It really is as simple as that. To clarify: there’s nothing inherently wrong with test helper methods, but they should be general enough that they reside in their own library. The support for mocks in Lookout is provided through a set of test helper methods that make it easier to create mocks than it would have been without them. Lookout-rack³ is another example of a library providing test helper methods (well, one method, actually) that are very useful in testing web applications that use Rack⁴. A final point at which some unit test frameworks try to fragment tests further is documentation. These frameworks provide ways of describing the whats and hows of what’s being tested, the rationale being that this will provide documentation of both the test and the code being tested. Describing how a stack data structure is meant to work is a common example. A stack is, however, a rather simple data structure, so such a description provides little, if any, additional information that can’t be extracted from the implementation and its tests themselves. The implementation and its tests is, in fact, its own best documentation. Taking the points made in the previous paragraphs into account, we should already have simple, self-describing, interfaces that have easily understood tests associated with them. Rationales for the use of a given data structure or system-design design documentation is better suited in separate documentation focused at describing exactly those issues. ¹ Read the Wikipedia entry for Resource Acquisition Is Initialization at http://en.wikipedia.org/wiki/Resource_Acquisition_Is_Initialization ² Read how 37signals had problems with slow Test::Unit tests at http://37signals.com/svn/posts/2742-the-road-to-faster-tests/ ³ Visit the Lookout-rack home page at http://disu.se/software/lookout-rack/ ⁴ Visit the Rack Rubyforge project page at http://rack.rubyforge.org/ § Internal Design The internal design of Lookout has had a couple of goals. • As few external dependencies as possible • As few internal dependencies as possible • Internal extensibility provides external extensibility • As fast load times as possible • As high a ratio of value objects to mutable objects as possible • Each object must have a simple, obvious name • Use mix-ins, not inheritance for shared behavior • As few responsibilities per object as possible • Optimizing for speed can only be done when you have all the facts § External Dependencies Lookout used to depend on Mocha for mocks and stubs. While benchmarking I noticed that a method in Mocha was taking up more than 300 percent of the runtime. It turned out that Mocha’s method for cleaning up back-traces generated when a mock failed was doing something incredibly stupid: backtrace.reject{ |l| Regexp.new(@lib).match(File.expand_path(l)) } Here ‹@lib› is a ‹String› containing the path to the lib sub-directory in the Mocha installation directory. I reported it, provided a patch five days later, then waited. Nothing happened. {254 days later}¹, according to {Wolfram Alpha}², half of my patch was, apparently – I say “apparently”, as I received no notification – applied. By that time I had replaced the whole mocking-and-stubbing subsystem and dropped the dependency. Many Ruby developers claim that Ruby and its gems are too fast-moving for normal package-managing systems to keep up. This is testament to the fact that this isn’t the case and that the real problem is instead related to sloppy practices. Please note that I don’t want to single out the Mocha library nor its developers. I only want to provide an example where relying on external dependencies can be “considered harmful”. ¹ See the Wolfram Alpha calculation at http://www.wolframalpha.com/input/?i=days+between+march+17%2C+2010+and+november+26%2C+2010 ² Check out the Wolfram Alpha computational knowledge engine at http://www.wolframalpha.com/ § Internal Dependencies Lookout has been designed so as to keep each subsystem independent of any other. The diff subsystem is, for example, completely decoupled from any other part of the system as a whole and could be moved into its own library at a time where that would be of interest to anyone. What’s perhaps more interesting is that the diff subsystem is itself very modular. The data passes through a set of filters that depends on what kind of diff has been requested, each filter yielding modified data as it receives it. If you want to read some rather functional Ruby I can highly recommend looking at the code in the ‹lib/lookout/diff› directory. This lookout on the design of the library also makes it easy to extend Lookout. Lookout-rack was, for example, written in about four hours and about 5 of those 240 minutes were spent on setting up the interface between the two. § Optimizing For Speed The following paragraph is perhaps a bit personal, but might be interesting nonetheless. I’ve always worried about speed. The original Expectations library used ‹extend› a lot to add new behavior to objects. Expectations, for example, used to hold the result of their execution (what we now term “evaluation”) by being extended by a module representing success, failure, or error. For the longest time I used this same method, worrying about the increased performance cost that creating new objects for results would incur. I finally came to a point where I felt that the code was so simple and clean that rewriting this part of the code for a benchmark wouldn’t take more than perhaps ten minutes. Well, ten minutes later I had my results and they confirmed that creating new objects wasn’t harming performance. I was very pleased. § Naming I hate low lines (underscores). I try to avoid them in method names and I always avoid them in file names. Since the current “best practice” in the Ruby community is to put ‹BeginEndStorage› in a file called ‹begin_end_storage.rb›, I only name constants using a single noun. This has had the added benefit that classes seem to have acquired less behavior, as using a single noun doesn’t allow you to tack on additional behavior without questioning if it’s really appropriate to do so, given the rather limited range of interpretation for that noun. It also seems to encourage the creation of value objects, as something named ‹Range› feels a lot more like a value than ‹BeginEndStorage›. (To reach object-oriented-programming Nirvana you must achieve complete value.) § News § 3.0.0 The ‹xml› expectation has been dropped. It wasn’t documented, didn’t suit very many use cases, and can be better implemented by an external library. The ‹arg› argument matcher for mock method arguments has been removed, as it didn’t provide any benefit over using Object. The ‹#yield› and ‹#each› methods on stub and mock methods have been removed. They were slightly weird and their use case can be implemented using block parameters instead. The ‹stub› method inside ‹expect› blocks now stubs out the methods during the execution of a provided block instead of during the execution of the whole except block. When a mock method is called too many times, this is reported immediately, with a full backtrace. This makes it easier to pin down what’s wrong with the code. Query expectations were added. Explicit query expectations were added. Fluent boolean expectations, for example, ‹expect nil.to.be.nil?› have been replaced by query expectations (‹expect :nil? do nil end›) and explicit query expectations (‹expect result.to.be.nil? do nil end›). This was done to discourage creating objects as the expected value and creating objects that change during the course of the test. The ‹literal› expectation was added. Equality (‹#==›) is now checked before “caseity” (‹#===›) for modules, ranges, and regular expressions to match the documentation. § Financing Currently, most of my time is spent at my day job and in my rather busy private life. Please motivate me to spend time on this piece of software by donating some of your money to this project. Yeah, I realize that requesting money to develop software is a bit, well, capitalistic of me. But please realize that I live in a capitalistic society and I need money to have other people give me the things that I need to continue living under the rules of said society. So, if you feel that this piece of software has helped you out enough to warrant a reward, please PayPal a donation to now@disu.se¹. Thanks! Your support won’t go unnoticed! ¹ Send a donation: https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=now%40disu%2ese&item_name=Lookout § Reporting Bugs Please report any bugs that you encounter to the {issue tracker}¹. ¹ See https://github.com/now/lookout/issues § Contributors Contributors to the original expectations codebase are mentioned there. We hope no one on that list feels left out of this list. Please {let us know}¹ if you do. • Nikolai Weibull ¹ Add an issue to the Lookout issue tracker at https://github.com/now/lookout/issues § Licensing Lookout is free software: you may redistribute it and/or modify it under the terms of the {GNU Lesser General Public License, version 3}¹ or later², as published by the {Free Software Foundation}³. ¹ See http://disu.se/licenses/lgpl-3.0/ ² See http://gnu.org/licenses/ ³ See http://fsf.org/
README ====== This is a simple API to evaluate information retrieval results. It allows you to load ranked and unranked query results and calculate various evaluation metrics (precision, recall, MAP, kappa) against a previously loaded gold standard. Start this program from the command line with: retreval -l <gold-standard-file> -q <query-results> -f <format> -o <output-prefix> The options are outlined when you pass no arguments and just call retreval You will find further information in the RDOC documentation and the HOWTO section below. If you want to see an example, use this command: retreval -l example/gold_standard.yml -q example/query_results.yml -f yaml -v INSTALLATION ============ If you have RubyGems, just run gem install retreval You can manually download the sources and build the Gem from there by `cd`ing to the folder where this README is saved and calling gem build retreval.gemspec This will create a gem file called which you just have to install with `gem install <file>` and you're done. HOWTO ===== This API supports the following evaluation tasks: - Loading a Gold Standard that takes a set of documents, queries and corresponding judgements of relevancy (i.e. "Is this document relevant for this query?") - Calculation of the _kappa measure_ for the given gold standard - Loading ranked or unranked query results for a certain query - Calculation of _precision_ and _recall_ for each result - Calculation of the _F-measure_ for weighing precision and recall - Calculation of _mean average precision_ for multiple query results - Calculation of the _11-point precision_ and _average precision_ for ranked query results - Printing of summary tables and results Typically, you will want to use this Gem either standalone or within another application's context. Standalone Usage ================ Call parameters --------------- After installing the Gem (see INSTALLATION), you can always call `retreval` from the commandline. The typical call is: retreval -l <gold-standard-file> -q <query-results> -f <format> -o <output-prefix> Where you have to define the following options: - `gold-standard-file` is a file in a specified format that includes all the judgements - `query-results` is a file in a specified format that includes all the query results in a single file - `format` is the format that the files will use (either "yaml" or "plain") - `output-prefix` is the prefix of output files that will be created Formats ------- Right now, we focus on the formats you can use to load data into the API. Currently, we support YAML files that must adhere to a special syntax. So, in order to load a gold standard, we need a file in the following format: * "query" denotes the query * "documents" these are the documents judged for this query * "id" the ID of the document (e.g. its filename, etc.) * "judgements" an array of judgements, each one with: * "relevant" a boolean value of the judgment (relevant or not) * "user" an optional identifier of the user Example file, with one query, two documents, and one judgement: - query: 12th air force germany 1957 documents: - id: g5701s.ict21311 judgements: [] - id: g5701s.ict21313 judgements: - relevant: false user: 2 So, when calling the program, specify the format as `yaml`. For the query results, a similar format is used. Note that it is necessary to specify whether the result sets are ranked or not, as this will heavily influence the calculations. You can specify the score for a document. By "score" we mean the score that your retrieval algorithm has given the document. But this is not necessary. The documents will always be ranked in the order of their appearance, regardless of their score. Thus in the following example, the document with "07" at the end is the first and "25" is the last, regardless of the score. --- query: 12th air force germany 1957 ranked: true documents: - score: 0.44034874 document: g5701s.ict21307 - score: 0.44034874 document: g5701s.ict21309 - score: 0.44034874 document: g5701s.ict21311 - score: 0.44034874 document: g5701s.ict21313 - score: 0.44034874 document: g5701s.ict21315 - score: 0.44034874 document: g5701s.ict21317 - score: 0.44034874 document: g5701s.ict21319 - score: 0.44034874 document: g5701s.ict21321 - score: 0.44034874 document: g5701s.ict21323 - score: 0.44034874 document: g5701s.ict21325 --- query: 1612 ranked: true documents: - score: 1.0174774 document: g3290.np000144 - score: 0.763108 document: g3201b.ct000726 - score: 0.763108 document: g3400.ct000886 - score: 0.6359234 document: g3201s.ct000130 --- **Note**: You can also use the `plain` format, which will load the gold standard in a different way (but not the results): my_query my_document_1 false my_query my_document_2 true See that every query/document/relevancy pair is separated by a tabulator? You can also add the user's ID in the fourth column if necessary. Running the evaluation ----------------------- After you have specified the input files and the format, you can run the program. If needed, the `-v` switch will turn on verbose messages, such as information on how many judgements, documents and users there are, but this shouldn't be necessary. The program will first load the gold standard and then calculate the statistics for each result set. The output files are automatically created and contain a YAML representation of the results. Calculations may take a while depending on the amount of judgements and documents. If there are a thousand judgements, always consider a few seconds for each result set. Interpreting the output files ------------------------------ Two output files will be created: - `output_avg_precision.yml` - `output_statistics.yml` The first lists the average precision for each query in the query result file. The second file lists all supported statistics for each query in the query results file. For example, for a ranked evaluation, the first two entries of such a query result statistic look like this: --- 12th air force germany 1957: - :precision: 0.0 :recall: 0.0 :false_negatives: 1 :false_positives: 1 :true_negatives: 2516 :true_positives: 0 :document: g5701s.ict21313 :relevant: false - :precision: 0.0 :recall: 0.0 :false_negatives: 1 :false_positives: 2 :true_negatives: 2515 :true_positives: 0 :document: g5701s.ict21317 :relevant: false You can see the precision and recall for that specific point and also the number of documents for the contingency table (true/false positives/negatives). Also, the document identifier is given. API Usage ========= Using this API in another ruby application is probably the more common use case. All you have to do is include the Gem in your Ruby or Ruby on Rails application. For details about available methods, please refer to the API documentation generated by RDoc. **Important**: For this implementation, we use the document ID, the query and the user ID as the primary keys for matching objects. This means that your documents and queries are identified by a string and thus the strings should be sanitized first. Loading the Gold Standard ------------------------- Once you have loaded the Gem, you will probably start by creating a new gold standard. gold_standard = GoldStandard.new Then, you can load judgements into this standard, either from a file, or manually: gold_standard.load_from_yaml_file "my-file.yml" gold_standard.add_judgement :document => doc_id, :query => query_string, :relevant => boolean, :user => John There is a nice shortcut for the `add_judgement` method. Both lines are essentially the same: gold_standard.add_judgement :document => doc_id, :query => query_string, :relevant => boolean, :user => John gold_standard << :document => doc_id, :query => query_string, :relevant => boolean, :user => John Note the usage of typical Rails hashes for better readability (also, this Gem was developed to be used in a Rails webapp). Now that you have loaded the gold standard, you can do things like: gold_standard.contains_judgement? :document => "a document", :query => "the query" gold_standard.relevant? :document => "a document", :query => "the query" Loading the Query Results ------------------------- Now we want to create a new `QueryResultSet`. A query result set can contain more than one result, which is what we normally want. It is important that you specify the gold standard it belongs to. query_result_set = QueryResultSet.new :gold_standard => gold_standard Just like the Gold Standard, you can read a query result set from a file: query_result_set.load_from_yaml_file "my-results-file.yml" Alternatively, you can load the query results one by one. To do this, you have to create the results (either ranked or unranked) and then add documents: my_result = RankedQueryResult.new :query => "the query" my_result.add_document :document => "test_document 1", :score => 13 my_result.add_document :document => "test_document 2", :score => 11 my_result.add_document :document => "test_document 3", :score => 3 This result would be ranked, obviously, and contain three documents. Documents can have a score, but this is optional. You can also create an Array of documents first and add them altogether: documents = Array.new documents << ResultDocument.new :id => "test_document 1", :score => 20 documents << ResultDocument.new :id => "test_document 2", :score => 21 my_result = RankedQueryResult.new :query => "the query", :documents => documents The same applies to `UnrankedQueryResult`s, obviously. The order of ranked documents is the same as the order in which they were added to the result. The `QueryResultSet` will now contain all the results. They are stored in an array called `query_results`, which you can access. So, to iterate over each result, you might want to use the following code: query_result_set.query_results.each_with_index do |result, index| # ... end Or, more simply: for result in query_result_set.query_results # ... end Calculating statistics ---------------------- Now to the interesting part: Calculating statistics. As mentioned before, there is a conceptual difference between ranked and unranked results. Unranked results are much easier to calculate and thus take less CPU time. No matter if unranked or ranked, you can get the most important statistics by just calling the `statistics` method. statistics = my_result.statistics In the simple case of an unranked result, you will receive a hash with the following information: * `precision` - the precision of the results * `recall` - the recall of the results * `false_negatives` - number of not retrieved but relevant items * `false_positives` - number of retrieved but nonrelevant * `true_negatives` - number of not retrieved and nonrelevantv items * `true_positives` - number of retrieved and relevant items In case of a ranked result, you will receive an Array that consists of _n_ such Hashes, depending on the number of documents. Each Hash will give you the information at a certain rank, e.g. the following to lines return the recall at the fourth rank. statistics = my_ranked_result.statistics statistics[3][:recall] In addition to the information mentioned above, you can also get for each rank: * `document` - the ID of the document that was returned at this rank * `relevant` - whether the document was relevant or not Calculating statistics with missing judgements ---------------------------------------------- Sometimes, you don't have judgements for all document/query pairs in the gold standard. If this happens, the results will be cleaned up first. This means that every document in the results that doesn't appear to have a judgement will be removed temporarily. As an example, take the following results: * A * B * C * D Our gold standard only contains judgements for A and C. The results will be cleaned up first, thus leading to: * A * C With this approach, we can still provide meaningful results (for precision and recall). Other statistics ---------------- There are several other statistics that can be calculated, for example the **F measure**. The F measure weighs precision and recall and has one parameter, either "alpha" or "beta". Get the F measure like so: my_result.f_measure :beta => 1 If you don't specify either alpha or beta, we will assume that beta = 1. Another interesting measure is **Cohen's Kappa**, which tells us about the inter-agreement of assessors. Get the kappa statistic like this: gold_standard.kappa This will calculate the average kappa for each pairwise combination of users in the gold standard. For ranked results one might also want to calculate an **11-point precision**. Just call the following: my_ranked_result.eleven_point_precision This will return a Hash that has indices at the 11 recall levels from 0 to 1 (with steps of 0.1) and the corresponding precision at that recall level.
# foundationallib <h2>Finally, a cross-platform, portable, well-designed, secure, robust, maximally-efficient C foundational library — Making Engineering And Computing Fast, Secure, Responsive And Easy.</h2> <br> <h2><i>Library Uses - What It Does, What It Is, And What It Is A Solution For</i></h2> <ul class="features-list"> <li><strong>Enables better Engineering Solutions and Security broadly and foundationally where Software Creation or Development or Script Creation is concerned - whether this be on a local, business, governmental or international basis, and makes things easier - and Computing in General.</strong> Don't Reinvent the Wheel - Use Good Wheels - Be Safe And Secure.</li> <br> <li><strong>Enables a free-flowing dynamic computer usage that you need, deserve and should have, simply because you have a computer. With full speed and with robustness. You deserve to be able to use your computer wholly and fully, with proper and fast operations.</strong></li> <br><li><strong>Enables flexibility and power - makes C accessible to the masses (and faster and more secure) with easy usage and strives to bring people up, not degrade the character or actions of people.</strong> This is a fundamental and unequivocal philosophy difference between this library and many subsections of Software Engineering and the mainstream engineering establishment. For instance, in Python, you cannot read a file easily – you have to read it line-by-line or open a file, read the lines, then close it. With this library, you can efficiently read 10,000 files in one function call. This library gives power. Any common operation, there ought to be a powerful function for.<br><br>We should not bitch around with assembly when we don't want to; we should also have full speed. Some old "solutions" deliver neither, then culturally degrade programmers because their tools are bad - actually, it just degrades programmers, and gives them bad tools. COBOL is an example ...<br><br>Human technology is about empowerment – people must fight for it to be empowerment, we don't have time to have AI systems kill us because we want to have bad tools and be weak. We must fight.</li> </ul> <br> <ul> <h2><i>About Foundationallib</i></h2> <li>→<strong>Cross platform</strong> - works perfectly in embedded, server, desktop, and all platforms - tested for Windows and UNIX - 64-bit and 32-bit, includes a 3-aspect test suite, with more to come.</li> <li>→<strong>Bug free. Reliable. Dependable. Secure. Tested well.</strong></li> <li>→<strong>Zero Overhead</strong> - Only 1 byte due to the power of the error handling, can be configured will full power.</li> <li>→<strong>Static Inline Functions if you want them</strong> (optional) - Eliminating function call overhead to 0 if you wish, for improved performance.</li> <li>→<strong>Custom allocators</strong> - if you want it.</li> <li>→<strong>Custom error handling</strong> - if you want it.</li> <li>→<strong>Safe functions</strong> warn the programmer about NULL values and unused return values. Can be configured to not compile if not Secure. Optional null-check macros in every library function. Does not use any of <code>"gets", "fgets", "strcpy", "strcat", "sprintf", "vsprintf", "scanf", "fscanf", "system", "chown", "chmod", "chgrp", "alloca", "execl", "execle", "execlp", "execv", "execve", "execvp", "bcopy", "bzero"</code>. You can configure it to never use any unsafe functions.</li> <li>→<strong>Portable</strong> - works on all platforms, using platform specific features (using #ifdefs) to make functions better and faster.</li> <li>→<strong>Multithreading support</strong> (optional), with list_comprehension_multithreaded (accepts any number of threads, works in parallel using portable C11 threads)</li> <li>→<strong>Networking support</strong> (optional), using libcurl - making it extremely easy to download websites and arrays of websites - features other languages do not have.</li> <li>→Very good and thorough <strong>Error Handling</strong> and <strong>allocation overflow</strong> checking (good for <strong>Security and Robustness</strong>) in the functions. Allows the programmer to dynamically choose to catch all errors in the functions with a handler (default or custom), or to ignore them. No need to ALWAYS say "if (.....) if you don't want to. Can be changed at runtime.</li> <li>→<strong>Public Domain</strong> so you make the code how you want. (No need to "propitiate" to some "god" of some library).</li> <li>→<strong>Minimal abstractions or indirection of any kind or needless slow things that complicate things</strong> - macros, namespace collision, typedefs, structs, object-orientation messes, slow compilation times, bloat, etc., etc.</li> <li>→<strong>No namespace pollution</strong> - you can generate your <span style=font-style:normal;><b>own version</b></span> with any prefix you like!</li> <li>→<strong>Relies <span style=font-style:normal;>minimally</span> on C libraries - it can be fully decoupled from LIB C and can be statically linked.</strong></li> <li>→<span style=font-style:normal;><b>Very small</b></span> - 13K Lines of Code (including Doxygen comments and following of Best Practices)</li> <li>→<strong>No Linkage Issues or dependency hell</strong></li> <li>→<strong>Thorough and clear documentation</strong>, with examples of usage.</li> <li>→<strong>No licensing restrictions whatsoever - use it for your engineering project, your startup, your Fortune 500 company, your personal project, your throw-away script, your government.</strong></li> <li>→<strong>Makes C like Python or Perl or Ruby in many ways - or more easy</strong></li> <li>→<strong>Easy Straightforward Transpilation Support</strong> - to make current code, much faster - all without any bloat (See transpile_slow_scripting_into_c.rb). <li><h4>In many cases, there is now a direct mapping of functions from other languages into optimized C. See the example script in this repository. This makes optimizing your Python / Perl / Ruby / PHP etc. script very easy, either manually or through the use of AI.</h4></li> </ul> </p> </div> <div class=pane style='border: 0;border-right: 1px dotted rgb(200, 200, 200); background-color: rgb(255, 255, 190);'> <div class="library-details"><h2 style=color:green;><i>Foundationallib Features</i></h2> <p class=feature> <strong>Functional Programming Features</strong> - <code>map, reduce, filter,</code> List Comprehensions in C and much more!</p> <p class=feature><strong>Expands C's Primitives for easy manipulation of data types</strong> such as Arrays, Strings, <code>Dict</code>, <code>Set</code>, <code>FrozenDict</code>, <code>FrozenSet</code> - <strong>and enables easy manipulation, modification, alteration, comparison, sorting, counting, IO (printing) and duplication of these at a very comfortable level</strong> - something very, very rare in C or C++, <i>all without any overhead.</i></p> <p class=feature><strong>More comfortable IO</strong> - read and write entire files with ease, and convert complex types into strings or print them on the screen with ease. </p> <p class=feature><strong>A powerful general purpose Foundational Library</strong> - <i>which has anything and everything you need</i> - from <code>replace_all()</code> to <code>replace_memory()</code> to <code>find_last_of()</code> to to <code>list_comprehension()</code> to <code>shellescape()</code> to <code>read_file_into_string()</code> to <code>string_to_json()</code> to <code>string_to_uppercase()</code> to <code>to_title_case()</code> to <code>read_file_into_array()</code> to <code>read_files_into_array()</code> to <code>map()</code> to <code>reduce()</code> to <code>filter()</code> to <code>list_comprehension_multithreaded()</code> to <code>frozen_dict_new_instance()</code> to <code>backticks()</code> - everything you would want to make quick and optimally efficient C programs, this has it.</p> <div style='height: 1px; border: 0;border-bottom: 1px dashed rgb(200, 200, 200);'></div> <p class=performance><span>Helps to make programs hundreds of times faster than other languages with similar ease of creation.</span> <hr> <p class=feature><strong>Easily take advantage of CPU cores with list_comprehension_multithreaded()</strong>.<br><br>You can specify the number of threads, the transform and the filter functions, and this will transform your data - all in parallel. Don't have a multithreaded environment? Then disable it (set the flag).</p> <hr> <h3>You don't want to be reinventing the wheel and hoping that your memory allocation is secure enough - and then failing. <strong>Security Is Paramount.</strong></h3> <h3>You don't want to be waiting <span style='color:rgb(240, 0, 0);'>a day</span> for an operation to complete when it could take <span style='color:rgb(30, 30, 255);'>less than an hour</span>.</h3> <br><p>This library is founded on very strong and unequivocal goals and philosophy. In fact, I have written many articles about the foundation of this library and more relevantly the broader context. See the Articles folder - for some of the foundation of this library.</p> <br><p>This library is an ideal and a dream - not just a Software Library. As such, I would highly suggest that you support me in this mission. Even if it's different from the status quo. Are you a Rust or Zig fan? Then make a Rust or Zig version of this ideal. Let's go. Give me an email.</p> </div> </div> <br> No Copyright - Public Domain - 2023, Gregory Cohen <gregorycohennew@gmail.com> DONATION REQUEST: If this free software has helped you and you find it valuable, please consider making a donation to support the ongoing development and maintenance of this project. Your contribution helps ensure the availability of this library to the community and encourages further improvements. Donations can be made at: https://www.paypal.com/paypalme/cfoundationallib Note: The best way to contact me is through email, not social media. Please feel very free to email me if you want to express feedback, suggest an improvement, desire to collaborate on this free and open source project, want to support me, or want to create something great. Complacency and obstructionism and whining are not tolerated. I desire to make this library the best theoretically possible, so please, let us connect. <h1>This code is in the public domain, fully. You can do whatever you want with it. See docs.html for API reference.  </h1> <h1>Here's some examples of some things you can do easily with Foundationallib.<br><br> <h3>Use it for scripting purposes...</h3> </h1>  <h1>Take control of the Web - in C.<br><br></h1> 
# foundationallib <h2>Finally, a cross-platform, portable, well-designed, secure, robust, maximally-efficient C foundational library — Making Engineering And Computing Fast, Secure, Responsive And Easy.</h2> <br> <ul class="features-list"> <li><strong>Enables better Engineering Solutions and Security broadly and foundationally where Software Creation or Development or Script Creation is concerned - whether this be on a local, business, governmental or international basis, and makes things easier - and Computing in General.</strong> Don't Reinvent the Wheel - Use Good Wheels - Be Safe And Secure.</li> <br> <li><strong>Enables a free-flowing dynamic computer usage that you need, deserve and should have, simply because you have a computer. With full speed and with robustness. You deserve to be able to use your computer wholly and fully, with proper and fast operations.</strong></li> <br><li><strong>Enables flexibility and power - makes C accessible to the masses (and faster and more secure) with easy usage and strives to bring people up, not degrade the character or actions of people.</strong> This is a fundamental and unequivocal philosophy difference between this library and many subsections of Software Engineering and the mainstream engineering establishment. For instance, in Python, you cannot read a file easily – you have to read it line-by-line or open a file, read the lines, then close it. With this library, you can efficiently read 10,000 files in one function call. This library gives power. Any common operation, there ought to be a powerful function for.<br><br>We should not bitch around with assembly when we don't want to; we should also have full speed. Some old "solutions" deliver neither, then culturally degrade programmers because their tools are bad - actually, it just degrades programmers, and gives them bad tools. COBOL is an example ...<br><br>Human technology is about empowerment – people must fight for it to be empowerment, we don't have time to have AI systems kill us because we want to have bad tools and be weak. We must fight.</li> </ul> <br> <ul> <h2>About Foundationallib</h2> <li>→<strong>Cross platform</strong> - works perfectly in embedded, server, desktop, and all platforms - tested for Windows and UNIX - 64-bit and 32-bit, includes a 3-aspect test suite, with more to come.</li> <li>→<strong>Bug free. Reliable. Dependable. Secure. Tested well.</strong></li> <li>→<strong>Zero Overhead</strong> - Only 1 byte due to the power of the error handling, can be configured will full power.</li> <li>→<strong>Static Inline Functions if you want them</strong> (optional) - Eliminating function call overhead to 0 if you wish, for improved performance.</li> <li>→<strong>Custom allocators</strong> - if you want it.</li> <li>→<strong>Custom error handling</strong> - if you want it.</li> <li>→<strong>Safe functions</strong> warn the programmer about NULL values and unused return values. Can be configured to not compile if not Secure. Optional null-check macros in every library function. Does not use any of <code>"gets", "fgets", "strcpy", "strcat", "sprintf", "vsprintf", "scanf", "fscanf", "system", "chown", "chmod", "chgrp", "alloca", "execl", "execle", "execlp", "execv", "execve", "execvp", "bcopy", "bzero"</code>. You can configure it to never use any unsafe functions.</li> <li>→<strong>Portable</strong> - works on all platforms, using platform specific features (using #ifdefs) to make functions better and faster.</li> <li>→<strong>Multithreading support</strong> (optional), with list_comprehension_multithreaded (accepts any number of threads, works in parallel using portable C11 threads)</li> <li>→<strong>Networking support</strong> (optional), using libcurl - making it extremely easy to download websites and arrays of websites - features other languages do not have.</li> <li>→Very good and thorough <strong>Error Handling</strong> and <strong>allocation overflow</strong> checking (good for <strong>Security and Robustness</strong>) in the functions. Allows the programmer to dynamically choose to catch all errors in the functions with a handler (default or custom), or to ignore them. No need to ALWAYS say "if (.....) if you don't want to. Can be changed at runtime.</li> <li>→<strong>Public Domain</strong> so you make the code how you want. (No need to "propitiate" to some "god" of some library).</li> <li>→<strong>Minimal abstractions or indirection of any kind or needless slow things that complicate things</strong> - macros, namespace collision, typedefs, structs, object-orientation messes, slow compilation times, bloat, etc., etc.</li> <li>→<strong>No namespace pollution</strong> - you can generate your <span style=font-style:normal;><b>own version</b></span> with any prefix you like!</li> <li>→<strong>Relies <span style=font-style:normal;>minimally</span> on C libraries - it can be fully decoupled from LIB C and can be statically linked.</strong></li> <li>→<span style=font-style:normal;><b>Very small</b></span> - 13K Lines of Code (including Doxygen comments and following of Best Practices)</li> <li>→<strong>No Linkage Issues or dependency hell</strong></li> <li>→<strong>Thorough and clear documentation</strong>, with examples of usage.</li> <li>→<strong>No licensing restrictions whatsoever - use it for your engineering project, your startup, your Fortune 500 company, your personal project, your throw-away script, your government.</strong></li> <li>→<strong>Makes C like Python or Perl or Ruby in many ways - or more easy</strong></li> <li>→<strong>Easy Straightforward Transpilation Support</strong> - to make current code, much faster - all without any bloat (See transpile_slow_scripting_into_c.rb). <li><h4>In many cases, there is now a direct mapping of functions from other languages into optimized C. See the example script in this repository. This makes optimizing your Python / Perl / Ruby / PHP etc. script very easy, either manually or through the use of AI.</h4></li> </ul> </p> </div> <div class=pane style='border: 0;border-right: 1px dotted rgb(200, 200, 200); background-color: rgb(255, 255, 190);'> <div class="library-details"><h2 style=color:green;>Foundationallib Features</h2> <p class=feature> <strong>Functional Programming Features</strong> - <code>map, reduce, filter,</code> List Comprehensions in C and much more!</p> <p class=feature><strong>Expands C's Primitives for easy manipulation of data types</strong> such as Arrays, Strings, <code>Dict</code>, <code>Set</code>, <code>FrozenDict</code>, <code>FrozenSet</code> - <strong>and enables easy manipulation, modification, alteration, comparison, sorting, counting, IO (printing) and duplication of these at a very comfortable level</strong> - something very, very rare in C or C++, <i>all without any overhead.</i></p> <p class=feature><strong>More comfortable IO</strong> - read and write entire files with ease, and convert complex types into strings or print them on the screen with ease. </p> <p class=feature><strong>A powerful general purpose Foundational Library</strong> - <i>which has anything and everything you need</i> - from <code>replace_all()</code> to <code>replace_memory()</code> to <code>find_last_of()</code> to to <code>list_comprehension()</code> to <code>shellescape()</code> to <code>read_file_into_string()</code> to <code>string_to_json()</code> to <code>string_to_uppercase()</code> to <code>to_title_case()</code> to <code>read_file_into_array()</code> to <code>read_files_into_array()</code> to <code>map()</code> to <code>reduce()</code> to <code>filter()</code> to <code>list_comprehension_multithreaded()</code> to <code>frozen_dict_new_instance()</code> to <code>backticks()</code> - everything you would want to make quick and optimally efficient C programs, this has it.</p> <div style='height: 1px; border: 0;border-bottom: 1px dashed rgb(200, 200, 200);'></div> <p class=performance><span>Helps to make programs hundreds of times faster than other languages with similar ease of creation.</span> <hr> <p class=feature><strong>Easily take advantage of CPU cores with list_comprehension_multithreaded()</strong>.<br><br>You can specify the number of threads, the transform and the filter functions, and this will transform your data - all in parallel. Don't have a multithreaded environment? Then disable it (set the flag).</p> <hr> <h3>You don't want to be reinventing the wheel and hoping that your memory allocation is secure enough - and then failing. <strong>Security Is Paramount.</strong></h3> <h3>You don't want to be waiting <span style='color:rgb(240, 0, 0);'>a day</span> for an operation to complete when it could take <span style='color:rgb(30, 30, 255);'>less than an hour</span>.</h3> <br><p>This library is founded on very strong and unequivocal goals and philosophy. In fact, I have written many articles about the foundation of this library and more relevantly the broader context. See the Articles folder - for some of the foundation of this library.</p> <br><p>This library is an ideal and a dream - not just a Software Library. As such, I would highly suggest that you support me in this mission. Even if it's different from the status quo. Are you a Rust or Zig fan? Then make a Rust or Zig version of this ideal. Let's go. Give me an email.</p> </div> </div> <br> No Copyright - Public Domain - 2023, Gregory Cohen <gregorycohennew@gmail.com> DONATION REQUEST: If this free software has helped you and you find it valuable, please consider making a donation to support the ongoing development and maintenance of this project. Your contribution helps ensure the availability of this library to the community and encourages further improvements. Donations can be made at: https://www.paypal.com/paypalme/cfoundationallib Note: The best way to contact me is through email, not social media. Please feel very free to email me if you want to express feedback, suggest an improvement, desire to collaborate on this free and open source project, want to support me, or want to create something great. Complacency and obstructionism and whining are not tolerated. I desire to make this library the best theoretically possible, so please, let us connect. <pre><code> Mirror Links Blog - https://foundationallib.wordpress.com/ Github - https://github.com/gregoryc/foundationallib Ruby Gem Mirror - https://rubygems.org/gems/foundational_lib Ruby Gem Mirror - https://rubygems.org/gems/foundational_lib2 Library Instagram - https://www.instagram.com/foundationallib Google Drive Mirrors ZIP - https://drive.google.com/file/d/1bK2njCRsH4waTm4LP16sloPQawk7JIR5/view?usp=sharing TAR.GZ - https://drive.google.com/file/d/1RCA1yy9R3cEqI_X9Lv0fxqh-zgNCK005/view?usp=sharing TAR.BZ2 - https://drive.google.com/file/d/1ljdlI_fEnMS_X5WmuhI1qavhgseWlD5j/view?usp=sharing </code></pre> <h1>This code is in the public domain, fully. You can do whatever you want with it. See docs.html for API reference.  </h1> <h1>Here's some examples of some things you can do easily with Foundationallib.<br><br> <h3>Use it for scripting purposes...</h3> </h1>  <h1>Take control of the Web - in C.<br><br></h1> 
No description provided.
No description provided.
No description provided.
No description provided.