BETAmodules.com is in beta — open to partnerships & joint ventures.Build with us
modules.com
⌘K

tokenizer

v0.3.0RubyGems· Ruby

A simple multilingual tokenizer for NLP tasks. This tool provides a CLI and a library for linguistic tokenization which is an anavoidable step for many HLT (human language technology) tasks in the preprocessing phase for further syntactic, semantic and other higher level processing goals. Use it for tokenization of German, English and French texts.

The verdict
Abandoned. Last published 10 years ago. No recent activity — look for a maintained alternative.
No recent activity — look for a maintained alternative.
Live from the RubyGems registry · derived rules, not AI
How it scores
MaintenanceAbandoned
PopularityNiche
SecurityClean
LicensePermissive
DepsZero deps
Maintenance
Last published 10 years ago.
Popularity
352 downloads / week
Security
No known advisories for this version (OSV).
License
MIT
Dependencies
No runtime dependencies
Recent releases
  • 0.3.010 years ago
  • 0.2.010 years ago
  • 0.1.210 years ago
  • 0.1.114 years ago
  • 0.1.015 years ago
  • 0.0.1.prealpha15 years ago
tokenizer — A simple multilingual tokenizer for NLP tasks. This tool provides a CLI and a library for linguistic tokenization which is an anavoidable step for many HLT (human language technology) tasks in the preprocessing phase for further syntactic, semantic and other higher level processing goals. Use it for tokenization of German, English and French texts. (Ruby / RubyGems) · Modules