BETAmodules.com is in beta — open to partnerships & joint ventures.Build with us
modules.com
⌘K

TokenizerProjectUT

v0.0.1RubyGems· Ruby

A simple multilingual tokenizer for NLP tasks. This tool provides a CLI and a library for linguistic tokenization which is an anavoidable step for many HLT (human language technology) tasks in the preprocessing phase for further syntactic, semantic and other higher level processing goals. Use it for tokenization of German, English and French texts.

The verdict
Abandoned. Last published 14 years ago. No recent activity — look for a maintained alternative.
No recent activity — look for a maintained alternative.
Live from the RubyGems registry · derived rules, not AI
How it scores
MaintenanceAbandoned
PopularityNiche
SecurityClean
LicenseUnknown
DepsZero deps
Maintenance
Last published 14 years ago.
Popularity
6 downloads / week
Security
No known advisories for this version (OSV).
License
No license declared.
Dependencies
No runtime dependencies
Recent releases
  • 0.0.114 years ago
TokenizerProjectUT — A simple multilingual tokenizer for NLP tasks. This tool provides a CLI and a library for linguistic tokenization which is an anavoidable step for many HLT (human language technology) tasks in the preprocessing phase for further syntactic, semantic and other higher level processing goals. Use it for tokenization of German, English and French texts. (Ruby / RubyGems) · Modules