tokensize
v1.0.0npm· JavaScriptThe `tokenizer` function uses the `js-tiktoken` library to encode the input string into tokens using the GPT-2 encoding scheme. It then decodes the tokens back into strings, maps the tokens to their positions in the input string using the `mapTokensToChun
The verdict
Abandoned. Last published 2 years ago. No recent activity — look for a maintained alternative.
No recent activity — look for a maintained alternative.
Live from the npm registry · derived rules, not AI
How it scores
MaintenanceAbandoned
PopularityUnknown
SecurityClean
LicensePermissive
DepsLean
Maintenance
Last published 2 years ago.
Popularity
Download count unavailable.
Security
No known advisories for this version (OSV).
License
ISC
Dependencies
1 direct dependency
Recent releases
- 1.0.02 years ago