This item only defines dependencies and configuration. Check the Configuration panel on the right for more details.
1 npm package • 3 registry items
Configuration
Name
tokenizer-preview
Type
UI
Description
A tokenizer playground for input text — alternating-tinted token chunks from a deterministic built-in splitter or your own, token/char/word counts, a live cost estimate at an editable per-Mtok rate, and markers for whitespace and unusually long tokens.
Files
1
Dependencies
npm packages
lucide-react
Registry items
input
label
utils
Registry Items
This item has no files to display
This item only defines dependencies and configuration. Check the Configuration panel on the right for more details.
1 npm package • 3 registry items
Configuration
Name
tokenizer-preview
Type
UI
Description
A tokenizer playground for input text — alternating-tinted token chunks from a deterministic built-in splitter or your own, token/char/word counts, a live cost estimate at an editable per-Mtok rate, and markers for whitespace and unusually long tokens.