tokenizer~cadmiumcr

A collection of tokenizers for Crystal

Help categorize this shard!

1.0.0 Yanked release released over 5 years ago

cadmiumcr/tokenizer

over 4 years ago

4 2 2

Source

Crystal Lang Tokenizer

A tool for buffering and tokenizing streaming inputs.

tokenizer:
  github: cadmiumcr/tokenizer
  version: ~> 1.0.0

Crystal none

Dependencies 0

Development Dependencies 1

ameba
```
{'github' => 'veelenga/ameba'}
```

Dependents 2

cadmium_classifier (1 dependents)
linguist (1 dependents)

Similar shards

tokenizer spider-gazelle/tokenizer

Releases

Last synced about 10 hours ago.