https://www.amazon.science/publications/byteflow-language-modeling-through-adaptive-byte-compression-without-a-tokenizer
ByteFlow: Language modeling through adaptive byte compression without a tokenizer - Amazon Science
Modern language models (LMs) still rely on fixed, pre-defined subword tokenizations. Once a tokenizer is trained, the LM can only operate at this fixed level...
language modelingamazon scienceadaptivebytecompression
https://ropensci.r-universe.dev/hunspell
hunspell: High-Performance Stemmer, Tokenizer, and Spell Checker
high performancespell checkerhunspelltokenizer
https://www.linkedin.com/pulse/tokenizer-currently-undergoing-technical-update-the-tokenizer-7vwtf
The Tokenizer is currently undergoing a technical update
Jul 2, 2025 - The Tokenizer will be unavailable during the month of July for a summer holiday technical update. In the meantime, we highly recommend the newly published...
tokenizercurrentlytechnicalupdate
https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizer
tf.keras.preprocessing.text.Tokenizer | TensorFlow v2.16.1
tfkeraspreprocessingtexttokenizer
https://github.blog/ai-and-ml/llms/so-many-tokens-so-little-time-introducing-a-faster-more-flexible-byte-pair-tokenizer/
So many tokens, so little time: Introducing a faster, more flexible byte-pair tokenizer - The...
We released a new open source byte-pair tokenizer that is faster and more flexible than popular alternatives.
manytokenslittletimeintroducing