https://research.google/blog/a-fast-wordpiece-tokenization-system/
A Fast WordPiece Tokenization System
Posted by Xinying Song, Staff Software Engineer and Denny Zhou, Senior Staff Research Scientist, Google Research Tokenization is a fundamental pre-...
fasttokenizationsystem
https://copr.fedorainfracloud.org/coprs/iucar/cran/package/R-CRAN-wordpiece.data/
Package R-CRAN-wordpiece.data in iucar/cran
data inpackager