https://www.mindspore.cn/docs/en/r2.0.0-alpha/note/api_mapping/pytorch_diff/SentencePieceTokenizer_Out_INT.html
Function differences with torchtext.data.functional.sentencepiece_numericalizer | MindSpore...
functiondifferencesdatamindspore
https://aclanthology.org/D18-2012/
SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text...
Taku Kudo, John Richardson. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2018.