https://developers.openai.com/cookbook/examples/how_to_count_tokens_with_tiktoken
How to count tokens with Tiktoken
tiktoken is a fast open-source tokenizer by OpenAI. Given a text string (e.g., "tiktoken is great!") and an encoding (e.g., "cl100k_base"),
how to counttokenstiktoken