tiktoken/tiktoken/model.py at 0.4.0 ยท openai/tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models. - openai/tiktoken