tiktoken/tiktoken/model.py at main ยท openai/tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models. - openai/tiktoken
github.com github.com