tiktoken/tiktoken/model.py at 0.4.0 ยท openai/tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models. - openai/tiktoken
github.com github.com