tiktoken/tiktoken/model.py at 46287bfa493f8ccca4d927386d7ea9cc20487525 ยท openai/tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models. - openai/tiktoken
github.com github.com