GitHub - openai/tiktoken: tiktoken is a fast BPE tokeniser for use with OpenAI's models.

tiktoken is a fast BPE tokeniser for use with OpenAI's models. - openai/tiktoken
github.com github.com