GitHub - microsoft/Tokenizer: Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.

Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs. - microsoft/Tokenizer
github.com github.com