Microsoft.ML.Tokenizers contains the implmentation of the tokenization used in the NLP transforms.