Understanding the Llama 3 Tokenizer

Aston Zhang, research scientist working on Llama at Meta discusses the new tokenizer in Meta Llama 3.