r/MachineLearning 🤖 Ai 👁 0

I animated what GPT-4o's tokenizer does to "strawberry": real token ids, a tiny BPE run, and why the space belongs to the word (6 min video) [D]

I made this explainer for my small channel and would like feedback from people learning this, especially if anything is wrong. It follows one word the whole way: "How many r's are in strawberry?" is 8 tokens in o200k_ba

📄

This source provides headlines only. Use the button below to read the complete article on the original site.

📰 Read the original article on r/MachineLearning

Originally published by r/MachineLearning. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.