Prompt Pad: What Are Tokens?

Your AI doesn't read your prompt word for word. It reads tokens.

In this episode of Prompt Pad, @rosendo_pili explains tokens, the atomic units of text that an AI actually reads and calculates.

The tip in a nutshell

Before an AI reads your prompt, your text is converted into numbers using a process called tokenization. Each token becomes a number, and those numbers are what the model actually calculates with to generate a response.

Everything we've covered in Prompt Pad so far, from framing a persona to few-shot examples, reaches the model as tokens. Understanding them gives you a clearer picture of what happens between hitting enter and getting an answer.

Try it yourself

Take a prompt you use often and think about how it might be split into tokens. Share your guesses (or any surprises) in the comments 👇

📣 Help us spread the word

Know someone curious about how AI really works? Send them this video. And subscribe so you don't miss the next Prompt Pad tip:

• ▶️ YouTube: @MajorLeagueHacking
• 📸 Instagram: @mlhacks
• 🐦 X: @mlhacks
• 💼 LinkedIn: Major League Hacking
• 💬 Discord: Join the MLH community