Turning Text into Numbers
Computers work with numbers, so text must be converted. The first step is tokenising: splitting text into pieces. Each distinct token gets a number from a vocabulary. Then a sentence becomes a list of numbers a model can use. Large language models use sub-word tokens, so a rare word may be split into several pieces.
Examplepython
Quick check
Why do models need a vocabulary?
Your turn: write the code
Print the token ids of the sentence 'the dog and the cat' using the vocabulary built from it (words sorted alphabetically).
Finish the quick check (and any code exercise) to complete this lesson, or mark it done yourself.
Tip: use the left and right arrow keys to move between lessons.