11
Understanding Generative AI: Part One - Tokenizer
(scorpil.com)
Unofficial ChatGPT community to discuss anything ChatGPT
This is really helpful, thanks. I followed up until the part about token IDs.
How and why is token text converted to an ID? Is the ID for a specific word always the same?
What are tokens and how to count them?