Text to Tensors
Write a byte-pair-encoding tokenizer, then turn token ids into batches and embeddings.
0 of 2 complete
- ▶ Up next20 min+50 XPTokens and byte-pair encoding: write a tokenizerWhy models use sub-word tokens, how byte-pair encoding learns merges from text, and a complete BPE trainer, encoder…
- Pro16 min+50 XPDatasets, batches and embeddings: from token ids to vectorsHow language-model training data is shaped: next-token targets as a one-position shift, context windows and batches…