tokenizer

A simple from-scratch word-level tokenizer built for educational purposes, to help understand the basics of how tokenizers work in large language models (LLMs): text preprocessing, vocabulary construction, encoding, and decoding.

Features

• Splits raw text using a regular expression, separating punctuation (,.:;?_!"()', --) from words
• Automatically builds a vocabulary (vocab) from the corpus, assigning a unique integer id to each token
• Implements simple encode / decode methods:
  • encode: converts text into a sequence of ids, falling back to the <UNK> id for out-of-vocabulary (OOV) words
  • decode: converts a sequence of ids back into text, fixing up extra whitespace before punctuation
• Ships with a sample corpus input.txt (Tiny Shakespeare, ~1.1MB of Shakespeare's plays)

Requirements

pip install transformers

Note: the script imports transformers.AutoTokenizer, but then defines its own class with the same name (AutoTokenizer) for demonstration purposes. At runtime, the custom class defined in this file is the one actually used.

Usage

Make sure input.txt is in the same directory as tokenizer.py, then run:

python tokenizer.py

The script will print, in order:

• The first 50 (token, id) pairs from the vocabulary
• The total character count of the corpus and a preview of the first 1000 characters
• The preprocessed token list and its length
• The vocabulary size (vocab_size)
• The encoded id sequence for a sample text
• The decoded text reconstructed from those ids

Example

tokenizer = AutoTokenizer(vocab)
text = 'He yelled, "Hello, world!" and then asked, "How are you?"'
ids = tokenizer.encode(text)
print(ids)
print(tokenizer.decode(ids))

Project structure

tokenizer/
├── tokenizer.py   # Tokenizer implementation and demo script
├── input.txt      # Corpus used for training/demo (Tiny Shakespeare)
└── README.md

Limitations

This is a simplified implementation intended for learning purposes, with known limitations:

• Purely rule-based splitting on whitespace/punctuation; no subword tokenization, so the vocabulary grows unbounded with the corpus
• All out-of-vocabulary words map to the same <UNK> id, losing information
• No case normalization or special handling of numbers/special symbols

License

This project is open-sourced under the MIT License.