Hugging Face Tokenizers
Hugging Face tokenizers convert text into the numbers a model actually reads. Models never see letters; they see token IDs, integers that index into a fixed vocabulary. Every model comes with its own tokenizer, and you must always use the matching one.
Tokens Are Sub-Words
Modern tokenizers (BPE, WordPiece, SentencePiece) split rare words into pieces and keep common words whole. That keeps the vocabulary small while still handling any input text.
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
print(tok.tokenize("Tokenization is very useful"))['token', '##ization', 'is', 'very', 'useful']
The ## prefix is how WordPiece marks a piece that continues the previous word. Other tokenizers use different markers, for example a special space symbol at the start of a word.
Encoding Text to IDs
enc = tok("Hello world")
print(enc["input_ids"])
print(enc["attention_mask"])
print(tok.decode(enc["input_ids"]))[101, 7592, 2088, 102] [1, 1, 1, 1] [CLS] hello world [SEP]
Calling the tokenizer returns a dictionary. input_ids are the token numbers, including special tokens such as [CLS] (101) and [SEP] (102) that BERT expects. attention_mask marks real tokens with 1.
Padding and Truncation
A batch must be a rectangle, so shorter sentences are padded and overly long ones are cut. return_tensors="pt" gives PyTorch tensors ready for the model.
batch = tok(
["Short one", "This sentence is quite a bit longer"],
padding=True, truncation=True, max_length=16, return_tensors="pt",
)
print(batch["input_ids"].shape)
print(batch["attention_mask"])torch.Size([2, 9])
tensor([[1, 1, 1, 1, 0, 0, 0, 0, 0],
[1, 1, 1, 1, 1, 1, 1, 1, 1]])The zeros in the attention mask tell the model to ignore padding positions.
Counting Tokens for LLMs
Context windows and API prices are measured in tokens, not words. A rough rule for English is about 0.75 words per token, but always measure with the real tokenizer of the model you use:
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
text = "Hugging Face makes open models easy to use."
print(len(tok(text)["input_ids"]))10 (sample — counts vary by tokenizer)
Tokenizer Tips
- Fast tokenizers (written in Rust, from the tokenizers library) are the default and much quicker on large batches.
- Decoder-only LLMs are usually padded on the left for batched generation: tok.padding_side = "left".
- If a model has no pad token, a common fix is tok.pad_token = tok.eos_token.
- For chat models, build prompts with tok.apply_chat_template instead of writing special tokens by hand.
Frequently asked questions
What does a tokenizer do in Hugging Face?
It splits text into tokens, maps them to integer IDs, adds special tokens, and pads or truncates batches so the model can process them. It can also decode IDs back into text.
What is an attention mask?
A list of 1s and 0s the same length as input_ids. 1 means a real token, 0 means padding the model should ignore.
Can I use one model's tokenizer with another model?
Generally no. Each model is trained with a specific vocabulary, and mismatched IDs produce garbage. Always load the tokenizer from the same repository as the model.