Skip to content
UPEEC

Text Preprocessing and Tokenization

Clean text and split it into tokens (words or sub-words) a model can process.

Easy
Comments 0

Modern models use sub-word tokenizers (BPE, WordPiece); classic pipelines lowercase, strip punctuation and remove stop-words.

Spotted a mistake or want to add detail? Suggest an improvement — Content Contributors' edits are reviewed by editors before they go live.

Discussion (0)

Sign in or join free to ask questions and discuss this topic.