Tokenizer

I·build·systems·that·think.⏎ ⏎ Deployed·a·ROS2·+·Autoware·LiDAR·pipeline,·then·wrapped·it·in·a·browser-based·HMI·rendering·live·sensor·streams·over·WebSockets.⏎ ⏎ RAG·over·PostgreSQL·with·pgvector,·served·by·FastAPI.

Real BPE tokenisation, running locally — the same merge tables GPT-4 and GPT-4o use, with no API call and nothing downloaded. Type anything and watch it fragment. Technical vocabulary is where it gets interesting: words a model has never seen shatter into pieces, and every piece costs you context and money.

Byte-pair encoding
Vocabulary merges
Context cost
cl100k vs o200k

Tokens are why non-English text and unusual technical vocabulary cost more to process: a word the merge table has never seen gets shattered into fragments, and every fragment consumes context and money. Compare the two encodings on the same string — GPT-4o's larger vocabulary usually swallows technical terms in fewer pieces.