Understanding Tokenization: AI Doesn't Actually Read Text the Way We Do

Notes from a full-stack developer exploring Generative AI.

Lately, I've been spending some time learning about Applied AI and Generative AI.

Coming from a full-stack web development background, I initially thought building AI-powered applications wouldn't be that different from integrating any other API.

You send a request, the model processes the input, and you get a response.

But the more I learn about how Large Language Models (LLMs) work, the more I realize there's a lot happening behind the scenes before a model generates an answer.

One of the first concepts that caught my attention was tokenization.

It sounds pretty straightforward at first. But as it turns out, tokenization plays an important role in how language models process text, handle context, and even how much it costs to use them.

I'm currently learning about Generative AI.

Our brains naturally recognize the words and understand the meaning behind the sentence.

But an LLM doesn't process raw text in quite the same way.

Before the model can work with the text, it goes through a process called tokenization.

In simple terms, tokenization is the process of breaking text into smaller units called tokens.

Here's where it gets interesting: one token doesn't necessarily mean one word.

A token can represent an entire word, part of a word, punctuation, or even individual characters, depending on the tokenizer being used.

For example, a tokenizer might break this sentence down like this:

Notice how spaces can actually be part of a token.

With some tokenizers, "love" and " love" may even have different token IDs.

So tokenization isn't simply splitting a string by spaces like we might do with string.Split().

One question I had while learning about this was:

Why can't the model just read the text directly?

The answer comes down to how computers process information.

Language models operate on numerical representations, not words as we see them on a screen.

That means text needs to be converted into something the model can process mathematically.

At a high level, the process looks like this:

Text → Tokens → Token IDs → Embeddings → Model

After tokenization, each token gets mapped to a number called a token ID.

These IDs are just examples, not actual values from a specific tokenizer.

The token IDs are then used to look up numerical representations called embeddings.

These embeddings are vectors of numbers that the neural network can work with.

A token ID doesn't represent the meaning of a word.

The richer representation comes from embeddings and how the model processes those representations in relation to the surrounding context.

This was one of those concepts that made me realize how different language processing is from traditional application logic.

Have you ever asked a chatbot something like:

How many times does the letter "r" appear in "strawberry"?

For humans, this seems like a ridiculously simple question.

But language models haven't always been reliable at answering questions like this.

A model doesn't necessarily receive every individual letter as a separate unit. A word like "strawberry" might be represented by a few tokens, each containing multiple characters.

So the model isn't automatically working with a character-by-character representation.

Of course, tokenization isn't the only reason an AI might get this wrong.

The model's learned patterns, reasoning capabilities, and access to external tools also play a role.

More capable models can often answer these questions correctly, especially when they use additional reasoning or tools to verify the result.

Still, I think this is a great example of something that's easy for humans but not necessarily straightforward for an AI model.

And sometimes, the opposite is also true.

This is where tokenization becomes especially relevant for developers building applications with LLM APIs.

When working with these APIs, you'll frequently come across terms like:

Because LLM API usage is generally priced based on the number of tokens processed, although input, output, and cached tokens may have different rates.

Imagine we're building an AI-powered customer support feature.

Every time a user sends a message, our backend might assemble several pieces of information:

All of that consumes part of the model's available context.

Now imagine sending the entire conversation history with every request, even when most of it is no longer relevant.

Potentially higher costs, slower processing, and less room for the information that actually matters.

This is where I started realizing something important:

Prompt engineering isn't just about writing better instructions. It's also about managing the information we send to a model.

And that becomes an engineering problem, not just a writing problem.

Here's another interesting thing I learned.

Two sentences with the same meaning, written in different languages, don't necessarily produce the same number of tokens.

Both sentences communicate roughly the same idea, but their token counts may differ depending on the tokenizer.

Because tokenizers are built around patterns found in the text data used to develop them.

Common character sequences may be represented as single tokens, while less common sequences might need to be broken into smaller pieces.

As a result, token efficiency can vary across languages.

Does that mean we should always use English to save tokens?

Token count matters, but so does the quality of the instructions and whether the model responds appropriately to the user.

If token efficiency is a concern, the better approach is to measure it using the actual tokenizer associated with the model we're using.

Rather than assuming one language will always be cheaper.

At first, I thought tokenization was something mainly relevant to machine learning engineers.

But the more I explored Applied AI, the more I realized that understanding these fundamentals can help us make better engineering decisions.

We don't need to build our own tokenizer to integrate an LLM into an application.

But knowing how tokens work becomes useful when designing features that involve large amounts of text.

Let's say we're building an application that reads and summarizes documents using AI.

The simplest approach might be to extract all the text from a document and send it directly to the model.

But what happens when the document contains hundreds of pages?

Suddenly, we have several things to think about.

We need to make sure the amount of information we're sending stays within the model's limits.

Instead of sending everything at once, we can divide documents into smaller sections.

But chunking isn't as simple as splitting text into equal-sized pieces. Important context might be lost when related information ends up in separate chunks.

More input tokens and longer generated responses can mean higher API costs.

For a feature used by thousands of users, small inefficiencies can become significant.

Not every piece of information needs to be included in every request.

With techniques like Retrieval-Augmented Generation (RAG), we can retrieve relevant parts of a document before passing them to the model.

Tokenization doesn't solve all these problems by itself.

But understanding tokens makes it much easier to see why these problems exist in the first place.

Learning about tokenization changed the way I look at LLM APIs.

Previously, I approached them like most backend services:

Send a request, receive a response, and integrate the result into the application.

Now I'm starting to realize that building AI-powered applications also requires understanding how the model receives, represents, and processes information.

Tokenization is only a small part of the overall LLM architecture.

But it's connected to several important concepts: embeddings, context windows, text processing, API costs, and the efficiency of AI-powered systems.

And the more I learn, the more I realize that building with Generative AI isn't just about successfully calling a model.

The real challenge is designing systems that provide the right context, generate useful responses, and remain efficient as they scale.

That's why I think understanding fundamentals like tokenization matters, even if our goal isn't to become an AI researcher.

Sometimes, being a better AI application developer starts with understanding what actually happens before the API returns a response.