Gen AI Concepts for Bedrock

Tokenization is the process of breaking down text into smaller units, called tokens, which can be words, subwords, or even individual characters. These tokens serve as the building blocks that AI models use to process and generate language.

A sentence like "Artificial Intelligence is powerful." The model tokenizes it into smaller units:

Word-level tokenization: ["Artificial", "Intelligence", "is", "powerful", "."] Subword tokenization (e.g., Byte-Pair Encoding): ["Artifi", "cial", "Intelli", "gence", "is", "power", "ful", "."] Character-level tokenization: ["A", "r", "t", "i", "f", "i", "c", "i", "a", "l",...]

Helps models understand and generate text efficiently. Handles out-of-vocabulary words by breaking them into subwords. Reduces complexity compared to processing raw text.

The context window refers to the maximum number of tokens an AI model can process at once when generating responses. It determines how much prior text the model "remembers" during inference.

If a model has a context window of 4,096 tokens, it can only consider the last 4,096 tokens when generating a response. Older parts of a conversation or document may be forgotten unless explicitly included in the prompt.

Short context windows (e.g., 512 tokens) limit long-form coherence. Larger context windows (e.g., 32,000+ tokens) improve memory and allow models to process entire documents in a single prompt. Optimizing context windows is crucial for applications like chatbots, legal analysis, and summarization.

Embeddings are numerical representations of words, sentences, or documents in a high-dimensional space, allowing AI models to understand their relationships based on meaning.

Instead of treating words as separate entities, embeddings represent them as vectors (lists of numbers) that capture their meanings. Similar words have embeddings that are closer together in space.

"King" → [0.21, -1.32, 0.88,...] "Queen" → [0.19, -1.29, 0.87,...] "Dog" → [-0.75, 0.42, 0.33,...]