Generative AI fundamentals: tokens, embeddings, transformers and foundation models

The machine learning article ended with models that answer a narrow question: is this transaction fraud, what will demand be next Tuesday. Generative AI asks a model to produce the answer itself, a paragraph, a picture, a function, and to do so for tasks nobody trained it for by name. Underneath the apparent magic is a small set of ideas that repeat everywhere: text is cut into tokens, tokens are turned into vectors, a transformer looks at all of them at once and guesses the next one, and a model trained that way on a very large pile of data becomes a foundation model that can be pointed at almost anything. This article builds those ideas in order, then adds pictures, use cases and the lifecycle that takes a model from raw data to a monitored deployment. It is the vocabulary every other generative AI article in the series assumes.

The map

Read this first when short on time. Every branch is a section below.

Generative versus discriminative: learning the distribution, not the boundary

A discriminative model learns the line that separates classes, and a generative model learns what the data itself looks like well enough to produce more of it.

Every model in the machine learning article was discriminative in spirit: given features, produce a label or a number. The model never needs to know what a fraudulent transaction looks like in full; it only needs to know which side of a boundary it falls on. A generative model has a harder job. Trained on text, it must learn the probability of every possible next word given the words so far; trained on images, it must learn what arrangements of pixels look like a photograph of a cat. Because it has learned the shape of the data, it can sample from that shape and produce something new that was not in the training set.

The word "generative" is older than the current wave. Naive Bayes is a generative classifier, and Markov chains have written nonsense poetry for decades. What changed around 2020 is scale: the transformer architecture, trained on hundreds of billions of tokens scraped from the web on thousands of GPUs, produced models whose outputs are fluent enough to be useful. The empirical finding behind that investment is the scaling law: loss falls predictably as parameters, data and compute grow together, so the same architecture keeps getting better without a new idea.

discriminative model

A model that learns to tell inputs apart: it estimates the probability of a label given the input, and outputs that label or a number.

generative model

A model that learns the distribution of the training data well enough to sample new data from it: text, images, audio, code.

scaling law

The observed rule that a model's loss improves smoothly and predictably as parameters, training data and compute are increased together.

Tokens: the unit a model reads, counts and bills

A token is a piece of text from a fixed vocabulary, usually a whole common word or a fragment of a rare one, and it is the only thing a language model ever sees.

Neural networks work on numbers, not letters, so before any text reaches a model it passes through a tokeniser that cuts it into pieces and replaces each piece with an integer id from a vocabulary of, typically, 30,000 to 200,000 entries. The vocabulary is learned from the training corpus by an algorithm such as byte pair encoding (BPE), which starts with single characters and repeatedly merges the most frequent adjacent pair until the vocabulary is full. Frequent words such as "the" and "model" end up as one token each. Rare words, names, misspellings and numbers are split: "tokenisation" might become "token" and "isation", a 10-digit account number might become five tokens of two digits each. Punctuation and leading spaces are usually part of the token that follows them.

flowchart TD
  T["Text<br/>The model tokenises text."]
  K["Tokens<br/>The · model · token · ises · text · ."]
  I["Token ids<br/>791 · 1646 · 4037 · 5014 · 1495 · 13"]
  E["Embedding lookup<br/>one vector of 1,024 numbers per token id"]
  V["Vectors<br/>the sequence the transformer actually reads"]
  T --> K --> I --> E --> V

Why a token is not a word

The rule of thumb for English is that one token is about four characters or three-quarters of a word, so 1,000 tokens is roughly 750 words, or about a page and a half of prose. The ratio worsens for anything the vocabulary saw less of during training: code with its brackets and underscores, languages written in scripts other than Latin, chemical names, base64 strings. A Japanese sentence can cost two to three times the tokens of its English translation, and a JSON document is far more expensive than the same information in plain sentences. This is why the same question costs different amounts in different languages and why "trim the whitespace and drop the boilerplate" is real cost engineering.

Tokens drive cost and limits

Two facts about tokens follow you through every later article. First, foundation models are billed per token: Bedrock on-demand pricing charges separately for the tokens you send (input) and the tokens the model produces (output), and output tokens cost more because generating them takes more compute than reading them. Second, a model's context window, the total amount of text it can consider at once, is measured in tokens, and it must hold the prompt, any retrieved documents, the conversation so far and the answer. A long system prompt spends the same budget on every call; a chatty history eventually pushes the earliest turns out of the window.

token

The unit of text a model reads and produces: a whole common word, a word fragment, a punctuation mark or a group of digits, each mapped to an integer id in the model's vocabulary.

tokeniser

The program that cuts text into tokens using a vocabulary learned from the training data; byte pair encoding, WordPiece and SentencePiece are the common algorithms.

vocabulary

The fixed list of tokens a model knows, tens of thousands to a few hundred thousand entries. Anything not in it is represented as smaller pieces that are.

Chunking: cutting documents into pieces a model can hold

Chunking splits a long document into passages of a few hundred tokens each so that they can be embedded, stored and retrieved individually and fed to a model without overflowing its context window.

A foundation model cannot read a 400-page manual in one go, and even when a window is large enough, stuffing everything in costs money and buries the relevant sentence among thousands of irrelevant ones. The standard answer, and the first step of retrieval-augmented generation (RAG, covered properly in the applications article), is to chunk: split the text into passages, embed each passage as a vector, store the vectors, and at question time retrieve only the handful of chunks that match the question. Chunking therefore decides what the model will later be able to find.

flowchart TD
  D["Document<br/>an 80-page PDF, about 40,000 tokens"]
  P["Parse to text<br/>drop headers, footers, keep section titles"]
  C["Split into chunks<br/>300 tokens each, 20 percent overlap"]
  M["Attach metadata<br/>source file, page, section, date"]
  E["Embed each chunk<br/>Titan Text Embeddings V2"]
  S["Store vector plus text plus metadata<br/>OpenSearch, Aurora pgvector, Pinecone"]
  D --> P --> C --> M --> E --> S

Size and overlap

Chunk size is a trade-off. Small chunks (a sentence or two) embed very precisely but lose the context around them: "it must be replaced every 500 hours" is useless without the sentence that says what "it" is. Large chunks (a whole page) keep context but their embedding becomes an average of many topics, so a specific question matches them weakly, and they eat context window when retrieved. Most systems land between about 200 and 500 tokens and tune from there by measuring retrieval quality.

Overlap repeats the last few dozen tokens of one chunk at the start of the next, so that a sentence or a definition that straddles a boundary survives whole in at least one chunk. Ten to twenty percent overlap is the usual range; more than that stores the same text several times for little gain.

Strategies

• Fixed-size chunking cuts every N tokens regardless of content. Simple, predictable, and the default in many tools.
• Semantic chunking splits where the meaning changes, detected by a drop in embedding similarity between adjacent sentences, so each chunk is about one thing.
• Hierarchical chunking stores small child chunks for precise matching but returns the larger parent chunk to the model, so retrieval is sharp and the context is whole.
• Structure-aware chunking respects headings, paragraphs, table rows and code blocks, so a table is never sliced in half.
• No chunking treats each file as one chunk, sensible only when the files are already short, such as FAQ entries.

chunk

One passage of a document, typically a few hundred tokens, embedded and stored as a unit for retrieval.

overlap

Text repeated at the end of one chunk and the start of the next so that nothing meaningful is cut in half at a boundary.

hierarchical chunking

Storing small child chunks for matching and larger parent chunks for reading, so retrieval is precise but the model gets full context.

Embeddings: meaning as a point in space

An embedding is a list of numbers, a vector, produced by a model so that pieces of text (or images) with similar meaning end up close together, which turns "find something similar" into arithmetic.

The embedding table in Figure 2 gives each token a vector, but the word "embedding" in generative AI usually means something bigger: a vector for a whole sentence, paragraph, product description or image, produced by a dedicated embedding model. The model is trained so that texts that mean the same thing produce vectors pointing in the same direction, whatever words they use. "How do I reset my password" and "forgotten login credentials" share almost no words but land next to each other; "password" and "passport" share most of their letters and land far apart. The dimensions have no human names; meaning is spread across all of them.

Measuring similarity

Three measures appear in practice. Cosine similarity is the cosine of the angle between two vectors: 1 when they point the same way, 0 when they are unrelated, negative when opposed. It ignores length, which is what you want when one text is a long paragraph and the other a five-word query. Euclidean distance is the straight-line distance between the two points; smaller is more similar. The dot product is fast and equals cosine similarity when the vectors are normalised to unit length, which many embedding models do by default. Vector databases index millions of these vectors with approximate nearest-neighbour algorithms (HNSW is the common one) so that "the 10 closest vectors to this query" returns in milliseconds rather than scanning everything.

Embedding models on AWS

• Amazon Titan Text Embeddings V2 takes up to a few thousand tokens of text and returns a vector of 1,024 numbers by default, with 512 and 256 options that trade a little accuracy for less storage. It is the default choice in Bedrock Knowledge Bases.
• Amazon Titan Multimodal Embeddings puts text and images in the same space, so a text query can find matching images and an image can find similar images.
• Cohere Embed on Bedrock offers English and multilingual models, also 1,024 dimensions, with an input type flag that tunes the vector for a query or a document.

Embeddings are used far beyond RAG: semantic search that matches meaning rather than keywords, recommendation ("products whose descriptions are near what you bought"), clustering support tickets into themes, deduplication of near-identical documents, anomaly detection for a log line unlike any seen before, and classification by nearest labelled neighbour when there is too little data to train a classifier.

embedding

A vector of a few hundred to a few thousand numbers produced by a model so that inputs with similar meaning have vectors that point in similar directions.

cosine similarity

The cosine of the angle between two vectors, from 1 (same direction) through 0 (unrelated) to −1 (opposite). The standard measure for comparing embeddings because it ignores vector length.

vector database

A store that indexes embeddings for fast approximate nearest-neighbour search, returning the k vectors closest to a query.

k-NN

k nearest neighbours: the k stored vectors closest to a query vector. HNSW is the graph index most vector stores use to find them quickly.

Transformers: attention over every token at once

A transformer is a neural network that processes a whole sequence of tokens in parallel and lets every token gather information from every other token through a mechanism called self-attention.

The architecture comes from a 2017 Google paper, "Attention Is All You Need", written for machine translation. Before it, sequence models were recurrent neural networks (RNNs, and their LSTM and GRU variants) that read one token at a time and carried a running summary forward. That design had two problems: information from early in a long sentence faded by the end, and reading one token at a time cannot be parallelised across a GPU's thousands of cores. The transformer threw away recurrence. Every token is processed at the same time, and the connection between distant tokens is one attention step away, not a hundred recurrent steps. That made training on web-scale data feasible, and everything called a large language model since is a transformer.

Self-attention in plain words

For each token, the model computes three vectors from its embedding: a query ("what am I looking for"), a key ("what do I contain") and a value ("what I pass on if chosen"). The query of one token is compared with the keys of every token in the sequence; the scores are turned into weights that sum to one with a softmax; and the token's new representation is the weighted sum of all the values. In "the animal did not cross the street because it was too tired", the token "it" places heavy weight on "animal", and its representation absorbs that. The model runs several of these in parallel, multi-head attention, so one head can track grammatical agreement while another tracks coreference and another tracks position in a list.

Because attention on its own has no idea of order (a bag of tokens gives the same weights in any order), a positional encoding is added to each embedding before the first layer so that the model can tell the third token from the thirtieth.

flowchart TD
  I["Token embeddings plus positional encoding<br/>one vector per token, all tokens at once"]
  A["Multi-head self-attention<br/>each token weighs every other token: query, key, value"]
  N1["Add and normalise<br/>residual connection keeps the original signal"]
  F["Feed-forward network<br/>the same small network applied to each position"]
  N2["Add and normalise"]
  R["Repeat the block N times<br/>dozens of layers in a large model"]
  O["Linear layer and softmax<br/>a probability for every token in the vocabulary"]
  I --> A --> N1 --> F --> N2 --> R --> O

Encoder, decoder, or both

The original transformer had two stacks. The encoder reads the whole input and produces a rich representation of it; every token can attend to every other, forwards and backwards. The decoder produces output one token at a time, attending to what it has produced so far (masked, so it cannot peek ahead) and, through cross-attention, to the encoder's output. Three families grew from that:

• Encoder-only models such as BERT are for understanding: classification, entity extraction, embeddings. They read but do not write.
• Decoder-only models such as GPT, Claude, Llama, Mistral and Amazon Nova are for generation. Almost every chat model is one; the "encoder" job is done by the same stack reading the prompt.
• Encoder-decoder models such as T5 and the original translation transformer are for tasks that map one sequence to another, translation and summarisation, though decoder-only models now do those well too.

self-attention

The operation in which each token computes weights over every other token in the sequence and takes a weighted sum of their values, so its representation reflects its context.

multi-head attention

Several attention operations run in parallel with different learned projections, so different heads can capture different kinds of relation.

positional encoding

A vector added to each token embedding that tells the model where in the sequence the token sits, because attention alone is order-blind.

encoder and decoder

The reading stack and the writing stack of a transformer. Encoder-only models understand, decoder-only models generate, encoder-decoder models map one sequence to another.

RNN

Recurrent neural network: the earlier sequence architecture that reads one token at a time and carries state forward. LSTM and GRU are its best-known variants.

Generation: next-token prediction, the context window and the autoregressive loop

A language model produces text by predicting a probability for every possible next token, picking one, appending it to the input and doing it again until it emits a stop token or hits the length limit.

The output of the transformer in Figure 5 is not a sentence. It is a list of numbers, one per vocabulary entry, saying how likely each token is to come next. The training objective was exactly that: shown billions of text fragments with the last token hidden, adjust the weights so the hidden token gets a higher probability. That single, simple objective, next-token prediction, is enough to force the model to learn grammar, facts, reasoning patterns and style, because predicting the next word of a physics textbook well requires knowing some physics. It needs no human labels, which is what makes web-scale training possible: the data is its own label, a setup called self-supervised learning.

flowchart TD
  P["Prompt tokens<br/>The capital of France is"]
  M["Transformer forward pass<br/>reads every token in the window"]
  D["Probability over the vocabulary<br/>Paris 0.91 · a 0.03 · the 0.02 · ..."]
  S["Pick one token<br/>greedy takes the top, sampling uses temperature and top-p"]
  A["Append it to the sequence<br/>The capital of France is Paris"]
  Q{"Stop token?"}
  O["Return or stream the text"]
  P --> M --> D --> S --> A --> Q
  Q -- "no, and room left in the window" --> M
  Q -- "yes, or max tokens reached" --> O

Autoregressive

"Autoregressive" means the model's own output becomes part of its next input. Each new token is chosen, appended and the whole sequence is read again. This is why generation is slow relative to reading (one forward pass per output token), why responses can be streamed token by token to the user, and why an early mistake compounds: once a wrong fact is in the sequence, the model conditions on it as though it were true.

Choosing the token: greedy or sampled

If the model always picked the most probable token (greedy decoding) it would be deterministic and often dull, repeating safe phrases. Instead it usually samples from the distribution, and three parameters shape that sampling. Temperature flattens or sharpens the distribution: near 0 behaves like greedy, higher values make unlikely tokens more likely and output more varied. Top-p (nucleus sampling) keeps only the smallest set of tokens whose probabilities add up to p, cutting off the long tail. Top-k keeps only the k most probable tokens. The Bedrock article has the table of what each does; here the point is that sampling is the source of the nondeterminism the next article lists as a limitation: the same prompt can give a different answer twice.

The context window

The context window is the maximum number of tokens the model can attend to in one pass: prompt, retrieved documents, conversation history and the answer so far, all together. Early models had windows of 2,000 to 4,000 tokens. Current Claude models on Bedrock accept 200,000 tokens, roughly a 500-page book; Amazon Nova Pro offers a 300,000-token window and Llama models 128,000. A larger window means fewer retrieval tricks but a higher per-call cost and slower responses, and models still attend less reliably to the middle of a very long context than to its start and end.

next-token prediction

The training objective of a language model: given the tokens so far, assign probabilities to what comes next.

autoregressive

Generating one token at a time, feeding each generated token back in as input for the next step.

context window

The maximum number of tokens a model can consider in one call, covering the prompt, any history and the answer together.

self-supervised learning

Training where the labels come from the data itself, such as hiding the next token and asking the model to predict it. It needs no human annotation.

stop token

A special token the model emits to signal that its answer is finished; applications can also define stop sequences of their own.

Foundation models: trained once on everything, adapted to anything

A foundation model is a very large model pre-trained on a broad, mostly unlabelled corpus so that it can be adapted, by prompting, retrieval or fine-tuning, to a wide range of tasks it was never specifically trained for.

The term was coined at Stanford in 2021 to name what GPT-3 and its peers had become: not a spam filter or a translator but a general substrate on which many applications are built. Three properties make a model "foundation". Scale: billions to hundreds of billions of parameters, trained on trillions of tokens. Breadth: the training data spans web pages, books, code, forums and, for multimodal models, images and audio, so the model has seen a little of almost everything. Adaptability: the same weights serve many tasks, and behaviour is changed by the input rather than by training a new model. A large language model (LLM) is a foundation model for text; image and multimodal foundation models exist too.

Pre-training and adaptation

Pre-training is the expensive step: months of next-token prediction on thousands of accelerators, costing tens of millions of dollars for the largest models. Only a handful of organisations do it, which is why the Bedrock catalogue lists providers rather than expecting you to train your own. After pre-training comes alignment: supervised fine-tuning on example conversations and reinforcement learning from human feedback (RLHF) so that the model follows instructions and declines harmful requests. The customising article covers those in detail.

Everything a business does with the model is adaptation, and it comes in increasing order of effort and cost: writing a good prompt (see the prompt engineering article), giving the model retrieved documents at run time (RAG), fine-tuning the weights on a few thousand labelled examples, and continued pre-training on a large body of domain text. One base model therefore stands behind many products: the same Claude model answers a bank's support chat, reviews a law firm's contracts and writes a retailer's product descriptions.

foundation model (FM)

A large model pre-trained on broad data at scale that serves as the base for many downstream tasks through prompting, retrieval or fine-tuning.

large language model (LLM)

A foundation model for text: a decoder transformer with billions of parameters trained by next-token prediction.

pre-training

The initial self-supervised training of a foundation model on a huge corpus; the costly step that only the model provider performs.

parameters

The learned weights of the network. A model's size is quoted as its parameter count: 8 billion, 70 billion, several hundred billion.

knowledge cut-off

The date after which the training data contains nothing, so the model has no knowledge of later events unless the prompt supplies it.

Multimodal models: text, image, audio and video on either side

A multimodal model accepts or produces more than one kind of data, typically text together with images, audio or video, inside one network rather than by chaining separate models.

The trick is to make everything look like tokens. An image is cut into patches and each patch is encoded into a vector by a vision encoder; audio is cut into short frames and encoded the same way; those vectors are placed in the sequence alongside the text token vectors, and the transformer's attention treats them all alike. A question about a photograph therefore becomes "attend from the question tokens to the image patches". On the output side, a model may produce text about the image (Claude describing a chart), produce an image from text (Nova Canvas), or produce speech from speech (Nova Sonic). "Multimodal" is used loosely for any of these; the exam cares that you know the input and output modalities of the named models.

flowchart TD
  T["Text tokens"]
  I["Image patches<br/>through a vision encoder"]
  A["Audio frames<br/>through an audio encoder"]
  V["Video frames<br/>sampled and encoded"]
  M["One transformer<br/>attention across every modality in one sequence"]
  O1["Text out<br/>describe, answer, extract"]
  O2["Image or video out<br/>generate, edit, extend"]
  O3["Speech out<br/>spoken reply"]
  T --> M
  I --> M
  A --> M
  V --> M
  M --> O1
  M --> O2
  M --> O3

What it enables

• Document understanding: send a scanned invoice as an image and ask for the total, the vendor and the due date, with no OCR step of your own.
• Visual question answering: a photograph of a damaged parcel with "is this claimable under our policy" plus the policy text.
• Video summarisation: Nova Lite and Pro accept video and can list the scenes or find the moment a product appears.
• Voice assistants: speech in and speech out in one model removes the transcribe, think, synthesise chain and its added latency.
• Cross-modal search: Titan Multimodal Embeddings lets a text query find images and an image find similar products.

modality

A kind of data: text, image, audio, video, or a vector. Multimodal means more than one, on the input side, the output side or both.

vision encoder

The network that turns image patches into vectors a transformer can attend to alongside text tokens.

cross-modal search

Finding items of one modality with a query of another, such as images from a text description, using a shared embedding space.

Diffusion models: images from noise, and GANs and VAEs for contrast

A diffusion model generates an image by learning to remove noise: trained to undo the gradual corruption of real images into static, it can start from pure static and, step by step, denoise it into a new picture guided by a text description.

Training has two directions. The forward process takes a real image and adds a little Gaussian noise, then a little more, over hundreds of steps until nothing but noise remains; this needs no learning. The reverse process is what the network learns: at each step, given the noisy image and the step number, predict the noise that was added so it can be subtracted. Show it millions of images and it becomes very good at "what would this look like slightly less noisy". At generation time you feed it random noise and run the reverse process from the last step to the first. Conditioning on text is done by feeding an embedding of the prompt into the network at each step, and a technique called classifier-free guidance pushes the result towards the prompt and away from a generic image.

Denoising a full-resolution image hundreds of times is slow, so Stable Diffusion introduced latent diffusion: a variational autoencoder first compresses the image into a much smaller latent grid, diffusion runs in that compressed space, and the decoder expands the final latent back into pixels. It is why Stable Diffusion can run on a consumer graphics card. Video models such as Amazon Nova Reel extend the same idea across frames.

flowchart TD
  subgraph Training
    direction LR
    X0["Real image"] --> X1["Add noise<br/>hundreds of small steps"] --> XT["Pure noise"]
  end
  subgraph Generation
    direction LR
    Z0["Random noise"] --> Z1["Remove noise<br/>step by step, guided by<br/>the text prompt embedding"] --> ZI["New image"]
  end
  Training ~~~ Generation

GANs and VAEs, for contrast

Two older generative architectures still appear in exam glossaries. A generative adversarial network (GAN) trains two networks against each other: a generator that makes fakes and a discriminator that tries to spot them, each improving until the fakes are convincing; GANs produce sharp images fast but are hard to train and prone to producing the same few outputs (mode collapse). A variational autoencoder (VAE) trains an encoder that compresses an input into a small probability distribution and a decoder that reconstructs it; sampling the distribution gives new outputs, typically blurrier than GANs but stable and useful, which is why a VAE is the compressor inside latent diffusion.

diffusion model

A generative model trained to reverse the gradual addition of noise, so that running the reversal from random noise produces a new sample.

latent diffusion

Diffusion performed in the compressed latent space of an autoencoder rather than on pixels, making it far cheaper; the design behind Stable Diffusion.

GAN

Generative adversarial network: a generator and a discriminator trained against each other until the generator's fakes fool the discriminator.

VAE

Variational autoencoder: an encoder that maps inputs to a small probability distribution and a decoder that reconstructs them; sampling the distribution generates new outputs.

inpainting and outpainting

Editing operations of image models: regenerate a masked region inside the picture, or extend the picture beyond its original edges.

The model zoo: what each type takes in and gives out

Four model types cover almost every generative AI system, and choosing between them starts with one question: what goes in and what comes out.

The catalogue in Bedrock or SageMaker JumpStart can look like a hundred unrelated names. It collapses to four kinds. Large language models turn text into text and are the general workhorse. Diffusion models turn text or an image into an image or video. Multimodal models take several kinds of input, usually to produce text about them. Embedding models take text or images and return a vector, producing nothing a human reads but powering search and retrieval. Most real applications use at least two: an embedding model to retrieve and an LLM to answer, or an LLM to write a prompt and a diffusion model to draw it.

Use cases: what fits generative AI and which modality does the job

Generative AI fits a task when there is no single right answer to check against, fluency and adaptability are worth more than perfect precision, and a human or a retrieval step can catch the mistakes that will happen.

The use-cases article asked when machine learning fits at all. Generative AI narrows that: it shines where the output is language-like (prose, code, structured text) or media (images, video, audio), where the input varies too much for rules, and where a draft that a person refines is more valuable than a system that must be right unattended. It is a poor fit where the answer must be exactly right every time with no review (a tax calculation), where the rules are already known and cheap to code, or where the cost per call outweighs the value of the output.

Prompt engineering: steering a model without retraining it

Prompt engineering is the craft of writing the input to a foundation model, instructions, context, examples and output format, so that the same fixed weights produce the behaviour you want.

Because a foundation model's behaviour is a function of its input, the cheapest way to change what it does is to change the prompt. A prompt is more than a question: it usually contains a system prompt setting the role and rules ("you are a support assistant for a bank, answer only from the provided policy"), context such as retrieved documents or the conversation so far, the user's actual request, and often examples of the desired output and a required format (JSON with these fields). Zero-shot prompting gives an instruction alone; few-shot adds a handful of worked examples; chain-of-thought asks the model to reason step by step before answering, which improves accuracy on multi-step problems. Every one of these is bounded by the context window and none adds knowledge the model does not have: for that you need retrieval or fine-tuning.

The prompt engineering article covers the techniques, the inference parameters and the attacks (prompt injection, jailbreaking, leaking) in full. Here the concept matters because it is the first rung of the adaptation ladder: try a better prompt before retrieval, retrieval before fine-tuning, fine-tuning before continued pre-training.

prompt

The full input to a foundation model: system instructions, context, examples and the user's request.

zero-shot, few-shot

Prompting with an instruction only, or with an instruction plus a few worked examples of the expected output.

chain of thought

Asking the model to write out its reasoning steps before the final answer, which improves results on multi-step tasks.

The foundation model lifecycle: from data selection to feedback

A foundation model passes through seven stages, data selection, model selection, pre-training, fine-tuning, evaluation, deployment and feedback, and AWS has a named service for each.

The exam guide lists these stages in this order and expects you to say what happens in each and where AWS helps. Two of them, pre-training and sometimes fine-tuning, are done by the model provider; most customers enter the lifecycle at model selection and spend their time from fine-tuning onwards. The loop closes at feedback: what users say about the deployed model becomes new evaluation data and, eventually, new fine-tuning data.

flowchart TD
  A["Stage 1 · Data selection<br/>Amazon S3, AWS Glue, SageMaker Data Wrangler, Ground Truth"]
  B["Stage 2 · Model selection<br/>Bedrock model catalogue, SageMaker JumpStart"]
  C["Stage 3 · Pre-training<br/>SageMaker HyperPod, Trainium, GPU instances"]
  D["Stage 4 · Fine-tuning<br/>Bedrock custom models, SageMaker training jobs"]
  E["Stage 5 · Evaluation<br/>Bedrock Model Evaluation, SageMaker Clarify"]
  F["Stage 6 · Deployment<br/>Bedrock on-demand or provisioned, SageMaker endpoints"]
  G["Stage 7 · Feedback<br/>invocation logs, CloudWatch, Amazon A2I, user ratings"]
  A --> B --> C --> D --> E --> F --> G
  G -. "new examples and failure cases" .-> D
  G -. "new data" .-> A

1. Data selection. Choose what the model will learn from: broad and diverse for a base model, focused and high-quality for fine-tuning. The work is collecting, deduplicating, filtering for quality and toxicity, removing personal data, and checking licences, because whatever is in the data comes out in the model. On AWS the data lives in Amazon S3; AWS Glue and SageMaker Data Wrangler clean and transform it; SageMaker Ground Truth produces labels and human-written examples; Amazon Comprehend or Amazon Macie can flag personal information before it gets in.
2. Model selection. Pick the base: which provider, which size, which modality, which context window, what licence, what price. For most teams this is browsing the Amazon Bedrock catalogue or SageMaker JumpStart and running the candidates against a small evaluation set. The next article lists the selection factors.
3. Pre-training. The provider trains the base model by self-supervised next-token prediction on trillions of tokens. It needs thousands of accelerators working for weeks: SageMaker HyperPod manages resilient clusters for this, AWS Trainium chips and GPU instances (P5 with NVIDIA H100, for example) in EC2 UltraClusters supply the compute. Customers rarely do this; Amazon Nova was trained on Trainium, and Anthropic trains Claude on Trainium-based clusters as well as GPUs.
4. Fine-tuning. Adapt the base model to a domain, a style or a task with a comparatively small labelled dataset of prompt-completion pairs, or continue pre-training on a body of unlabelled domain text. Amazon Bedrock offers both as custom models from JSONL files in S3; SageMaker fine-tunes JumpStart and custom models with full control over the training job. Parameter-efficient methods such as LoRA update a small slice of weights to keep cost down. Detail is in the customising article.
5. Evaluation. Measure whether the model is good enough before it meets users: automatic metrics (accuracy, robustness, toxicity, ROUGE, BLEU, BERTScore), public benchmarks, and human evaluation for the qualities metrics miss. Amazon Bedrock Model Evaluation runs automatic and human evaluations against built-in or your own datasets; SageMaker Clarify evaluates foundation models for quality and bias and explains predictions.
6. Deployment. Put the model behind an endpoint with the right throughput, latency and cost profile: Bedrock on-demand for variable traffic, provisioned throughput for steady high volume or custom models, batch inference for offline jobs; SageMaker real-time, serverless and asynchronous endpoints for models you host yourself. Wrap it with Bedrock Guardrails and IAM before exposing it.
7. Feedback. Watch the model in production and feed what you learn back. Bedrock model invocation logging writes prompts and completions to CloudWatch Logs or S3; CloudWatch metrics track latency, token counts and errors; Amazon Augmented AI (A2I) routes low-confidence or sampled outputs to human reviewers; thumbs-up and thumbs-down from users become the next evaluation set and the next fine-tuning file.

fine-tuning

Further training of a pre-trained model on a small, task-specific labelled dataset so that its weights adapt to a domain, style or format.

continued pre-training

Further self-supervised training on a large body of unlabelled domain text, to teach the model vocabulary and facts of a field.

SageMaker HyperPod

A managed, resilient cluster service for large distributed training jobs that recovers automatically from hardware faults.

Amazon A2I

Amazon Augmented AI: a service that routes model outputs to human reviewers according to rules you set, and returns their verdicts.

Recap