Customising and evaluating foundation models: fine-tuning, RLHF, ROUGE, BLEU and benchmarks

Every foundation model on Amazon Bedrock arrived there through the same three-stage journey: a provider pre-trained it on a slice of the written internet, fine-tuned it on examples of the behaviour it should have, and aligned it with human feedback. A customer who is unhappy with what the model does can repeat the second stage on their own data, and that is what "customising a foundation model" means in practice. This article covers how each stage works, the methods that make fine-tuning affordable, what the training data has to look like, where AWS runs the jobs, and then the harder half of the job: proving that the customised model is actually better. That proof comes in three layers, public benchmarks, automatic metrics such as ROUGE and BLEU, and human judgement, and finally the business numbers that decide whether the project paid for itself. The applications article already covers when to fine-tune rather than prompt or retrieve; this one is about how, and how to tell whether it worked.

The map

Read this first when short on time. Every branch is a section below.

Pre-training: how a model learns language by predicting the next token

Pre-training is the first and by far the most expensive stage: a randomly initialised transformer is shown trillions of tokens of text and trained, over and over, to predict the next token, until the weights encode grammar, facts, reasoning patterns and the style of every kind of text it saw.

The remarkable thing about the objective is that it needs no labels. A sentence from a web page is its own training example: hide the last word, ask the model to guess it, score the guess against the word that was actually there, adjust the weights a little, repeat. This is self-supervised learning: the supervision signal is manufactured from the data itself rather than written by a person. Decoder models such as GPT, Llama, Claude and Amazon Nova use next-token prediction (also called causal language modelling); encoder models such as BERT use masked language modelling, where random tokens in the middle of a sentence are hidden and predicted from both sides. The fundamentals article covers the tokens, embeddings and attention that make this possible; the point here is the training loop and its scale.

The corpus

The training corpus is assembled from web crawls, books, encyclopaedias, code repositories, scientific papers and licensed data sets. It is filtered for quality, deduplicated so that a page copied ten thousand times does not count ten thousand times, and weighted so that high-quality sources such as books and reference text appear more often than their share of the raw crawl. Whatever the corpus contained on the day it was frozen is what the model knows; that date is the model's knowledge cut-off, and nothing after it is in the weights.

The cost

Pre-training a frontier model takes thousands of accelerators (NVIDIA GPUs, or AWS Trainium chips) running for weeks or months, with the data sharded across them and the weights kept in sync after every step. The bill is tens of millions of dollars for the largest models, plus the engineering team to keep a run of that size alive. This is why pre-training is done by model providers, Amazon, Anthropic, Meta, Mistral, Cohere, AI21 and a few others, and why for a business the question is never "shall we pre-train?" but "which pre-trained model shall we start from?". The compute also has an environmental cost, which the responsible AI article returns to.

What comes out

The output is a base model: a very good text completer that does not yet know it is supposed to be an assistant. Ask a base model "What is the capital of France?" and it may answer "Paris", or it may continue with "What is the capital of Germany?" because quiz lists are common on the web. Making it answer the question, follow instructions and refuse harmful requests is the job of the next two stages, which are the subject of the rest of the training half of this article.

flowchart TD
  C["Raw corpus<br/>web pages, books, code, trillions of tokens, no labels"] --> P["Pre-training<br/>self-supervised next-token prediction<br/>thousands of accelerators for weeks"]
  P --> B["Base model<br/>completes text, follows no instructions"]
  B --> D["Continued pre-training, optional<br/>unlabelled domain text, same objective"]
  D --> F["Supervised fine-tuning<br/>labelled prompt and completion pairs"]
  B --> F
  F --> I["Instruction-tuned model<br/>answers, formats, follows rules"]
  I --> R["RLHF<br/>human rankings, reward model, policy optimisation"]
  R --> A["Aligned assistant model<br/>helpful, honest, harmless"]

self-supervised learning

Training where the labels are derived from the data itself, such as hiding the next word and predicting it, so no human labelling is needed.

next-token prediction

The pre-training objective of decoder models: given the tokens so far, output a probability for every possible next token; the loss is how much probability was given to the token that actually followed.

base model

The output of pre-training alone. It completes text in the style of its corpus but has not been taught to follow instructions or behave as an assistant.

knowledge cut-off

The date after which no text was included in the training corpus, and therefore the edge of what the model knows from its weights.

Fine-tuning: teaching a pre-trained model your task with labelled pairs

Fine-tuning is supervised training that starts from a pre-trained model's weights and continues on a much smaller set of labelled examples, each a prompt and the completion the model should have produced, so that the model's behaviour shifts towards those examples while keeping everything it learned in pre-training.

Mechanically it is the same loop as pre-training, gradient descent on next-token prediction, with two differences. First, the data is labelled: every example is a pair someone wrote or approved, and the loss is usually computed only on the completion tokens, so the model learns to produce the answer rather than to reproduce the prompt. Second, the scale is tiny by comparison: hundreds to tens of thousands of examples and hours of training on a handful of accelerators, not trillions of tokens and months. Because the starting point already understands language, a few thousand examples of "here is a support ticket, here is the category" are enough to make a general model a reliable ticket classifier in the company's own taxonomy.

Instruction tuning

The most important kind of fine-tuning is instruction tuning: training on pairs of an instruction and a good response ("Summarise the following article in three bullets" followed by three bullets, thousands of times across many task types). It is what turns a base model into an "instruct" or "chat" model that answers questions instead of continuing them, follows a requested format and stops when it is done. Every assistant model on Bedrock has been instruction-tuned by its provider; when a customer fine-tunes on Bedrock, they are adding their own instructions and examples on top of that.

Adapting to a domain

The second common purpose is domain adaptation: teaching the model the vocabulary, style and conventions of a field (clinical notes, insurance claims, legal clauses, a company's product names) so that it stops misusing terms and writes the way that field writes. Labelled pairs do this well when the domain is about how things are said; when it is about knowing a large body of new facts, the next section's method fits better, and when the facts change often, retrieval beats both, as the applications article explains.

Transfer learning, the idea underneath

Both are instances of transfer learning: knowledge learned on one task (predicting text across the whole internet) is reused as the starting point for another (classifying tickets). The general features, grammar, world knowledge, reasoning patterns, transfer; only the task-specific behaviour has to be learned, which is why so little data is needed. The same idea predates foundation models: image classifiers fine-tuned from ImageNet weights have been standard for a decade, and the fundamentals article introduces it in that setting.

fine-tuning

Supervised training of a pre-trained model on labelled prompt and completion pairs so that it adopts a task, style or domain. Produces a new, private model.

instruction tuning

Fine-tuning on instruction and response pairs across many task types, which teaches a model to follow instructions rather than merely continue text.

transfer learning

Reusing what a model learned on one task as the starting point for another, so the second task needs far less data. Fine-tuning a foundation model is transfer learning at its largest scale.

catastrophic forgetting

The loss of previously learned abilities when a model is trained hard on a narrow new task.

Continued pre-training: giving the model a new domain's vocabulary and knowledge

Continued pre-training (Bedrock's name; the exam guide also writes "continuous pre-training") runs the self-supervised next-token objective again, this time on a large body of unlabelled text from one domain, so the model absorbs the domain's vocabulary, facts and style without anyone writing a single prompt and completion pair.

The difference from fine-tuning is the data, not the maths. Fine-tuning needs pairs and teaches a task; continued pre-training needs raw documents and teaches a field. A hospital group with twenty years of de-identified discharge summaries, a law firm with its archive of contracts, a manufacturer with thousands of maintenance manuals: none of that is labelled, all of it is full of terms and phrasing that a general model has seen rarely or never. Feed it in as plain text and the model's internal representation of that field sharpens. Afterwards the model is usually still fine-tuned on a small labelled set so that it also knows what to do with the new knowledge; Figure 2 shows continued pre-training as the optional step between the base model and supervised fine-tuning.

It sits one rung above fine-tuning on the cost ladder. It needs far more text (millions of tokens at least, and the more the better), longer training and, on Bedrock, the same Provisioned Throughput to host the result. It also shares fine-tuning's weakness for facts that change: whatever was in the documents at training time is frozen into the weights, so it complements rather than replaces retrieval.

continued pre-training

Further self-supervised training of a foundation model on unlabelled domain text. Also written "continuous pre-training" in the exam guide. Teaches vocabulary and knowledge rather than a task.

domain adaptation

Any method that makes a general model perform well in one specialist field; fine-tuning and continued pre-training are its two training-based forms, RAG its retrieval-based form.

The customisation methods compared: full fine-tuning, PEFT and LoRA, instruction tuning, continued pre-training, RLHF

"Fine-tuning" covers several techniques that differ in which weights move, what data they need and what they cost, and the exam expects the names to be told apart: full fine-tuning updates every weight, parameter-efficient fine-tuning freezes the model and trains a small addition, instruction tuning and continued pre-training are defined by their data, and RLHF is defined by its feedback signal.

Full fine-tuning

Every parameter of the model is updated on the new data. It is the most flexible method and, for a small model, the simplest. For a large model it is expensive in three ways: the training run needs enough accelerator memory to hold the weights, their gradients and the optimiser state (several times the size of the model itself); every task produces a complete copy of the model to store and host; and the risk of catastrophic forgetting is highest because nothing is anchored.

Parameter-efficient fine-tuning

Parameter-efficient fine-tuning (PEFT) keeps the pre-trained weights frozen and trains a small number of new parameters that modify the model's behaviour. Two families matter:

• LoRA (Low-Rank Adaptation). Beside each large weight matrix W the method adds two small matrices A and B whose product has the same shape as W but a low rank r (often 8 or 16). Only A and B are trained; at inference their product is added to W, or merged into it once so there is no extra latency. The trainable parameters are typically well under one percent of the model, the adapter for a task is megabytes rather than gigabytes, and many adapters can share one copy of the base model. QLoRA goes further by holding the frozen base in 4-bit precision so the whole run fits on a single GPU.
• Adapters. Small new layers inserted between the transformer's existing blocks, trained while the rest is frozen. Same idea, different placement; LoRA is the version that has become the default because it adds no inference cost after merging.

Prompt tuning and prefix tuning belong to the same family: they learn a handful of virtual tokens prepended to every input rather than changing any layer. All of them trade a small loss of peak quality for a large saving in compute, storage and forgetting.

flowchart TD
  X["Input activations x"] --> W["Frozen pre-trained weight W<br/>billions of parameters, never updated"]
  X --> A["LoRA matrix A, d × r<br/>trainable, r is small"]
  A --> B["LoRA matrix B, r × d<br/>trainable"]
  W --> S(("add"))
  B --> S
  S --> Y["Output = W·x + B·A·x<br/>merged into one matrix for serving"]

The comparison

PEFT

Parameter-efficient fine-tuning: any method that freezes the pre-trained weights and trains a small number of additional parameters instead.

LoRA

Low-Rank Adaptation: the PEFT method that learns two small matrices whose low-rank product is added to a frozen weight matrix. The adapter is tiny and can be merged for serving.

adapter

A small trainable layer inserted into a frozen model; also the general word for the file produced by any PEFT method.

distillation

Training a smaller "student" model to reproduce the outputs of a larger "teacher", so the student is cheaper to serve. Offered on Bedrock as model distillation.

RLHF: turning human preferences into a reward the model can optimise

Reinforcement learning from human feedback is the alignment stage: humans rank several candidate answers to the same prompt, a reward model is trained to predict those rankings, and the language model is then optimised with reinforcement learning to produce answers the reward model scores highly, which makes it more helpful, more honest and less harmful than instruction tuning alone can.

The insight behind it is that people are bad at writing the perfect answer but good at recognising the better of two. Supervised fine-tuning needs someone to write the ideal response to every prompt, which is slow, expensive and caps quality at what the writers can produce. Preferences are cheap: show a rater two answers and ask which is better. Enough of those comparisons define what "good" means well enough that a model can be trained to score it, and once "good" is a number, ordinary optimisation can chase it.

The three stages

1. Supervised fine-tuning (SFT). Start from the pre-trained model and fine-tune it on a modest set of human-written demonstrations so it produces sensible answers at all. This is the instruction tuning from the previous sections.
2. Train the reward model. Sample several answers from the SFT model for each of many prompts. Human labellers rank them from best to worst (or pick the better of a pair). A second model, usually a copy of the language model with a scalar output head, is trained so that it gives a higher score to the answer the humans preferred. It is now a learned stand-in for human judgement, cheap to run millions of times.
3. Optimise the policy. Treat the language model as a reinforcement learning policy: for a prompt it produces an answer, the reward model scores the answer, and an algorithm, classically Proximal Policy Optimisation (PPO), updates the weights to raise the expected score. A penalty on how far the policy's output distribution drifts from the SFT model (a KL divergence term) keeps it from finding nonsense that fools the reward model, and a little of the original pre-training loss is often mixed in to prevent forgetting.

sequenceDiagram
  autonumber
  participant L as Human labellers
  participant S as SFT model
  participant R as Reward model
  participant P as Policy (the FM)
  L->>S: write ideal answers to prompts
  S->>S: supervised fine-tuning
  S->>L: several answers per prompt
  L->>R: rank them, best to worst
  R->>R: learn to score like the raters
  P->>R: answer to a new prompt
  R->>P: reward score
  P->>P: PPO step, KL penalty to SFT

Variants

• RLAIF (reinforcement learning from AI feedback) replaces the human raters in step 4 with a capable model that ranks answers against written guidelines; humans write the guidelines instead of judging every pair. Anthropic's Constitutional AI is the best-known form: a list of principles (the constitution) guides a model to critique and revise its own answers, and the preferences generated that way train the reward model.
• DPO (Direct Preference Optimisation) skips the reward model and the reinforcement learning loop: it turns the preference pairs directly into a supervised loss on the policy. Simpler and cheaper, and widely used for open models; the exam only needs RLHF and, in passing, RLAIF.

RLHF

Reinforcement learning from human feedback: aligning a model by training a reward model on human preference rankings and then optimising the model to maximise that reward.

reward model

A model trained to score an answer the way human raters would, used as the reward signal in RLHF so humans need not rate every sample.

policy

Reinforcement learning's word for the thing being optimised; in RLHF it is the language model itself, choosing tokens as actions.

PPO

Proximal Policy Optimisation, the reinforcement learning algorithm classically used in RLHF; it makes small, bounded updates to the policy.

RLAIF

Reinforcement learning from AI feedback: the same loop with a model, guided by written principles, supplying the preference rankings instead of people.

Preparing the data: what a fine-tuning set has to look like before training starts

A fine-tuning run is only as good as its examples, and preparing them is most of the work: choosing examples that represent the real inputs, labelling them consistently, removing duplicates and personal data, putting them into the format the service expects, splitting off a validation set and keeping a record of where every example came from.

The model will imitate the data exactly, including its mistakes. If a fifth of the training completions are sloppy, a fifth of the model's answers will be sloppy. If all the examples are short, the model will answer briefly no matter what it is asked. If none of the examples contain a refusal, the model will not refuse. This is why a few thousand carefully curated pairs beat a hundred thousand scraped ones, and why the exam frames data preparation as a governance task and not just a formatting one.

The checklist

1. Define the task and collect candidates. Write down exactly what the model should do and gather real inputs from production logs, support systems or documents, with the outputs a good employee produced or would produce. Real inputs are better than invented ones because they carry the messiness the model will actually meet.
2. Check representativeness. The set should cover the range of inputs in the same proportions as production: every category, every language, the hard cases and the easy ones, the requests that should be refused. A set that over-represents one product line teaches the model that everything is about that product line. This is the same sampling-bias concern the responsible AI article describes for classical models.
3. Label, and check the labelling. Where completions have to be written or corrected, give labellers a written guide, use more than one labeller on a sample and measure their agreement. Amazon SageMaker Ground Truth manages the workflow; the workforce can be the company's own staff, a vendor or Amazon Mechanical Turk's public crowd.
4. Deduplicate. Exact and near-duplicate examples inflate the weight of whatever they contain and make validation scores dishonest when a duplicate lands in both splits. Hash exact matches; use similarity search for near ones.
5. Remove or mask personal and sensitive data. Names, addresses, account numbers and health details in the training set end up in the weights, where they cannot be deleted and can be regurgitated. Amazon Comprehend detects and redacts PII in text; Amazon Macie finds sensitive data in S3 buckets. Check licences and consent for every source at the same time.
6. Balance and size. Bedrock and JumpStart work from hundreds of examples upwards; a few thousand high-quality pairs is the usual sweet spot, and more helps with diminishing returns. Rebalance under-represented classes rather than letting the majority swamp them.
7. Format. Convert to the shape the service expects, JSON Lines with one example per line for Bedrock, and upload to an S3 bucket the training service role may read. SageMaker training jobs take their data as named channels (typically train and validation), each an S3 prefix mounted into the container.
8. Split. Hold out a validation set the training never sees (ten to twenty percent is typical) to measure real quality and to catch overfitting; Bedrock accepts a separate validation file and reports its loss alongside the training loss. Keep a further test set for the evaluation half of this article.
9. Record provenance. Where each example came from, who labelled it, which version of the guide they used, and what was removed. Governance questions later ("was any customer's data used?") are unanswerable without it, and the same record lets a poisoned example be traced.

// Bedrock fine-tuning data: JSON Lines, one prompt and completion per line
{"prompt": "Classify the ticket: My card was charged twice for one order.", "completion": "billing/duplicate-charge"}
{"prompt": "Classify the ticket: The app crashes when I open my basket.", "completion": "app/crash"}
{"prompt": "Classify the ticket: How do I change my delivery address?", "completion": "account/address-change"}

// Bedrock continued pre-training data: unlabelled text only
{"input": "Under clause 14.2 the lessee shall indemnify the lessor against ..."}
{"input": "Heads of terms are not binding unless expressly stated to be so ..."}

flowchart TD
  A["Collect real inputs and good outputs<br/>logs, tickets, documents"] --> B["Curate<br/>representative, balanced, licensed"]
  B --> C["Label and review<br/>Ground Truth, written guide, agreement check"]
  C --> D["Clean<br/>deduplicate, remove PII with Comprehend"]
  D --> E["Format<br/>JSONL prompt and completion pairs"]
  E --> F["Split<br/>train, validation, held-out test"]
  F --> G["Upload to S3<br/>training job reads it through a service role"]

JSON Lines (JSONL)

A text file with one JSON object per line. The training-data format for Bedrock customisation and a common one for SageMaker.

data channel

A named input to a SageMaker training job, backed by an S3 prefix and made available inside the training container; conventionally train, validation and test.

validation set

Examples held out of training and scored during it to measure real quality and detect overfitting. Distinct from the test set, which is used only at the end.

provenance

The recorded origin and history of a data item: source, licence, labeller, transformations. The basis of data governance for training sets.

Where AWS does it: Bedrock custom models, SageMaker JumpStart, training jobs and Ground Truth

AWS offers the training half of this article at three levels of effort: Amazon Bedrock runs fine-tuning and continued pre-training as a managed job with no infrastructure to see, SageMaker JumpStart fine-tunes open models with a prepared script on infrastructure you pick, and a SageMaker training job runs any script on any container; Amazon SageMaker Ground Truth and Amazon Mechanical Turk supply the labelled and preference data all three need.

Amazon Bedrock custom models

In the Bedrock console or API a customisation job takes a base model that supports it (Amazon Titan Text and Titan Image Generator, Amazon Nova, and selected Meta Llama and Cohere models), a training file in S3, optional validation data and a few hyperparameters (epochs, batch size, learning rate), and produces a custom model that lives in the account, encrypted with an AWS-managed or customer-managed KMS key and never shared with the model provider. Two job types exist: fine-tuning on labelled pairs and continued pre-training on unlabelled text. The output metrics (training loss, validation loss) are written to S3 for inspection.

To invoke a custom model it has to be hosted, and the mechanism the exam expects is Provisioned Throughput: you buy a number of model units for a term (no commitment, one month or six months), which reserves capacity for that model. Base models are billed on demand per token; custom models have historically required provisioned throughput because a private copy of the weights has to be loaded onto dedicated capacity. AWS has since added on-demand inference for some custom models, but "a custom model on Bedrock needs Provisioned Throughput" remains the exam's fact. The Bedrock article covers model units and the rest of the pricing.

flowchart TD
  D["Training data in S3<br/>JSONL prompt and completion pairs"] --> J["Bedrock customisation job<br/>fine-tuning or continued pre-training"]
  V["Validation data in S3, optional"] --> J
  J --> M["Custom model in the account<br/>encrypted, private to you"]
  J --> O["Output metrics in S3<br/>training and validation loss"]
  M --> T["Provisioned Throughput<br/>model units reserve capacity"]
  T --> I["InvokeModel or Converse<br/>same API as the base model"]

SageMaker JumpStart

JumpStart is the model hub inside SageMaker Studio: hundreds of open and proprietary models (Llama, Mistral, Falcon, Stable Diffusion, Hugging Face models, plus Amazon's own) that can be deployed to an endpoint in a few clicks or fine-tuned with a prepared training script. For text models the script is typically LoRA-based; you point it at a data set in S3, choose an instance type and hyperparameters, and get back model artefacts to deploy on a SageMaker endpoint. More control than Bedrock (you choose the instance, you own the endpoint), less than a hand-written job.

SageMaker training jobs

A training job is the general mechanism the lifecycle article describes: a container (a built-in algorithm, a framework container for PyTorch or Hugging Face, or your own image), a script, instance type and count, input channels from S3 and an output location. For foundation models this is where full fine-tuning, custom PEFT, RLHF and DPO pipelines run, on GPU instances or on Trainium (trn1) instances, with distributed training libraries for models that do not fit on one device. Managed Spot Training cuts the cost of interruptible runs; SageMaker HyperPod is the resilient cluster option for runs that take days.

Labelling: Ground Truth and Mechanical Turk

Amazon SageMaker Ground Truth builds labelled data sets: it presents items to human workers with a task template (classification, bounding boxes, text spans, and for generative work, writing a completion or ranking several model answers), consolidates several workers' answers per item, and can use automated labelling to reduce the human share. The workforce is one of three: a private workforce of the company's own staff, a vendor workforce from AWS Marketplace, or Amazon Mechanical Turk, the public crowd marketplace, suitable for tasks with no confidential data. Ground Truth Plus is the fully managed version where AWS runs the workforce. For RLHF the ranking template produces the preference pairs a reward model needs.

customisation job

The Bedrock job that produces a custom model from a base model plus training data; either a fine-tuning job or a continued pre-training job.

Provisioned Throughput

Reserved capacity on Bedrock, bought in model units for a term, required to host a custom model and available for base models that need guaranteed throughput.

model unit

The unit of Provisioned Throughput: a fixed amount of input and output tokens per minute for one model.

Ground Truth Plus

The fully managed labelling service in which AWS supplies and runs the workforce; the customer supplies data and instructions.

Human evaluation: rubrics, pairwise preference and Bedrock human evaluation jobs

Human evaluation is people reading model outputs and scoring them, either against a written rubric or by choosing the better of two, and it is the only method that measures what actually matters for open-ended tasks, whether a person finds the answer helpful, correct and appropriate, at the price of being slow, expensive and only as consistent as the people.

Automatic metrics compare an output with a reference; for a summary or a translation that is workable, but for "draft a reply to this customer" there is no single right reference, and a metric that rewards word overlap will punish a better answer phrased differently. So the gold standard for generative quality is human judgement, and every automatic method in the following sections is ultimately validated by how well it agrees with humans.

Rubrics

A rubric names the criteria (accuracy, completeness, tone, harmlessness, adherence to format) and defines what each score means (1 to 5, or pass and fail) so that two raters looking at the same answer reach the same number. Good rubrics are specific ("contains no claim not supported by the source" rather than "is accurate"), and the evaluation uses at least two raters per item with their agreement measured, exactly as labelling does. Scores per criterion are more useful than one overall score because they say what to fix.

Pairwise preference

Show a rater the same prompt with two answers, from two models or two prompts or two versions, and ask which is better. People are more reliable at comparison than at absolute scoring, the results convert to a ranking (Elo-style ratings, as in chess), and the pairs are the same data RLHF's reward model is trained on. It is the method behind public chatbot leaderboards and the standard way to decide "is the new model better than the old one?".

Bedrock human evaluation jobs

Amazon Bedrock model evaluation has a human evaluation mode alongside its automatic one. You supply a prompt data set (your own, in JSONL), choose one or two models to compare, define the metrics and rating method (thumbs up or down, a Likert scale, choice between the two answers, ordinal ranking), and choose the workforce: your own team, set up as a private work team with instructions you write, or an AWS-managed team for which you describe the task and AWS supplies the people. The job produces a report with the scores per metric and, for a comparison, the preference rate. A related later addition, LLM-as-a-judge, lets a strong Bedrock model score another model's answers against a rubric, cheaper than people and closer to them than n-gram metrics.

rubric

A written scoring guide that names the criteria and defines each score level, so different raters score alike.

pairwise preference

An evaluation where a rater picks the better of two answers to the same prompt; aggregated into rankings and used as RLHF training data.

inter-rater agreement

How often independent raters give the same score; low agreement means the rubric or the task is unclear and the scores cannot be trusted.

LLM-as-a-judge

Using a capable model to score another model's outputs against a rubric; cheaper and faster than human raters, and closer to them than n-gram metrics.

Benchmarks: the public exams a model sits, and what each one measures

A benchmark is a fixed public data set with a scoring rule, so that every model can be given the same test and the numbers compared; the well-known ones each probe one ability, general knowledge, common sense, language understanding, reading comprehension, truthfulness or coding, and a model card quotes a handful of them the way a school report quotes subjects.

Benchmarks answer "which base model should we start from?" and "did our fine-tuning damage the model's general abilities?" (run the benchmarks before and after: a drop is catastrophic forgetting made visible). They do not answer "does it do our task well?", which needs the task-specific metrics and human evaluation of the other sections. Most are multiple-choice or short-answer so they can be scored automatically, and most are scored as accuracy.

Two more names are worth recognising. GSM8K tests multi-step arithmetic word problems and is the usual reasoning benchmark next to MMLU. And the Hugging Face Open LLM Leaderboard is not a benchmark but a table: it runs open models through a fixed set of benchmarks (MMLU, HellaSwag, TruthfulQA, GSM8K and others in its first version) with one harness so the numbers are comparable, and ranks them. Model cards on Bedrock and elsewhere quote benchmark scores; the leaderboard is where the same scores are compared across open models on equal terms.

benchmark

A fixed, public data set and scoring rule used to compare models on the same test.

pass@k

The code-generation score: the fraction of problems for which at least one of k generated solutions passes all the tests.

contamination

Benchmark questions appearing in a model's training data, which inflates its score without improving its ability.

leaderboard

A ranked table of models scored on a shared set of benchmarks with one harness; the Hugging Face Open LLM Leaderboard is the best known for open models.

Metrics: ROUGE, BLEU, BERTScore, perplexity, exact match and F1

An automatic metric turns a model output and a reference answer into a number without a human in the loop, and each of the standard ones measures a different notion of "close": ROUGE counts how much of the reference the output recalled, BLEU counts how much of the output is precise, BERTScore compares meaning through embeddings, perplexity measures how well the model predicts text, and exact match and F1 score short answers.

ROUGE: recall, for summaries

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) asks: of the n-grams in the human-written reference, how many appear in the model's output? It is recall-oriented, so it rewards an output that covers what the reference covers, which is the right question for summarisation: did the summary keep the important content? ROUGE-1 counts single words, ROUGE-2 counts word pairs (so word order starts to matter), and ROUGE-L uses the longest common subsequence between output and reference, which credits words in the same order even if they are not adjacent. Implementations usually report precision and F1 as well, but recall is the headline.

BLEU: precision with a brevity penalty, for translation

BLEU (Bilingual Evaluation Understudy) asks the opposite: of the n-grams in the model's output, how many appear in the reference? It is precision-oriented, combines precision for 1-grams to 4-grams as a geometric mean, and multiplies by a brevity penalty so that a very short output cannot score well by containing only safe words. It was built for machine translation, where a translation should say what the source says and nothing more, and it is still the standard number there. Scores run from 0 to 1 (or 0 to 100); anything much above 0.4 on real translation is good.

BERTScore: meaning, not wording

BERTScore replaces exact word matching with embeddings: each token in the output and the reference is embedded with a BERT-style model, every output token is matched to its most similar reference token by cosine similarity, and the similarities are averaged into precision, recall and F1. "The firm's profits rose" and "The company's earnings increased" share no words and would score near zero on ROUGE or BLEU, but their embeddings are close and BERTScore says so. It is the metric to reach for when paraphrase is acceptable, at the cost of needing a model to compute it and being harder to explain.

A worked example

Reference : the cat sat on the mat            (6 words)
Candidate : the cat sat on a mat              (6 words)

Unigrams   matched: the, cat, sat, on, mat      = 5
  ROUGE-1 recall    = 5 / 6 reference unigrams  = 0.83
  BLEU-1 precision  = 5 / 6 candidate unigrams  = 0.83

Bigrams    reference: the-cat, cat-sat, sat-on, on-the, the-mat
           candidate: the-cat, cat-sat, sat-on, on-a,   a-mat
           matched  : the-cat, cat-sat, sat-on           = 3
  ROUGE-2 recall    = 3 / 5                     = 0.60
  BLEU-2 precision  = 3 / 5                     = 0.60

ROUGE-L    longest common subsequence: the cat sat on ... mat = 5
           recall = 5 / 6 = 0.83

Brevity penalty: candidate length 6 = reference length 6, so BP = 1.0

Now a candidate that games precision:
Candidate : the cat                           (2 words)
  BLEU-1 precision  = 2 / 2 = 1.00   but BP = exp(1 - 6/2) = 0.14, BLEU collapses
  ROUGE-1 recall    = 2 / 6 = 0.33   recall exposes the missing content

The others

• Perplexity measures how surprised a language model is by a text: the exponential of the average negative log-probability it assigned to each actual token. Lower is better; a perplexity of 10 means the model was, on average, as uncertain as if choosing uniformly among 10 tokens. It needs no reference answers, only text, so it is the metric of pre-training and continued pre-training (did the model learn the domain's language?), and it says nothing about whether answers are correct or helpful.
• Exact match (EM) is the fraction of answers identical to the reference after normalising case and punctuation, the strict score for extractive question answering. F1 at the token level softens it: the harmonic mean of the precision and recall of the answer's tokens against the reference's, so "Barack Obama" against a reference of "Obama" gets partial credit. SQuAD reports both. (Classification F1, over predicted classes, is the same formula applied to labels, as in the fundamentals article.)
• Toxicity scores are produced by a classifier that rates each output for hate, insults, threats, sexual content and the like; reported as the average score or the share of outputs above a threshold. Robustness measures how much the output changes when the input is perturbed (typos, case changes, whitespace, paraphrase); a robust model gives the same answer to the same question misspelt. Both are among the built-in metrics of Bedrock model evaluation and SageMaker Clarify.

n-gram

A run of n consecutive words (or tokens): unigrams, bigrams, trigrams. Overlap metrics count how many n-grams two texts share.

brevity penalty

BLEU's multiplier, below 1 when the output is shorter than the reference, so a short output cannot score highly just by being precise.

longest common subsequence

The longest sequence of words that appears in both texts in the same order, not necessarily adjacent; the basis of ROUGE-L.

perplexity

The exponential of a model's average negative log-likelihood on a text; how many equally likely choices it was effectively facing per token. Lower is better.

Choosing an evaluation approach: automatic or human, and which metric for which task

The evaluation approach follows from the task: if there is a single correct answer, score it automatically with accuracy, exact match or F1; if there is a reference text, use the overlap metric built for that task; if there is neither, use people or a judge model; and in practice a serious evaluation layers all three, automatic metrics on every build and human review on a sample.

flowchart TD
  S["What does the task produce?"] --> Q1{Fixed answer?}
  Q1 -- "yes: a label or a span" --> E["Accuracy, exact match, F1<br/>classification, extractive QA"]
  Q1 -- no --> Q2{Reference texts?}
  Q2 -- no --> H["Human evaluation or judge model<br/>rubric or pairwise preference"]
  Q2 -- yes --> Q3{Which task?}
  Q3 -- summarisation --> R["ROUGE, recall<br/>plus BERTScore for paraphrase"]
  Q3 -- translation --> B["BLEU, precision<br/>with brevity penalty"]
  Q3 -- other --> BS["BERTScore<br/>meaning over wording"]

The practical pattern is a pyramid. Every change to a prompt, model or fine-tuning set runs the automatic suite on a fixed test set of a few hundred of your own prompts with references, and fails the build on a regression. Every release candidate gets a judge-model pass over a larger sample and a human pass over a smaller one. Benchmarks run when the base model changes. And after launch, the business metrics of the next section take over.

Meeting the business objective: productivity, engagement, cost, latency and A/B tests

A model that scores well on every metric above can still fail the project, because the objective was never "high ROUGE" but "agents resolve tickets faster" or "customers finish the checkout"; determining whether the model meets the business objective means naming that outcome, choosing the numbers that track it, and measuring them on real traffic with A/B tests and human feedback loops.

Outcome metrics

• Productivity. Time saved per task, tasks completed per person per day, tickets resolved without escalation, first drafts accepted without edits. Measured by comparing a group with the assistant against one without.
• User engagement. Adoption (what share of eligible users use it), retention (do they come back), session length, thumbs up and down, the acceptance rate of suggestions. A model people stop using has failed whatever its accuracy.
• Task engineering. Defining the business task narrowly enough that it can be measured at all: "answer any question" cannot be scored, "classify a ticket into one of forty categories and draft the first reply" can. Much of a successful project is reshaping the task so that a model can do it and a metric can see it; the exam lists task engineering as one of the business-facing evaluation concerns for that reason.

Operational metrics

• Accuracy on the business task, measured on real traffic: the share of ticket classifications an agent left unchanged, the share of extracted fields that passed validation. This is the metric that matters, and it is usually different from the benchmark or overlap scores.
• Cost per request: input and output tokens times price, plus provisioned throughput or endpoint hours divided by requests, plus retrieval and guardrail calls. A fine-tuned model shortens prompts (fewer few-shot examples) and can pay for its own hosting at volume, or not; the number decides.
• Latency: time to first token and time to complete answer, at the percentiles users feel (p95, not the mean). A smaller or fine-tuned model is often chosen for latency rather than accuracy.

How to measure them

A/B testing sends a slice of live traffic to the new model or prompt and the rest to the current one, and compares the outcome metrics with enough volume for the difference to be real. On AWS, a SageMaker endpoint can host several production variants with traffic weights, and Bedrock applications route in the application layer or with Bedrock Prompt Management versions. Human feedback loops close the circle: thumbs down, edits to a draft, escalations to a person and Amazon A2I reviews are collected, labelled and fed back both as the next evaluation set and as the next fine-tuning set, so the system improves on exactly the cases it got wrong. After launch, monitoring for drift (a change in the inputs, a fall in acceptance rate) tells you when to run the loop again.

flowchart TD
  O["Business objective<br/>fewer escalations, faster drafts, more conversions"] --> K["Choose the numbers<br/>task accuracy, cost per request, p95 latency, engagement"]
  K --> T["A/B test<br/>new model or prompt on a slice of live traffic"]
  T --> Q{Better?}
  Q -- yes --> R["Roll out and keep monitoring for drift"]
  Q -- no --> I["Iterate: prompt, RAG, fine-tune, other model"]
  I --> T
  R --> F["Human feedback loop<br/>thumbs, edits, A2I reviews become eval and training data"]
  F --> K

A/B test

Splitting live traffic between two versions and comparing their outcome metrics, so that the better one is chosen on evidence from real use.

task engineering

Shaping the business task so that a model can perform it and a metric can measure it; the definition work that precedes prompt or model work.

production variant

One of several models behind a single SageMaker endpoint, each receiving a configured share of traffic; the SageMaker mechanism for A/B tests.

feedback loop

Collecting user reactions and human corrections to model outputs and feeding them back as evaluation and training data.

Recap