AI, machine learning and deep learning: the vocabulary everything else assumes

Every other article in this series leans on about thirty words: model, algorithm, training, inference, label, feature, supervised, overfit, token, endpoint. The exam leans on them too, and most of its wrong answers are built by swapping one of these words for its neighbour. This article pins each one down once, with a picture where a picture helps, so that later articles can say "a supervised classifier on tabular data behind a real-time endpoint" and mean something exact. The previous article laid out the exam and the AWS stack; this one is the dictionary for both.

The map

Read this first when short on time. Every branch is a section below.

The nested circles: AI, machine learning, deep learning and generative AI

Artificial intelligence is the whole field of making machines do things that look intelligent; machine learning is the part of it that gets there by learning from data; deep learning is the part of machine learning that uses many-layered neural networks; and generative AI is the part of deep learning whose models produce new content rather than a label or a number. Each circle sits inside the one before it.

Artificial intelligence (AI) is the oldest and loosest term. A chess program that searches millions of positions, a thermostat with hand-written rules, a route planner, a spam filter and a chatbot are all AI. What they share is the goal, not the method: perceive, reason, decide or act in a way a person would call intelligent. Many AI systems have no learning in them at all; they are rules, search and logic written by people. That kind is often called rule-based or symbolic AI, and the next article spends a section on when it is the better choice.

Machine learning (ML) flips the programming model. Instead of writing the rules, you give the computer examples and an algorithm that finds the rules for itself. The output of that process is a model: a function, with learned parameters, that turns an input into a prediction. If the data changes, you retrain and the rules change with it. The price is that you need data, often a lot of it, and that you can no longer read the rules off the page.

Deep learning (DL) is machine learning done with neural networks that have many layers (that is what "deep" means). Older ML needs a person to decide what the useful inputs are: the number of exclamation marks in an email, the ratio of a house's floor area to its plot. Deep networks learn those features themselves from raw pixels, raw audio samples or raw text, which is why they took over vision, speech and language, and why they need far more data and compute than the shallow methods.

Generative AI is the newest inner circle: deep learning models trained on enormous datasets that generate text, images, code, audio or video. The models are called foundation models because one model is trained once and then adapted to many tasks; the text-only ones are large language models. This article gives them one section near the end; the fifth article takes them apart properly.

Same goal, different tools: when each applies

The circles are about method, so the practical question is which method fits the problem. Rules win when the logic is known, small and must be exact: tax brackets, a firewall policy, "block the card if the PIN is wrong three times". Classical ML wins when the pattern is in the data but a person cannot write it down, the data is tabular (rows and columns) and there are thousands to millions of examples: fraud scores, churn, price prediction. Deep learning wins when the input is unstructured (pixels, audio, text) or the pattern is too rich for hand-made features, and there are enough examples and enough compute to pay for it. Generative AI wins when the output itself is content: a summary, an answer, an image, a block of code.

Rule-based system

Software whose behaviour is a set of if-then rules written by people. Deterministic, explainable, and brittle when the world changes.

Feature

One input variable the model looks at: a column in a table, or something derived from the raw data such as the number of words in a review.

Foundation model

A very large deep learning model trained once on broad data, then adapted to many tasks through prompts or fine-tuning rather than retrained from scratch.

Model, algorithm, training and inference: the four words in every sentence

An algorithm is the learning procedure, a model is what that procedure produces from data, training is running the procedure, and inference is using the finished model to make a prediction on something it has not seen. The exam swaps these words in distractors more than any others, so it is worth being pedantic.

The algorithm is the recipe. Linear regression, XGBoost, k-means and "a convolutional network with this architecture" are algorithms: they say how to turn a dataset into a model. The same algorithm run on two datasets gives two different models. On SageMaker an algorithm is a container image, either one of the built-in ones or your own.

The model is the output of training: an architecture plus its learned numbers, the parameters. For linear regression the parameters are the slope and intercept; for a neural network they are the millions or billions of weights. The model is a file (SageMaker writes it to S3 as model.tar.gz) and it is a function: give it an input, it returns a prediction. Settings you choose before training rather than learn from data, such as the depth of a tree or the learning rate, are hyperparameters.

Training is fitting the parameters to the data. The algorithm makes a prediction on each example, compares it with the truth through a loss function (a number that says how wrong it was), and adjusts the parameters to shrink that number. It is expensive, needs the whole dataset, often needs GPUs, and happens rarely: once, then again when the data drifts.

Inference (the exam uses "inferencing") is running the trained model on a new input to get a prediction: is this transaction fraud, what will demand be next Tuesday, what is the next word. It is cheap per call and happens constantly, so it is where most of the money and most of the engineering go in a production system. The ways to serve it, batch and real-time, get their own section below.

flowchart TD
  D["Training data<br/>examples with the right answers"] --> A["Algorithm<br/>XGBoost, a neural network, k-means"]
  H["Hyperparameters<br/>tree depth, learning rate, epochs"] --> A
  A -->|"training: minimise the loss"| M["Model<br/>architecture plus learned parameters<br/>saved as an artifact"]
  N["New, unseen input"] -->|"inference"| M
  M --> P["Prediction<br/>a number, a class, a sentence"]

Features, labels and the split

Each training example is a set of features (the inputs) and, in supervised learning, a label (the answer). A house is [floor area, bedrooms, postcode] with the label sold price. Before training, the data is split three ways: a training set the algorithm learns from, a validation set used to choose hyperparameters and detect overfitting while training, and a test set touched exactly once at the end to report an honest number. A common split is 80/10/10 or 70/15/15. Leaking test rows into training is the most common way to get a model that looks brilliant and fails in production.

Loss function

A formula that scores how far a prediction is from the truth for one example or a batch. Training exists to make it small. Mean squared error for regression and cross-entropy for classification are the usual ones.

Model artifact

The saved model: the file or files that hold the learned parameters and enough description to load them. What a training job produces and what an endpoint loads.

Hyperparameter

A knob set before training rather than learned: learning rate, number of layers, tree depth, number of epochs, regularisation strength.

Neural networks: neurons, layers, weights and backpropagation in plain words

A neural network is a stack of simple arithmetic units, each of which multiplies its inputs by learned weights, adds them up and passes the total through a squashing function; stacked in layers, and with the weights tuned by backpropagation, the stack can approximate almost any function from input to output.

One neuron

A neuron (also called a unit or node) takes several numbers in, multiplies each by a weight, adds a bias term (a constant, nothing to do with the fairness kind of bias later in this article), and passes the sum through an activation function. The activation is what makes the network more than a big linear equation: without it, stacking layers would collapse to a single straight line. The common choices are ReLU (zero for negative inputs, the input itself for positive ones; cheap and the default in hidden layers), sigmoid (squashes to between 0 and 1, used for a yes-or-no output) and softmax (turns a set of scores into probabilities that sum to 1, used for a multi-class output).

Layers

Neurons are arranged in layers. The input layer is just the features (pixel values, word embeddings, columns). One or more hidden layers do the work; each layer's outputs are the next layer's inputs. The output layer gives the answer: one neuron for a regression number, one sigmoid for a binary class, a softmax over N neurons for N classes. "Deep" simply means many hidden layers. Early layers of an image network learn edges and blobs, later ones learn textures, parts and whole objects; nobody programs that, it falls out of training.

How the weights are learned

1. Forward pass. Feed a batch of examples in. Each layer computes its weighted sums and activations; the output layer produces predictions.
2. Loss. Compare predictions with the labels through the loss function and get one number: how wrong the batch was.
3. Backpropagation. Work backwards from the output layer to the input, and for every weight compute how much a tiny nudge to it would change the loss. That quantity is the gradient. The chain rule from calculus lets each layer pass its share of the blame to the layer before it, which is where the name comes from.
4. Gradient descent. Move every weight a small step in the direction that reduces the loss. The size of the step is the learning rate: too big and training bounces around or diverges, too small and it takes forever.
5. Repeat. Take the next mini-batch and do it again. One pass over the whole training set is an epoch; training runs for as many epochs as it takes for the validation loss to stop falling.

That loop is all a GPU does for days when a big model trains. GPUs suit it because the forward and backward passes are enormous matrix multiplications that parallelise well. AWS also builds its own chips for the two halves: AWS Trainium for training and AWS Inferentia for inference, both offered as EC2 instance families (Trn and Inf) and as SageMaker instance types.

Activation function

The non-linear function applied to a neuron's weighted sum: ReLU in hidden layers, sigmoid for a probability, softmax for a probability over several classes.

Backpropagation

The algorithm that computes, for every weight in the network, how much the loss would change if that weight changed, by passing the error backwards from the output layer.

Gradient descent

Adjusting every weight a small step against its gradient so the loss falls. Stochastic gradient descent does it per mini-batch rather than per full dataset.

Epoch

One complete pass through the training set. Models train for several epochs; too many and they start to overfit.

Supervised learning: learning from examples with the answer attached

Supervised learning trains a model on examples that each carry the correct answer, the label, so that it can produce that answer for new examples; it is the most common kind of machine learning in production, and it splits into regression when the answer is a number and classification when the answer is a category.

The word "supervised" means a teacher marked the homework. Someone, or some process, attached a label to every training row: this email is spam, this transaction was charged back, this house sold for 412,000, this photo contains a cat. Getting labels is usually the expensive, slow part of a supervised project, which is why AWS sells a labelling service (SageMaker Ground Truth, covered in the lifecycle article) and why so much recent progress has come from methods that need fewer labels.

Regression: the answer is a number

Regression predicts a continuous value: a price, a temperature, tomorrow's demand, how many minutes the delivery will take. The simplest algorithm is linear regression: fit a straight line (or a plane through many features) that minimises the squared distance to the points. It is fast, explainable (each coefficient says how much the output moves per unit of that feature) and surprisingly hard to beat on small clean tabular data. Gradient-boosted trees such as XGBoost take over when the relationship is not a straight line. Regression is judged by how far off the numbers are: mean absolute error, root mean squared error, and friends, all defined in the metrics section of the lifecycle article.

Classification: the answer is a category

Classification predicts which of a fixed set of classes an example belongs to. Binary classification has two classes (fraud or not, churn or stay, positive or negative review). Multi-class classification picks one of many (which digit, which product category, which language). Multi-label allows several at once (a photo tagged both "beach" and "dog"). The workhorse for binary problems is logistic regression: despite the name it is a classifier, a linear model whose output goes through a sigmoid to give a probability, and you choose a threshold (0.5 by default) to turn that probability into a decision. Classification is judged by how many it got right and, more usefully, by precision and recall, also in the lifecycle article.

The algorithm shelf

The exam names algorithms without asking you to run them. What it wants is that each name lands in the right box: supervised or not, regression or classification, tabular or unstructured, simple or ensemble.

Label

The correct answer attached to a training example. Also called the target. Its presence is what makes learning supervised.

Ensemble

Several models combined. Random forests average many trees trained in parallel (bagging); XGBoost trains trees one after another to correct the previous ones (boosting).

Threshold

The probability above which a classifier says yes. Raising it gives fewer false alarms and more misses; lowering it does the opposite.

Unsupervised learning: finding structure when nobody labelled anything

Unsupervised learning works on data with no labels, so instead of predicting an answer it finds structure that is already in the data: groups of similar items, a smaller set of features that explain most of the variation, or the points that do not fit anywhere.

Unlabelled data is cheap and plentiful (every log line, every purchase, every sensor reading), which is the appeal. The difficulty is that there is no right answer to check against, so evaluating an unsupervised model is a judgement call: do the clusters make business sense, does the compressed data still work, do the flagged anomalies turn out to be real problems? Three tasks cover almost everything the exam means by the term.

Clustering

Clustering groups similar examples together. k-means is the standard algorithm: pick k, place k centre points, assign each example to its nearest centre, move each centre to the middle of its members, repeat until nothing moves. You choose k; the algorithm does not know how many groups there should be, and picking it is part of the craft (the "elbow method" plots the fit against k and looks for the bend). Typical uses: customer segments for marketing, grouping documents by topic, compressing an image to k colours, finding the natural groups in sensor behaviour before anyone names them.

Dimensionality reduction

A dataset with hundreds of features is slow to train on, hard to visualise and full of redundancy (height in centimetres and height in inches are two columns with one piece of information). Dimensionality reduction squeezes the features down while keeping most of what matters. Principal component analysis (PCA) does it linearly: it finds new axes, ordered by how much of the data's variance each one carries, and you keep the first few. The result trains faster, overfits less and can be plotted in two dimensions. Embeddings, which the generative AI article covers, are the deep-learning cousin: a learned low-dimensional representation of text or images.

Anomaly detection

Anomaly detection flags the examples that look nothing like the rest: a server whose latency pattern is unlike the fleet's, a transaction unlike any of the customer's previous ones, a machine whose vibration signature has changed. It is often unsupervised because anomalies are rare and unlabelled by nature; you have millions of normal readings and a handful of known failures. SageMaker's built-in Random Cut Forest algorithm and the Amazon Lookout family of services (Metrics, Equipment, Vision, in the next article) are built around this task.

Clustering

Grouping examples so that members of a group resemble each other more than they resemble other groups. No labels involved.

Dimensionality reduction

Replacing many features with fewer that keep most of the information. PCA is the linear version; embeddings are the learned version.

Anomaly detection

Finding the examples that do not fit the pattern of the rest. Usually unsupervised because the unusual cases are rare and unlabelled.

Reinforcement learning: learning by trial, error and reward

Reinforcement learning trains an agent to act in an environment by letting it try actions and rewarding the ones that lead to good outcomes; there are no labelled examples, only a score, and the agent has to discover for itself which sequence of actions earns the most of it.

The pieces have fixed names. The agent is the learner (a robot's controller, a game player, a bidding strategy). The environment is everything the agent acts on and observes (the warehouse floor, the game, the auction). At each step the agent sees a state, picks an action, and the environment returns a new state and a reward: a number, possibly zero, possibly negative. The agent's policy is its rule for choosing an action given a state, and training means improving the policy so the total reward over time goes up. Because the reward for a good move may arrive many steps later (the race is won at the finish line, not at the first corner), the agent has to learn which early actions deserve credit, which is what makes the problem hard.

flowchart TD
  A["Agent<br/>policy: state to action"] -->|"action"| E["Environment<br/>the track, the warehouse, the game"]
  E -->|"new state and reward"| A
  R["Reward function<br/>designed by you: stay on track, finish fast"] --> E

Two tensions run through every RL project. Exploration versus exploitation: the agent must sometimes try actions it has not tried, or it never finds the better strategy, but every experiment costs reward now. And reward design: the agent optimises exactly what you reward, not what you meant, so a car rewarded for speed learns to cut corners, and a robot rewarded for "objects in the bin" learns to drop them from a height. Because real trials are slow and can break hardware, most RL is trained in a simulator first and transferred to the real thing afterwards.

The cousins the exam mentions in passing

Semi-supervised learning uses a small labelled set plus a large unlabelled one: train on the labels, use the model to guess labels for the rest, retrain. It is the practical answer when labelling is expensive but raw data is not. Self-supervised learning makes labels out of the data itself, for example hide a word and predict it, or predict the next token; this is how large language models are pre-trained, and it is why they need no human labels for the bulk of their training. Reinforcement learning from human feedback (RLHF) combines RL with human preference ratings to steer an LLM's behaviour after pre-training; the article on customising foundation models covers it.

flowchart TD
  S([A prediction problem]) --> L{Labels available?}
  L -- Yes --> T{Target type?}
  T -- "quantity" --> RG["Regression<br/>linear regression,<br/>XGBoost"]
  T -- "category" --> CL["Classification<br/>logistic regression,<br/>random forest, CNN"]
  L -- No --> G{Reward signal?}
  G -- "yes" --> RL["Reinforcement<br/>agent, environment,<br/>reward, policy"]
  G -- "data only" --> UN["Unsupervised<br/>k-means, PCA,<br/>Random Cut Forest"]

Policy

The agent's decision rule: given the current state, which action to take. Training RL means improving the policy.

Reward function

The formula that scores each step or outcome. Written by the designer; the agent maximises it literally, so it must express what is actually wanted.

Self-supervised learning

Training where the labels are manufactured from the data itself (predict the hidden word, predict the next token), so huge unlabelled corpora become training data.

Fit: underfitting, overfitting and the bias-variance trade-off

A model fits well when it captures the real pattern in the data and nothing else; it underfits when it is too simple to capture the pattern, and overfits when it has also memorised the noise, so it scores brilliantly on the data it trained on and badly on anything new.

Underfitting is the easy one to spot: the model is poor even on its own training data. A straight line through data that curves, a tree with two splits for a problem that needs twenty, a network trained for one epoch. The cure is more capacity: a more flexible algorithm, more features, more training.

Overfitting is the one that bites. The model has enough capacity to fit every wiggle, including the wiggles that are just noise in this particular sample, so training error goes to nearly zero while the validation error stops improving and then rises. A decision tree grown until every leaf has one row, a network with millions of weights trained on a thousand images, a model that has learned that houses sold on Tuesdays are cheaper because three of them were. It generalises badly: the whole point of a model is to work on data it has not seen, and an overfit model does not.

The same idea has a statistician's name, the bias-variance trade-off. Bias here means systematic error from a model too simple for the truth (it is "biased" toward its own shape); variance means sensitivity to the particular training sample (retrain on a different sample of the same data and an overfit model gives very different answers). Simple models have high bias and low variance; flexible models have low bias and high variance. Total error is the sum of both plus irreducible noise, and it is lowest somewhere in the middle. Finding that middle is what the validation set is for.

flowchart TD
  S([Model trained]) --> A{Training poor?}
  A -- Yes --> U["Underfitting<br/>high bias: add capacity, features, epochs"]
  A -- No --> B{Validation gap?}
  B -- "large" --> O["Overfitting<br/>high variance: more data, regularise, stop early, simplify"]
  B -- "small" --> G["Good fit<br/>confirm once on the test set, then deploy"]

The toolbox against overfitting

• More training data. The only fix that attacks the cause; noise averages out when there is more signal.
• Regularisation. Add a penalty to the loss for large weights (L1 pushes some weights to exactly zero, L2 shrinks them all), so the model cannot fit every wiggle.
• Early stopping. Watch the validation loss each epoch and stop when it starts rising, even if training loss is still falling.
• Dropout. In neural networks, randomly switch off a fraction of neurons on each training step so no single path can memorise.
• Simplify. Shallower trees, fewer features, a smaller network.
• Cross-validation. Split the training data into k folds, train k times each holding out one fold, and average; it gives a more honest estimate for choosing hyperparameters when data is scarce.
• Data augmentation. For images and audio, make new training examples by flipping, cropping, rotating or adding noise to existing ones.

Generalisation

How well a model performs on data it did not train on. The only performance that matters in production.

Regularisation

Any technique that penalises model complexity during training to reduce overfitting: L1 and L2 weight penalties, dropout, early stopping.

Cross-validation

Training and validating k times on different splits of the same data and averaging the results, to get a stable estimate of generalisation.

Bias and fairness: when the data, not the code, is the problem

In the responsible-AI sense, bias is a systematic slant in the data or the model that makes it treat some groups or cases worse than others, and fairness is the goal of predictions that do not depend on attributes that should not matter; both are properties of the data and of how the model is used, and neither is fixed by better code alone.

Bias gets into a model through the data at several doors. Sampling bias: the training data does not represent the population the model will meet (a face dataset mostly of one skin tone, a speech corpus mostly of one accent). Measurement bias: the feature or label measures the wrong thing (using arrests as a label for crime, or past loan approvals as a label for "creditworthy"). Historical bias: the data faithfully records a world that was itself unfair, and the model learns to reproduce it. Label bias: the people who labelled the data brought their own assumptions. Class imbalance: one outcome is rare, so the model barely learns it (fraud is a fraction of a percent of transactions; a model that always says "not fraud" is 99.9 percent accurate and useless).

Fairness asks whether the model's behaviour differs across groups defined by a sensitive attribute (age, gender, ethnicity, postcode as a proxy for either). There is no single definition: one version asks for equal approval rates across groups, another for equal error rates, another that the decision not change if only the sensitive attribute changed, and they can conflict with each other. What the exam expects is that you know the questions, know that they must be measured before and after training, and know the AWS tools. Amazon SageMaker Clarify computes bias metrics on the training data (pre-training, for example class imbalance and difference in proportions of labels) and on predictions (post-training), and produces feature-importance explanations of why a model decided what it did; Amazon A2I routes low-confidence or high-stakes predictions to human reviewers; SageMaker Model Monitor watches for bias drift after deployment. The responsible AI article is the full treatment.

Sampling bias

Training data that does not represent the population the model will serve, so the model is accurate for some groups and not others.

Class imbalance

One label being far rarer than another, so accuracy is misleading and the rare class is under-learned. Countered by resampling, class weights and better metrics.

Fairness

The property that a model's predictions or errors do not differ across groups in ways that are unjustified. Several formal definitions exist and they can conflict.

Computer vision and natural language processing: where deep learning earns its keep

Computer vision is the field of getting machines to understand images and video, and natural language processing is the field of getting them to understand and produce human language; both were transformed by deep learning because their raw inputs, pixels and words, are exactly the kind of unstructured data where learned features beat hand-made ones.

Computer vision

The tasks stack up in difficulty. Image classification says what is in the picture (one label per image). Object detection says what and where, drawing a bounding box around each object. Semantic segmentation labels every pixel. Face detection and recognition find faces and match them to identities. Optical character recognition (OCR) reads text out of images. Video analysis does any of these frame by frame and tracks objects across frames. The algorithm behind almost all of it is the convolutional neural network (CNN): small learned filters slide over the image and respond to edges, then combinations of edges, then parts, then whole objects, with pooling layers shrinking the picture as it goes so the later layers see the whole scene. On AWS, Amazon Rekognition does the common tasks as an API, Amazon Textract does OCR plus document structure, and Amazon Lookout for Vision spots defects; the next article covers each.

Natural language processing

NLP covers understanding and generating text. The common tasks are sentiment analysis (is this review positive), named entity recognition (find the people, places, dates, products), key phrase extraction, language detection, topic modelling, text classification, translation, summarisation, question answering and text generation. The first step is always tokenisation, chopping the text into tokens (words or pieces of words), and the second is turning each token into a vector of numbers, an embedding, so that a network can compute on it. For a decade the sequence models were recurrent neural networks (RNNs) and their longer-memory variant LSTM (long short-term memory), which read text one token at a time carrying a hidden state forward. Since 2017 the transformer, which reads all the tokens at once and lets each attend to every other, has replaced them, and scaled-up transformers are the large language models of the next section. Amazon Comprehend does the classic NLP tasks as an API; Amazon Translate translates; Amazon Bedrock serves the generative side.

Speech

Speech sits between the two. Automatic speech recognition (ASR), speech to text, turns audio into a sequence of tokens; text to speech (TTS) does the reverse. Both are sequence-to-sequence deep learning problems and both are APIs on AWS: Amazon Transcribe and Amazon Polly.

flowchart TD
  I["Image<br/>pixels"] --> C["Convolutional layers<br/>edges, textures, parts"] --> O["Classifier or detector<br/>label, boxes, masks, text"]
  T["Text<br/>characters"] --> K["Tokenise<br/>words or word pieces"] --> E["Embed<br/>one vector per token"] --> M["Sequence model<br/>LSTM or transformer"] --> N["Output<br/>sentiment, entities, translation, next token"]

Convolutional neural network

A network whose early layers slide small learned filters across the image, so the same feature detector works anywhere in the picture. The standard architecture for vision.

Tokenisation

Splitting text into the units a model reads: words, sub-words or characters. Model prices and limits are quoted in tokens.

Recurrent neural network

A network that processes a sequence one step at a time while carrying a hidden state forward. LSTM is the variant that remembers over longer spans. Largely replaced by transformers.

Large language models: the deep learning that now dominates the exam

A large language model is a transformer neural network with billions of parameters, trained by self-supervision on a huge body of text to predict the next token, which turns out to teach it grammar, facts, styles and reasoning patterns well enough to answer questions, write, summarise, translate and code from a plain-language prompt.

The word large is literal: parameter counts run from a few billion to well over a trillion, and training data is measured in trillions of tokens. The word language is becoming loose: many current models are multimodal, taking images, audio or video as input alongside text. The training is self-supervised in the sense of the reinforcement learning section: the label is simply the next token in real text, so no human labels the corpus. After that pre-training, a much smaller round of supervised fine-tuning on example conversations and RLHF on human preferences turns a text predictor into a helpful assistant.

An LLM is a foundation model: trained once at enormous cost, then reused for thousands of tasks by changing the prompt, adding context (retrieval-augmented generation) or fine-tuning, rather than training a new model per task. That reuse is the economic fact behind Amazon Bedrock. The limits are as important as the powers for the exam: an LLM can hallucinate (state a plausible falsehood with confidence), knows nothing after its training cut-off, is non-deterministic unless told otherwise, costs money per token in and out, and carries whatever bias its training text carried. Tokens, embeddings, attention, context windows and the rest are the subject of the fifth article in this series.

Large language model

A very large transformer trained to predict the next token on vast amounts of text, usable for many language tasks through prompting.

Hallucination

Fluent, confident output that is false. A property of predicting likely text rather than retrieving facts.

Multimodal

A model that accepts or produces more than one kind of data: text and images, or text and audio.

Types of inferencing: batch, real-time and the variants in between

Batch inference runs a trained model over a whole dataset at once, on a schedule, with no one waiting for any single answer; real-time inference keeps the model loaded behind an endpoint and answers each request in milliseconds; the choice is set by who is waiting, how big the payload is and how steady the traffic is.

Real-time inference is a model behind an API. A request arrives, the model runs, the answer goes back, typically in tens to hundreds of milliseconds. It is the only option when a person or a system is waiting: a fraud check while the card is at the till, a product recommendation while the page loads, a chatbot turn. The cost is that the compute is always on: you pay for the instances behind the endpoint whether traffic is heavy or zero, and you must size and scale them.

Batch inference collects the inputs, runs the model over all of them in one job, and writes the results somewhere. Nobody waits for an individual answer; the job takes minutes or hours. It fits nightly risk scores for every account, weekly demand forecasts for every store, tagging a backlog of ten million images. It is the cheapest per prediction because the hardware runs flat out and is released when the job ends, and it is the simplest to operate because there is no endpoint to keep alive.

Two SageMaker variants fill the gap. Asynchronous inference keeps an endpoint but queues each request and notifies you when the result lands in S3, for large payloads (up to about a gigabyte) and long processing times (up to about an hour) that a synchronous call cannot wait for; it also scales the instance count to zero when the queue is empty. Serverless inference gives a real-time endpoint with no instances to manage: it scales to zero between requests and you pay per millisecond of compute, at the price of a cold start when traffic resumes and limits on payload size and duration, so it suits spiky or infrequent traffic that still needs a synchronous answer. Amazon Bedrock offers the same split for foundation models: on-demand and provisioned throughput for real-time calls, and batch inference that processes a file of prompts asynchronously at a discount.

A fifth place to run inference is the edge: on the device itself, for latency, privacy or lack of connectivity. SageMaker Neo compiles a model for a specific target chip and AWS IoT Greengrass runs it there; the lifecycle article covers both.

flowchart TD
  U["User at the checkout"] -->|"one request"| EP["Real-time endpoint<br/>model loaded, instances always on"]
  EP -->|"answer in milliseconds"| U
  S3i["S3 bucket<br/>a million rows to score"] -->|"nightly job"| BT["Batch transform<br/>instances start, run, stop"]
  BT -->|"results file"| S3o["S3 bucket<br/>scores for every row"]

Endpoint

A hosted, always-available URL with a model behind it. In SageMaker, a real-time endpoint runs on instances you choose; a serverless one runs on capacity AWS manages.

Batch transform

SageMaker's batch inference job: read inputs from S3, run the model on temporary instances, write outputs to S3, release the instances.

Cold start

The delay when a serverless endpoint receives a request after being idle and must load the model before answering.

Types of data: the shape of the input decides the model

Data for AI comes in a handful of shapes, and each shape has its natural algorithms, its storage on AWS and its own labelling cost; the two questions to ask of any dataset are whether it is labelled and whether it is structured.

Labelled versus unlabelled is the supervised-or-not question already covered: labelled data carries the answer for each example, unlabelled does not. Most raw data in a company is unlabelled; labelling it is a project (SageMaker Ground Truth exists for this), which is why unsupervised and self-supervised methods are attractive.

Structured data is rows and columns with a fixed schema: a relational table, a CSV, a Parquet file in a data lake. Every row is the same shape and every column has a type. It is what classical ML eats, it is easy to query, and it is where most business prediction problems live. Unstructured data has no fixed schema and cannot be dropped into a table without processing: images, video, audio, free text, PDFs. It is the bulk of the world's data by volume and it is deep learning's territory. Semi-structured data sits between: JSON, XML and log lines have structure (keys, nesting, tags) but not a fixed set of columns; it is parsed into structured features or fed to models that handle text.

The shapes by name

Time series deserves a note because the exam treats forecasting as its own problem. The rows are ordered, the next value depends on the previous ones, and the usual random shuffling into train and test would leak the future into the past; you split by time instead. Seasonality (weekly, yearly), trend and holidays are the features that matter, and Amazon Forecast (no longer taking new customers, but still on the exam) and SageMaker's built-in DeepAR algorithm were built for it.

Structured data

Data with a fixed schema of typed columns: tables, CSV, Parquet. Query it with SQL, model it with classical ML.

Unstructured data

Data without a fixed schema: images, video, audio, free text. Needs deep learning or a managed AI service to turn into features.

Semi-structured data

Data with tags or keys but no fixed columns: JSON, XML, logs. Parsed into structured features before classical modelling.

Time series

Observations ordered by time where the past predicts the future. Split by time, never shuffled; forecast with DeepAR, ARIMA or a sequence model.

Recap