What generative AI can and cannot do, and the AWS stack built for it

The previous article explained how a foundation model works. This one is about whether to use one, and with what. Generative AI is the first technology in years that a business can try in an afternoon and misjudge for a year: the demo is dazzling, the failure modes are quiet, and the bill arrives per token. So the first half of this article is an honest ledger, the advantages, the limitations, the factors that pick a model and the metrics that say whether it paid off. The second half is the AWS stack that exists to make the ledger come out positive: Amazon Bedrock and SageMaker JumpStart for building, PartyRock for playing, Amazon Q for using, and Trainium, Inferentia and the GPU fleet underneath, ending with how each of them is priced and what you are trading when you choose one.

The map

Read this first when short on time. Every branch is a section below.

Advantages: adaptability, responsiveness and simplicity

Generative AI earns its place because one model adapts to many tasks without retraining, answers in seconds in natural language, and removes the data collection and feature engineering that a classical machine learning project needs before it produces anything.

The exam guide names three advantages and they are worth taking one at a time, because each one is also the reason a specific limitation exists.

Adaptability

A classical model does one thing. A foundation model does the thing the prompt describes. The same Claude model that summarises a contract this morning classifies support tickets this afternoon and drafts SQL tonight, and none of those tasks appeared in a training plan. New requirements become new prompts, not new projects, and a task that would never have justified a data science team (rewrite these 300 product descriptions in a friendlier tone) becomes a ten-minute job. Adaptability also means graceful handling of inputs nobody anticipated: a misspelled question, a mixed-language email, a screenshot instead of a form.

Responsiveness

A foundation model answers in seconds and in the user's own language, and it streams its answer token by token so the wait feels shorter than it is. That is a different interaction from a nightly batch job or a form with fifteen fields. A customer types "my order arrived broken, what now" and gets a relevant answer, not a menu. A developer writes a comment and the code appears below it. Responsiveness is why generative AI spread first into chat, search and coding assistants: the places where a human is waiting.

Simplicity

Before generative AI, using machine learning meant collecting a labelled dataset, engineering features, training, tuning and deploying, months of work described in the SageMaker lifecycle article. A foundation model skips all of it for a first version: an API call with a prompt is the whole system. There is no training data to gather because the model already knows English, law, code and chemistry to a useful degree, and no infrastructure to run because the model is behind a managed endpoint. The simplicity is real, and it is why the exam repeatedly pairs "lowest operational overhead" with Bedrock.

flowchart TD
  M["One foundation model<br/>weights fixed, behaviour set by the prompt"]
  A["Summarise a 40-page contract<br/>Classify support tickets by urgency"]
  B["Draft SQL from a plain question<br/>Rewrite 300 product descriptions"]
  M --> A
  M --> B

adaptability

The ability of one foundation model to perform many different tasks, chosen at run time by the prompt, without retraining.

responsiveness

Fast, conversational output in natural language, often streamed token by token as it is generated.

simplicity

Getting useful output from an API call and a prompt, with no dataset, feature engineering or training pipeline of your own.

Limitations: hallucination, nondeterminism, bias and the other quiet failures

Foundation models fail in ways that look like success: fluent text that is wrong, answers that change between runs, prejudices absorbed from the training data, knowledge that stops at a date, and a bill and a delay attached to every call.

The exam guide lists these limitations, and every one of them is a question waiting to happen, usually paired with the AWS feature that mitigates it. Take them in turn.

Hallucination

A hallucination is a confident, fluent statement that is false: an invented court case, a product feature that does not exist, a function that was never in the library. It is not a bug to be fixed but a consequence of how the model works: it predicts plausible next tokens, and a plausible-sounding citation is exactly what next-token prediction produces when the true one is not in the weights. Hallucinations cluster where the model lacks knowledge (your company's data, anything after its cut-off, niche facts), where the prompt pressures it to answer rather than decline, and where a long chain of reasoning lets an early error compound.

Interpretability

A foundation model has billions of weights and no readable rule for why it produced a sentence. You can ask it to explain itself, but the explanation is another generated text, not a trace of the computation. That is a problem in regulated domains where a decision must be justified, and it is the reason the responsible AI article spends time on explainability tools and on choosing a simpler model where a reason is required.

Inaccuracy

Separate from hallucination, models are simply imprecise at some things: arithmetic on long numbers (because numbers are chopped into tokens), counting, exact quotation, following twenty constraints at once. Where the output must be exactly right, have the model call a tool that is (a calculator, a database query) rather than trust its own answer.

Nondeterminism

The same prompt can give a different answer each time because the next token is sampled, not chosen. That is fine for brainstorming and fatal for a test suite. Lowering temperature towards zero makes output nearly deterministic; asking for structured output (JSON with fixed keys) reduces variation in shape even when wording varies.

Bias

The model learned from text people wrote, with all the prejudice that text contains, and then from human raters whose preferences have their own slant. Bias appears as stereotyped associations, uneven quality across languages and dialects, and images that default to one kind of face. Curated training data, evaluation across demographic groups and output filters reduce it; they do not remove it. Amazon SageMaker Clarify measures bias in data and predictions, and Bedrock Guardrails can block the worst outputs.

Knowledge cut-off

The model knows nothing after the date its training data was gathered. It will describe last year's product line as current and be unaware of a law passed last month. Retrieval is the fix: current facts arrive in the prompt at question time. Fine-tuning does not solve it well, because fine-tuning teaches style and format far more reliably than facts.

Cost and latency

Every call costs money per token and takes seconds, more for long outputs since each output token is a full pass over the sequence. A chat application re-sends its history every turn, so cost grows with conversation length. Large models are slower and dearer than small ones. The levers are shorter prompts, capped output length, smaller models where they pass, caching common answers and batch processing where nobody is waiting.

Prompt sensitivity

Small changes in wording, order or formatting of the prompt can change the answer a lot, which makes behaviour hard to guarantee and makes prompts something to version and test like code. It also opens the door to prompt injection, where text inside a document or a user message rewrites the model's instructions.

flowchart TD
  U["User question"]
  R["Retrieve facts<br/>Knowledge Base adds the source text<br/>fixes cut-off and missing knowledge"]
  P["Prompt with instructions<br/>answer only from context, low temperature<br/>fixes guessing and nondeterminism"]
  M["Foundation model"]
  G["Guardrails on the output<br/>grounding check, content and PII filters<br/>fixes unsupported claims and harmful text"]
  H["Human review for high stakes<br/>Amazon A2I on sampled or low-confidence answers"]
  U --> R --> P --> M --> G --> H

hallucination

Output that is fluent and confident but false or unsupported by any source, produced because the model predicts plausible text rather than checking facts.

interpretability

The degree to which a human can understand why a model produced an output. Foundation models score low; their explanations are themselves generated text.

nondeterminism

Different outputs for the same input, caused by sampling the next token; reduced by lowering temperature.

knowledge cut-off

The date after which the model's training data contains nothing; anything newer must be supplied in the prompt.

prompt sensitivity

Large changes in output from small changes in the prompt's wording, order or format.

Choosing a model: the factors that decide

Selecting a foundation model means listing what the task needs (modality, quality, latency, throughput), what the business allows (cost, compliance, licence, region, data handling), and how much you intend to customise, then testing the shortlist on your own data and picking the smallest model that passes.

The Bedrock catalogue alone has dozens of models across seven providers, and the temptation is to take the most capable. The exam, and a sensible budget, prefer the model that meets the requirement at the lowest cost and latency. The factors below are the ones the exam guide names, grouped by the question they answer.

flowchart TD
  S["List the requirement<br/>task, modality, quality bar, latency, volume, region, compliance"]
  Q1{"Modality?"}
  T["Text and reasoning<br/>Claude, Nova, Llama, Mistral, Command, Jamba"]
  I["Images or video out<br/>Nova Canvas, Titan Image, Stability AI, Nova Reel"]
  V["Vectors<br/>Titan Embeddings, Cohere Embed"]
  E["Shortlist two or three sizes<br/>evaluate on your own prompts and data"]
  Q2{"Passes?"}
  C["Take the smallest that passes<br/>then choose the pricing mode by traffic shape"]
  U["Step up a size, add retrieval,<br/>or consider fine-tuning"]
  S --> Q1
  Q1 -- text --> T
  Q1 -- media --> I
  Q1 -- search --> V
  T --> E
  I --> E
  V --> E
  E --> Q2
  Q2 -- yes --> C
  Q2 -- no --> U
  U --> E

The evaluation step is not optional. Public benchmarks rank models on generic tasks; your task is specific. Bedrock's playgrounds let you send the same prompt to several models side by side, and Amazon Bedrock Model Evaluation runs a dataset of your prompts through candidate models and scores them automatically or with human raters. The evaluation article covers the metrics.

context window

The number of tokens a model can consider in one call; a selection factor whenever inputs are long.

open-weights model

A model whose trained weights are published under a licence (Llama, Mistral), so it can be hosted and fine-tuned anywhere, unlike a proprietary model reachable only through its provider's API.

EULA

End-user licence agreement: the provider's terms you accept when enabling a model in Bedrock, which set what you may do with it and its output.

Business value: the metrics that say it worked

A generative AI application is justified by business metrics, time saved, tasks completed, conversion, revenue per user, satisfaction, cost per outcome, and not by model metrics alone, because a model can score well on a benchmark and still change nothing a customer or an accountant notices.

The exam guide lists the metrics below. The habit to build is pairing every model-level measure (accuracy, ROUGE, toxicity) with a business-level one it is supposed to move, and measuring the second before and after the launch. A pilot that cannot name its business metric is a demo.

Two of these deserve a note. Cross-domain performance is the metric of adaptability: it asks whether the one model you deployed is genuinely serving every department or quietly failing in the one whose vocabulary it did not know. Efficiency must include the token bill: a chatbot that resolves 80 percent of tickets but costs more per resolution than a human is not efficient, and the fix (a smaller model, shorter prompts, caching) is technical.

conversion rate

The fraction of users who take the desired action, such as a purchase or a sign-up.

average revenue per user (ARPU)

Total revenue in a period divided by the number of users in that period.

customer lifetime value (CLV)

The total revenue a business expects from one customer over the whole relationship.

cross-domain performance

How consistently one model performs across different subject areas or business functions.

Three layers: how the AWS generative AI stack is arranged

AWS organises its generative AI offering in three layers: infrastructure to train and run models (chips, instances, clusters), tools to build with models (Amazon Bedrock, Amazon SageMaker), and finished applications that use models (Amazon Q, PartyRock).

The first article drew the whole AI stack; this is the generative slice of it. The layering is how AWS talks about itself and how the exam frames "which service" questions: a scenario that wants to use generative AI for a business task with no code is at the top; one that wants to build an application on a foundation model is in the middle; one that wants to train or host a model at scale is at the bottom. Most customers live in the middle layer and never touch the bottom one directly, because Bedrock hides it.

flowchart TD
  A["Applications<br/>Amazon Q Business, Developer, QuickSight, Connect · PartyRock<br/>your own apps on Bedrock Agents and Knowledge Bases"]
  T["Tools<br/>Amazon Bedrock: serverless FMs, one API<br/>SageMaker JumpStart: model hub, your endpoints<br/>Amazon SageMaker: train, tune, host"]
  I["Infrastructure<br/>AWS Trainium for training, AWS Inferentia for inference<br/>GPU instances (P and G families)<br/>HyperPod and UltraClusters: resilient clusters at scale"]
  A --> T --> I

Amazon Bedrock: the middle layer in one API

Amazon Bedrock is a fully managed, serverless service that offers foundation models from Amazon and other providers behind one API, together with the managed pieces an application needs around a model: knowledge bases for retrieval, agents for multi-step tasks, guardrails for safety, customisation, evaluation and logging.

The next article is the full reference; this section places Bedrock in the stack and lists what the exam expects you to associate with it. You choose a model from the catalogue (Amazon Titan and Nova, Anthropic Claude, Meta Llama, Mistral AI, Cohere, AI21 Labs, Stability AI), call it with a prompt and inference parameters, and pay per token or per image on demand. There is no server to size. Your prompts and completions are not used to train the providers' models, are encrypted in transit and at rest, and stay in the region you called. Around the model, Knowledge Bases give managed RAG over your documents, Agents plan and call your APIs, Guardrails filter input and output, custom models let you fine-tune or continue pre-training some models on your data, and Model Evaluation compares candidates.

Amazon SageMaker JumpStart: the model hub you deploy yourself

SageMaker JumpStart is a catalogue inside Amazon SageMaker of pre-trained models and solution templates that you deploy with one click to an endpoint in your own account and can fine-tune with your own data, trading Bedrock's convenience for a wider choice of models and full control of the instance they run on.

JumpStart lives in SageMaker Studio. It lists hundreds of models: open-weights foundation models (Meta Llama, Mistral and Mixtral, Falcon, Stable Diffusion, Hugging Face text and vision models), proprietary models from the AWS Marketplace, and classical models for tabular and vision tasks, plus end-to-end solution templates (document summarisation, demand forecasting) with notebooks. For a foundation model, you pick it, choose an instance type, and JumpStart creates a SageMaker endpoint in your account running that model; you call the endpoint like any SageMaker model. To fine-tune, you point JumpStart at a dataset in S3 and it runs a SageMaker training job that produces your own version of the model, which you then deploy the same way. The model artefacts never leave your account and the endpoint sits inside your VPC if you want it to.

flowchart TD
  H["JumpStart model hub<br/>Llama, Mistral, Falcon, Stable Diffusion, Hugging Face"]
  P["Pick a model and an instance type<br/>for example an ml.g5 GPU instance"]
  F["Optional: fine-tune<br/>training job on your data in S3"]
  E["SageMaker endpoint in your account<br/>you pay per instance-hour while it runs"]
  A["Your application<br/>calls InvokeEndpoint"]
  H --> P --> F --> E --> A

JumpStart versus Bedrock

SageMaker endpoint

A hosted HTTPS endpoint running your model on instances you chose, billed per instance-hour; real-time, serverless and asynchronous variants exist.

model hub

A catalogue of ready-to-deploy pre-trained models; JumpStart is SageMaker's, Hugging Face Hub is the best-known public one.

solution template

A JumpStart package that deploys a complete example (data, model, notebook, endpoint) for a business problem in one step.

PartyRock: the Bedrock playground for no-code apps

PartyRock is a free web playground built on Amazon Bedrock where anyone can build, share and remix small generative AI applications by wiring text, image and chat widgets to prompts, with no code and no AWS account.

PartyRock exists to teach prompting and to let people try ideas before they build them properly. An app is a page of widgets: a text input, a "generate text" widget whose prompt references the input, an image generation widget, a chatbot widget. You describe the app you want and PartyRock drafts the widgets, or you build them by hand and edit each prompt. Apps are shareable by link and any user can remix one to see and change its prompts, which is why it works as a learning tool: every app is a worked example of prompt engineering with the answers visible. It uses Bedrock models behind the scenes (Anthropic Claude, Amazon Nova, Meta Llama and others), and it is free within usage limits, with a sign-in through social accounts rather than an AWS account.

What PartyRock is not: a production service. There is no API, no SLA, no VPC and no place for company data. The path from a PartyRock idea to a product is to rebuild it on Bedrock with the same prompts.

Amazon Q: the assistants at the top of the stack

Amazon Q is AWS's family of ready-made generative AI assistants: Amazon Q Business answers questions over a company's own data with permissions respected, Amazon Q Developer helps write code and operate AWS, and Amazon Q inside QuickSight and Connect brings the same assistant to business intelligence and contact centres.

Where Bedrock gives you the parts, Amazon Q gives you the finished assistant. You do not choose a model, write a prompt template or set up a vector store; you connect data, assign users and use it. All Amazon Q products run on Bedrock and inherit its data handling: your content is not used to train the underlying models.

Amazon Q Business

Amazon Q Business is an enterprise assistant that answers employees' questions from the company's own documents and systems, with citations. You connect data sources through built-in connectors (Amazon S3, SharePoint, Confluence, Salesforce, ServiceNow, Jira, Google Drive, Slack, web crawlers and dozens more) and Q Business indexes the content. Users sign in through AWS IAM Identity Center, and every answer respects the permissions of the source: a user only gets answers drawn from documents they were already allowed to open. Plugins let Q take actions in connected systems (create a Jira ticket, open a ServiceNow case), and Amazon Q Apps let employees build small purpose-built apps from a description, PartyRock-style but on company data. It is priced per user per month, in tiers.

flowchart TD
  S["Data sources<br/>S3, SharePoint, Confluence, Salesforce, Jira, web"]
  C["Connectors crawl and index<br/>content plus each document's access list"]
  I["Amazon Q Business<br/>retrieval over the index, generation on Bedrock"]
  U["Employee signs in<br/>IAM Identity Center"]
  A["Answer with citations<br/>only from documents this user may open"]
  P["Plugins take actions<br/>create a ticket, update a record"]
  S --> C --> I
  U --> I
  I --> A
  I --> P

Amazon Q Developer

Amazon Q Developer, formerly Amazon CodeWhisperer, is the assistant for people who build on AWS. In the IDE (VS Code, JetBrains, Visual Studio, Eclipse) and the command line it completes code as you type, answers questions in a chat panel with awareness of your project, and runs agents that implement a feature across files from a description, write unit tests, document code, and scan for security vulnerabilities. Its code transformation agent upgrades Java applications across major versions. In the AWS Management Console it answers questions about services, explains errors, helps diagnose issues and writes CLI commands; it is also in Slack and Teams through AWS Chatbot. There is a free tier and a Pro tier with higher limits and enterprise controls.

Amazon Q in QuickSight

Inside Amazon QuickSight, the BI service, Amazon Q lets business users ask questions of their data in plain language and get a chart back, build dashboards by describing them, generate executive summaries of a dashboard, and produce "data stories", narrative documents assembled from the data.

Amazon Q in Connect

Inside Amazon Connect, the contact centre service, Amazon Q listens to the live conversation and offers agents suggested responses and relevant knowledge articles in real time, so the agent does not search while the customer waits. Together with generative post-contact summaries, it is the contact-centre face of the same assistant.

connector

A built-in integration that crawls a data source (SharePoint, Salesforce, S3, Confluence and others) into Amazon Q Business's index along with each document's permissions.

IAM Identity Center

The AWS single sign-on service that Amazon Q Business uses to know who the user is and therefore which documents they may see.

plugin

An Amazon Q Business integration that lets the assistant perform an action in another system, such as creating a ticket.

code transformation

The Amazon Q Developer agent that upgrades an application's code between language versions, notably Java 8 or 11 to Java 17.

The infrastructure layer: Trainium, Inferentia, GPUs, HyperPod and UltraClusters

Underneath every model on AWS are accelerators (AWS's own Trainium and Inferentia chips and NVIDIA GPUs), grouped into resilient clusters (SageMaker HyperPod) and very large networked fleets (EC2 UltraClusters), and the exam expects you to know which is for training and which for inference.

Foundation models are made of matrix multiplications, and the hardware that does those fastest per dollar decides who can afford to train and serve them. AWS offers two routes. NVIDIA GPU instances are the general answer: the P family (P4d with A100, P5 with H100 and newer generations) for training and heavy inference, the G family (G5 and G6, with A10G and L4-class GPUs) for cost-efficient inference and graphics. AWS's own silicon is the specialised answer, built to cut the cost of exactly this workload.

• AWS Trainium is the training chip, in Trn1 and Trn2 instances, designed to train large models at a lower cost than comparable GPU instances. Amazon Nova was trained on it, and Anthropic trains on Trainium clusters. It also serves inference for large models.
• AWS Inferentia is the inference chip, in Inf1 and Inf2 instances, built for high-throughput, low-cost serving of trained models. Inf2 is sized for large language models and diffusion models; Bedrock and Amazon's own services run inference on it where it fits.
• AWS Neuron is the SDK that compiles PyTorch and JAX models for Trainium and Inferentia, so the code does not change.
• Amazon SageMaker HyperPod is a managed cluster service for long distributed training jobs: it provisions the accelerators, monitors node health, replaces faulty nodes and resumes from the last checkpoint automatically, so a weeks-long run survives hardware failures without a human waking up.
• Amazon EC2 UltraClusters are the very large placement groups of P-family or Trainium instances joined by petabit-scale, low-latency networking (Elastic Fabric Adapter), which is what training across thousands of accelerators requires.

accelerator

A processor built for the parallel arithmetic of neural networks: a GPU, or a purpose-built chip such as Trainium or Inferentia.

AWS Neuron

The compiler and runtime SDK that runs PyTorch and JAX models on Trainium and Inferentia.

Elastic Fabric Adapter (EFA)

The low-latency network interface that lets thousands of instances in an UltraCluster exchange gradients fast enough for distributed training.

checkpoint

A saved snapshot of model weights during training, from which a run resumes after a failure instead of starting over.

Why build on AWS: service advantages and infrastructure benefits

The case for AWS's generative AI services is that they make foundation models accessible to teams without ML expertise, at lower cost and faster than building from scratch, on infrastructure whose security, compliance and data-handling promises are already audited.

The exam guide splits the argument into two lists, and a question may draw on either.

Advantages of using AWS generative AI services

• Accessibility. Any developer with an AWS account can call a frontier model with an API call; a business user can use Amazon Q with none.
• Lower barrier to entry. No GPUs to buy, no model to train, no ML team required for a first version.
• Efficiency. Managed retrieval, agents, guardrails and evaluation replace months of plumbing, and the service scales without capacity planning.
• Cost-effectiveness. Pay per token or per user, with no idle hardware, and a ladder of model sizes and pricing modes to fit the workload.
• Speed to market. A prototype in a day and a production system in weeks, because the hard parts are services rather than projects.
• Ability to meet business objectives. The point of all the above: the metrics of the earlier section move sooner and with less risk.

Benefits of AWS infrastructure for generative AI

• Security. IAM controls who may call which model; data is encrypted in transit and at rest with AWS KMS keys you can own; Bedrock is reachable over private VPC endpoints through AWS PrivateLink so traffic never crosses the internet.
• Compliance. Bedrock and SageMaker are in scope for the major programmes: HIPAA eligible, GDPR-ready, SOC 1, 2 and 3, ISO 27001 and others; AWS Artifact holds the reports. The governance article covers this properly.
• Responsibility. AWS publishes AI service cards and provides Guardrails, SageMaker Clarify and model evaluation so that responsible use is a feature rather than an aspiration.
• Safety. Guardrails filter harmful content, PII and off-topic requests on input and output; Titan and Nova images carry invisible watermarks.
• Your data stays yours. Prompts, completions and fine-tuning data are never used to train the base models, are not shared with model providers, and stay in the AWS account and region where you used them; custom models are private to your account.

AWS PrivateLink

Private connectivity from your VPC to an AWS service over the AWS network, exposed as a VPC endpoint, so calls to Bedrock need no internet route.

AWS KMS

Key Management Service: the managed keys that encrypt data at rest; customer-managed keys give you control over who can decrypt.

AWS Artifact

The portal for downloading AWS's compliance reports (SOC, ISO, PCI) and agreements such as the HIPAA BAA.

AI service card

AWS's published document for an AI service describing its intended use, limitations and responsible-use guidance.

Cost trade-offs: tokens, pricing modes, model size and placement

The cost of a generative AI application is set by how many tokens it moves, which pricing mode it uses (on-demand, provisioned throughput or batch), how big a model it calls, and where and how redundantly it runs, and every one of those trades money against responsiveness, availability or quality.

Token-based pricing

Bedrock on-demand pricing charges per 1,000 (or per million) tokens, with separate rates for input tokens (what you send) and output tokens (what the model writes), and output is the dearer of the two. Image models charge per image, with resolution and quality steps. Larger models charge more per token than smaller ones in the same family. Two consequences: the prompt (system instructions, retrieved chunks, history) is the bulk of most bills, because it is re-sent every turn, and capping the maximum output length is the simplest cost control. Prompt caching, where a provider reuses the processed prefix of a repeated long prompt at a discount, is the newer lever for applications with a big fixed preamble.

Pricing modes

flowchart TD
  S["A workload to price"]
  Q1{"Custom model?"}
  PT["Provisioned throughput<br/>model units, hourly or committed"]
  Q2{"Anyone waiting?"}
  BA["Batch inference<br/>JSONL in S3, discounted tokens"]
  Q3{"Steady and heavy?"}
  OD["On-demand<br/>pay per input and output token"]
  S --> Q1
  Q1 -- yes --> PT
  Q1 -- no --> Q2
  Q2 -- no --> BA
  Q2 -- yes --> Q3
  Q3 -- yes --> PT
  Q3 -- no --> OD

Custom model hosting

A fine-tuned or continued pre-trained model in Bedrock costs three ways: the training job (per token processed), monthly storage for the custom model, and provisioned throughput to run it, because custom models cannot be invoked on demand. That last item is what makes customisation a decision rather than a default: a fine-tuned model that runs one hour a day still pays for its reserved units. On JumpStart the equivalent is the endpoint's instance-hours.

Model size versus cost and latency

Within a family, a larger model gives better reasoning and instruction following, costs more per token and answers more slowly. A small model (Nova Micro, Claude Haiku class) is the right default for classification, extraction, routing and short answers; a mid-size model for most generation; the largest for complex reasoning and agents. Because latency scales with output length as well as size, a small model with a short output is dramatically cheaper and faster than a large one with a long one. Bedrock's latency-optimised inference option on some models trades a little cost for faster tokens where a human is waiting.

Responsiveness, availability, redundancy and regional coverage

Cost also buys placement. Running in one region on demand is cheapest; a latency-sensitive global product may need the model in several regions and a multi-region design with failover, which multiplies provisioned units or requires cross-region inference, where Bedrock routes requests across regions within a geography to absorb bursts. Not every model is in every region, so regional coverage can force a model choice or a cross-region profile. Redundancy (a second model or region as fallback) improves availability and costs the standby capacity. Each step up in responsiveness, availability and redundancy is a step up in spend; the requirement, not the maximum, should set the level.

on-demand

Pay per token or image processed, with no commitment; the default Bedrock pricing mode for base models.

provisioned throughput

Reserved model units that guarantee a level of throughput for a model, billed hourly with optional commitments; required to run custom models.

batch inference

Submitting many prompts as a file in S3 and receiving the results as a file, at a discount, with turnaround in hours rather than seconds.

model unit

The unit of provisioned throughput: a fixed number of input and output tokens per minute for a given model.

cross-region inference

A Bedrock inference profile that routes requests to a set of regions within a geography to increase available throughput.

Recap