Designing applications on foundation models: model choice, RAG, vector stores and agents

A foundation model on its own is a very well-read stranger: it knows a great deal about the world up to the day its training stopped, nothing about your company, and it cannot look anything up or press a button. Building an application on one is the work of fixing those three gaps. You pick a model whose size, price and modality fit the job, tune the knobs that shape its answers, hand it your own documents at question time through Retrieval Augmented Generation, and, when the task needs more than one step, wrap it in an agent that can call tools. This article walks that path in the order a team actually takes it, and names the AWS service at every step, because Domain 3 of the exam is the largest and most of its questions are "which approach, which service".

The map

Read this first when short on time. Every branch is a section below.

Choosing a model: the criteria that decide before any prompt is written

Model selection is the act of matching a task's requirements (what goes in, what comes out, how fast, how often, in which language, at what price) against the published properties of the pre-trained models on offer, and it is done before a single prompt is written because every later decision inherits it.

Amazon Bedrock puts dozens of models from several providers behind one API, and SageMaker JumpStart adds hundreds of open-weight ones, so "which model" is a real decision with real consequences. The exam lists a specific set of criteria, and a question usually makes one of them the deciding factor while the rest are noise. Learn to spot which one is doing the work.

Two shortcuts help. First, the task decides the modality and the floor on complexity; everything else is a trade against cost. Second, the context window decides the architecture: if the knowledge the model needs does not fit in the window, or changes often, the answer is RAG rather than a bigger model. A model card, or the Bedrock model catalogue, states the window, the modalities, the languages, the customisation support and the Regions, so these are facts to look up, not to guess.

token

The unit a model reads and writes, roughly three quarters of an English word. Pricing, context windows and length limits are all counted in tokens.

context window

The maximum number of tokens a model can attend to in a single call: system prompt, conversation history, retrieved documents and the answer all share it.

open weights

A model whose trained parameters are published for download, so it can be hosted, inspected and fine-tuned by anyone the licence allows. Contrast with a proprietary model reachable only through its provider's API.

provisioned throughput

Bedrock's mode of buying dedicated model capacity by the hour instead of paying per token; required for running customised models. The Bedrock article covers the pricing modes.

Inference parameters: shaping the answer without touching the model

Inference parameters are the settings sent with each request that control how the model turns its probability distribution over the next token into an actual token, and how long it keeps going; they change the character of the output without changing a single weight.

At every step a language model produces a score for every token in its vocabulary. Something has to turn tens of thousands of scores into one chosen token, and the sampling parameters are the rules for that choice. The length parameters then say when to stop. The Bedrock article shows where they live in the InvokeModel and Converse requests; here the point is what each one does to the answer.

flowchart TD
  A["Model scores every token in the vocabulary"] --> B["Temperature reshapes the scores<br/>low sharpens, high flattens"]
  B --> C["Top-k keeps only the k most likely tokens"]
  C --> D["Top-p keeps the smallest set whose<br/>probabilities add up to p"]
  D --> E["Sample one token from what is left"]
  E --> F{Stop?}
  F -- "stop sequence or max tokens" --> G["Return the response"]
  F -- "no" --> A

The combination matters. A temperature of 0 makes the other sampling parameters irrelevant, because the top token is always taken. A high temperature with a tight top-p still produces varied but sensible text, because the tail of nonsense tokens has been cut off before sampling. Length settings interact with cost directly: output tokens are the expensive ones, and a generous max tokens on a chatty model is where a bill grows.

greedy decoding

Always taking the single most probable next token; equivalent to temperature 0 or top-k of 1. Repeatable, but can loop or sound flat.

nucleus sampling

Another name for top-p: sample only from the smallest "nucleus" of tokens whose probabilities sum to p.

Retrieval Augmented Generation: give the model the facts at question time

Retrieval Augmented Generation, RAG, is the pattern of looking up the passages relevant to a question in your own documents and pasting them into the prompt, so the model answers from evidence it was just shown instead of from memory it may not have.

A foundation model's knowledge is frozen at its training cut-off and never included your intranet, your contracts or last week's price list. Asked about them it will either decline or, worse, invent a plausible answer, which is the hallucination problem described in the capabilities article. RAG fixes this without retraining: the knowledge lives in a searchable index outside the model, and the model's job shrinks to reading the retrieved passages and composing an answer from them, ideally with citations to the passages it used.

RAG has two phases that run at different times. Ingestion happens whenever documents change: load them, split them into chunks, turn each chunk into an embedding vector, and store vector plus text plus metadata in a vector store. Query happens at every request: embed the question with the same model, find the nearest chunks, put them into the prompt with the question, and generate. The next two sections take each phase in turn; this one covers ingestion and the reasons the pattern wins.

flowchart TD
  S["Documents in Amazon S3<br/>PDF, HTML, Word, wiki exports"] --> L["Load and parse<br/>extract text, keep title and URL as metadata"]
  L --> C["Chunk<br/>fixed size with overlap, hierarchical or semantic"]
  C --> E["Embed each chunk<br/>Titan Text Embeddings V2 or Cohere Embed"]
  E --> V["Store vector, text and metadata<br/>OpenSearch Serverless, Aurora pgvector, others"]
  V --> I["Vector index ready for search"]
  I -. "documents change: sync again" .-> S

The four ingestion steps

1. Load. Pull documents from wherever they live (an S3 bucket, a SharePoint site, Confluence, Salesforce, a web crawl) and extract plain text plus whatever metadata will be useful later: title, URL, author, date, access group. Amazon Bedrock Knowledge Bases connects to S3 and to several such sources directly.
2. Chunk. Split each document into pieces small enough to embed well and to fit several into a prompt. Fixed-size chunking cuts every few hundred tokens with some overlap so that a sentence straddling a boundary appears in both chunks. Hierarchical chunking keeps small child chunks for precise matching and larger parent chunks for context. Semantic chunking splits where the meaning changes. Bedrock offers all three plus "no chunking" for documents that are already the right size.
3. Embed. Run each chunk through an embedding model to get a vector of a few hundred to a few thousand numbers that encodes its meaning. Amazon Titan Text Embeddings V2 and Cohere Embed are the Bedrock options; the same model must be used at query time.
4. Store. Write vector, chunk text and metadata into a vector store that can answer "which stored vectors are nearest to this one" quickly. The store also keeps the metadata so results can be filtered by, say, product line or access group before ranking.

Why RAG rather than fine-tuning for knowledge

• Freshness. Re-syncing an index takes minutes; fine-tuning takes hours and produces a new model that must be redeployed. Knowledge that changes weekly cannot live in weights.
• Private data stays data. Documents remain in S3 or a database under IAM, encryption and access controls. A fine-tuned model has absorbed the data into weights where it cannot be deleted, audited or permission-checked per user.
• Citations. The answer can point at the exact passages used, which lets a human verify it. A fine-tuned model cannot say where it learned something.
• Less hallucination. Given the evidence and told to answer only from it, a model invents far less. Fine-tuning teaches style and format well but is a poor way to inject facts; models fine-tuned on facts still confabulate.
• Cost. No training run, no provisioned throughput for a custom model, just embedding and storage costs plus longer prompts.

chunk

A piece of a document small enough to embed as one vector and to include in a prompt alongside others; typically a few hundred tokens with a little overlap to its neighbours.

embedding model

A model that maps text (or images) to a fixed-length vector such that similar meanings land close together. It does not generate text; it only measures.

data source sync

The Bedrock Knowledge Bases job that re-reads a data source, re-chunks and re-embeds what changed, and updates the vector index.

The query path: from a question to a cited answer

At request time RAG embeds the user's question, asks the vector store for the chunks nearest to it, builds a prompt that contains the question and those chunks with an instruction to answer only from them, and sends that prompt to the generating model, which replies with an answer and the citations that support it.

sequenceDiagram
  autonumber
  participant U as User
  participant A as Application
  participant V as Vector store
  participant M as Foundation model
  U->>A: "What is the refund window?"
  A->>A: embed the question with the same model
  A->>V: k nearest chunks, filter by region and product
  V-->>A: top 5 chunks with text and source metadata
  A->>A: build prompt: instruction, chunks, question
  A->>M: augmented prompt
  M-->>A: answer grounded in the chunks
  A-->>U: answer plus citations to the sources

1. Embed the question. The user's text goes through the same embedding model that indexed the documents. Mixing models here is a silent failure: the vectors live in different spaces and nothing matches.
2. Similarity search. The store returns the k chunks whose vectors are closest to the question's vector, typically k between 3 and 10. Metadata filters narrow the candidates first: only this customer's region, only documents this user may read, only the current version.
3. Augment the prompt. The application assembles a prompt: a system instruction ("answer using only the passages below, say so if they do not contain the answer, cite the passage number"), the retrieved chunks each labelled with its source, the conversation history if any, and the question.
4. Generate with citations. The model writes the answer. Because each chunk was labelled, it can name the ones it used, and the application turns those labels back into links or page references for the user.

Two design points decide quality. The retriever's recall puts a ceiling on the answer: a fact that was not retrieved cannot be used, however good the model. And the instruction to stay grounded is what stops the model from filling gaps with memory; without it RAG degrades into "a prompt with some extra text". Bedrock Knowledge Bases writes both of these for you, including a default prompt template for the generation step that can be edited.

retriever

The component that takes a question and returns the most relevant chunks. It may be a vector store query, a keyword search, a managed service such as Kendra, or a combination.

grounding

Constraining the model to answer from supplied evidence. A "grounded" answer is one whose claims can be traced to the retrieved passages.

citation

A pointer from a statement in the answer back to the chunk it came from, carried in Bedrock responses as the source URI and the matching text span.

Embeddings and similarity: how "nearest" is measured

An embedding is a list of numbers, typically several hundred to a few thousand long, that an embedding model produces for a piece of text so that pieces with similar meaning get lists that point in similar directions; similarity search is the arithmetic of finding the stored lists that point most nearly the same way as the query's.

The fundamentals article introduced embeddings as the way a transformer represents tokens. For RAG the useful picture is geometric: every chunk becomes a point in a space with, say, 1,024 dimensions, and "the refund window is 30 days" lands near "returns are accepted within a month" even though they share no words. The question "how long do I have to return something?" lands in the same neighbourhood, and the search is simply "which stored points are closest to this one".

Distance measures

• Cosine similarity compares the angle between two vectors and ignores their length; it is the default for text because embedding lengths carry little meaning. 1 means identical direction, 0 means unrelated.
• Dot product is cosine similarity without normalising the lengths; for models that output unit-length vectors, which most do, the two are identical and dot product is cheaper.
• Euclidean (L2) distance is the straight-line distance between the points; common for image embeddings and in pgvector's default operator.

Exact versus approximate search

Comparing the query against every stored vector (exact k-nearest-neighbours, k-NN) is perfectly accurate and gets slow past a few hundred thousand vectors. Every vector store therefore offers approximate nearest neighbour (ANN) indexes that trade a little recall for a lot of speed. The common one is HNSW, a graph where each vector links to its neighbours so a search can hop towards the target in logarithmic time; the other is IVF, which clusters vectors and searches only the nearest clusters. OpenSearch's k-NN plugin, pgvector, DocumentDB and MemoryDB all implement HNSW.

k-NN

k-nearest-neighbours: return the k stored vectors closest to the query. "Exact" k-NN checks every vector; approximate variants use an index.

HNSW

Hierarchical Navigable Small World, the graph-based approximate index most vector stores use; fast queries, more memory, slower to build.

recall

In retrieval, the fraction of the truly relevant items that were returned. Approximate indexes trade a little recall for speed; the same word means something different in classification metrics.

Where RAG goes wrong, and the fixes: chunking, hybrid search, re-ranking

RAG fails in four recognisable ways (the wrong chunk boundaries, a retrieval that misses the passage that matters, a prompt that overflows the context window, and an index that no longer matches the documents) and each has a standard remedy that the exam expects you to know by name.

Hybrid search

Vector search finds meaning; keyword search finds exact strings. A question that contains "error E-4471" or a product SKU needs the second, and a question phrased differently from the document needs the first. Hybrid search runs both, normalises the scores and merges the lists, so neither kind of question falls through. OpenSearch supports this natively, and Bedrock Knowledge Bases offers a hybrid search option on stores that can do it.

Re-ranking

Embedding similarity is a cheap first pass, computed once per chunk in advance. A re-ranker is a second, slower model that reads the query and each candidate chunk together and produces a relevance score. Retrieving 25 candidates by vector and keeping the 5 best after re-ranking gives much better evidence than retrieving 5 by vector alone. Bedrock offers rerank models (from Amazon and Cohere) and a Rerank API that Knowledge Bases can call automatically.

hybrid search

Combining vector (semantic) similarity with keyword (lexical, BM25) matching and merging the two ranked lists.

re-ranking

A second scoring pass in which a model reads query and candidate together and reorders the candidates by relevance before the best are placed in the prompt.

BM25

The classic keyword relevance formula behind most full-text search engines: term frequency weighted by how rare the term is across the corpus.

Vector stores on AWS: which database holds the embeddings

A vector store is any database that can save embedding vectors alongside their text and metadata and answer nearest-neighbour queries over them with an index, and on AWS that capability is offered inside several existing databases rather than as one dedicated product, so the choice usually follows the database a team already runs.

The exam names a specific list of AWS services "for storing embeddings within vector databases", and expects you to know the keyword that distinguishes each. Here they are, with the partner stores that Bedrock Knowledge Bases also supports.

flowchart TD
  Q1{Graph relationships?} -- yes --> N["Amazon Neptune Analytics, GraphRAG"]
  Q1 -- no --> Q2{Lowest latency?}
  Q2 -- yes --> M["Amazon MemoryDB vector search"]
  Q2 -- no --> Q3{Already PostgreSQL?}
  Q3 -- yes --> P["Aurora PostgreSQL or RDS with pgvector"]
  Q3 -- no --> Q4{JSON documents?}
  Q4 -- yes --> D["Amazon DocumentDB vector search"]
  Q4 -- no --> Q5{Many sources?}
  Q5 -- "yes, with permissions" --> K["Amazon Kendra as retriever"]
  Q5 -- no --> O["Amazon OpenSearch Serverless, the default"]

Bedrock Knowledge Bases also introduced Amazon S3 Vectors, a vector storage type inside S3 itself aimed at very large, cost-sensitive indexes where sub-second rather than millisecond latency is acceptable. It is newer than the exam guide, so recognise it as the low-cost option rather than expecting it as an answer.

pgvector

An open-source PostgreSQL extension that adds a vector column type, distance operators and HNSW and IVFFlat indexes; available on both Aurora PostgreSQL and RDS for PostgreSQL.

GraphRAG

RAG that also builds a knowledge graph of entities and relationships from the documents and retrieves along the graph as well as by similarity; in Bedrock it is backed by Neptune Analytics.

vector search collection

The OpenSearch Serverless collection type dedicated to k-NN workloads, created automatically by Bedrock Knowledge Bases.

The customisation ladder: prompting, RAG, fine-tuning, pre-training and what each costs

There are four ways to make a foundation model do your task better, and they form a ladder of rising cost, effort and data requirements: in-context learning through prompts, Retrieval Augmented Generation, fine-tuning, and training a model from scratch; the rule is to climb only as high as the problem forces you.

The training and evaluation article covers how fine-tuning and pre-training actually work. This section is about the choice between them, which is the way the exam asks.

flowchart TD
  S["A task the base model does imperfectly"] --> Q1{Prompt enough?}
  Q1 -- yes --> P["Prompt engineering, few-shot examples<br/>cheapest, change any time"]
  Q1 -- no --> Q2{Missing knowledge?}
  Q2 -- "yes, and it changes or is private" --> R["RAG with a knowledge base<br/>fresh, cited, no training"]
  Q2 -- no --> Q3{Style or format?}
  Q3 -- "yes, prompts cannot pin it down" --> F["Fine-tuning on labelled pairs<br/>custom model, provisioned throughput"]
  Q3 -- no --> Q4{New domain?}
  Q4 -- "yes, vocabulary the model lacks" --> C["Continued pre-training on domain text"]
  Q4 -- no --> T["Pre-train a new model<br/>model providers only"]

Two of these are often combined. A model fine-tuned to speak in the company's voice and follow its output schema, used inside a RAG pipeline that supplies the facts, is a common production shape: fine-tuning for how, RAG for what. And in-context learning never goes away; even a fine-tuned model gets a system prompt.

in-context learning

A model's ability to pick up a task from instructions and examples inside the prompt, with no weight updates. Zero-shot, one-shot and few-shot prompting are its forms; the prompt engineering article goes into them.

fine-tuning

Further supervised training of a pre-trained model on labelled examples so that it adopts a behaviour, style or domain; produces a new custom model.

continued pre-training

Further self-supervised training on unlabelled domain text, so the model learns the vocabulary and facts of a field. Bedrock calls it "continued pre-training"; it is also written "continuous pre-training".

Agents: a model that plans, calls tools and checks its work

An agent is an application loop in which a foundation model is asked not for a final answer but for the next action, the application performs that action (a search, an API call, a database query), feeds the result back, and repeats until the model decides the task is done; it is how a model that can only produce text gets to do things.

A single prompt answers a single question. "Book me the cheapest flight to Bengaluru next Friday and add it to my calendar" needs several steps, each depending on the last: look up flights, compare prices, book one, read the confirmation, create the event. No amount of prompting lets one model call do that, because the model cannot see the flight prices until something fetches them. Agents solve this with three ingredients.

• Planning. The model breaks the goal into steps and decides which to take next. The common prompting style is ReAct (reason and act): the model writes a short thought, names an action with its arguments, receives the observation, and thinks again. Chain-of-thought prompting, covered in the next article, is the reasoning half of this.
• Tool use. The model is given descriptions of the tools available (name, purpose, parameters) and produces a structured call when it wants one. The application, never the model, executes the call, and returns the result as text. Tools are APIs, Lambda functions, database queries, calculators, code interpreters, or a knowledge base search.
• Memory. Short-term memory is the conversation and the observations so far, carried in the context window. Long-term memory is anything persisted between sessions (user preferences, past summaries) and retrieved when relevant, which is RAG again.

flowchart TD
  G["User goal<br/>book the cheapest flight and add it to my calendar"] --> T["Think: what is the next step?<br/>the model plans"]
  T --> Q{Need a tool?}
  Q -- yes --> A["Act: call a tool with arguments<br/>searchFlights, bookFlight, createEvent"]
  A --> O["Observe: the tool result comes back as text"]
  O --> T
  Q -- "no, the task is complete" --> R["Respond to the user with the outcome"]

The loop is what makes agents both useful and risky. Useful, because the model can recover from a failed call, ask a clarifying question, or try another route. Risky, because a model that is wrong about the next step will act on it, so tools must be scoped to what the task needs, destructive actions should require confirmation, and the number of laps should be bounded.

ReAct

A prompting pattern in which the model alternates reasoning ("thought") with actions ("call tool X") and reads the observation before the next thought. The name is "Reason and Act".

tool (function) calling

A model's ability to output a structured request to invoke a named function with arguments, which the application executes. The Bedrock Converse API exposes this as "tool use".

orchestration

The application code (or managed service) that runs the loop: sends prompts, parses tool calls, executes them, appends results, and stops when the model finishes.

Amazon Bedrock Agents: the managed agent loop, with knowledge bases and guardrails attached

Amazon Bedrock Agents (launched as Agents for Amazon Bedrock) is the managed implementation of the loop above: you describe the agent's job in plain language, attach action groups that define the tools and the Lambda functions behind them, optionally attach knowledge bases and guardrails, and Bedrock runs the planning, tool calling and memory for you.

The Bedrock article describes the console and API mechanics; this section is about the parts and how they fit an application design.

flowchart TD
  U["User request"] --> GI["Guardrail on input<br/>denied topics, PII, prompt attacks"]
  GI --> AG["Agent: FM plus orchestration prompt<br/>plans, picks actions, keeps session memory"]
  AG <--> KB["Knowledge base<br/>RAG over company documents"]
  AG <--> AC["Action group<br/>OpenAPI schema or function details"]
  AC <--> LF["AWS Lambda<br/>calls the booking API, the database, another service"]
  AG --> GO["Guardrail on output<br/>grounding check, word filters"]
  GO --> RSP["Response to the user"]

• Instructions. A natural-language description of what the agent does and how it should behave ("You are a travel assistant for employees. Only book economy fares."). Bedrock turns it, together with the tool definitions, into the orchestration prompt.
• Action groups. Each group is a set of operations the agent may perform, defined either by an OpenAPI schema (paths, parameters, descriptions) or by function details (name, description, parameters) entered directly. Good descriptions matter: they are what the model reads to decide which tool fits.
• AWS Lambda. Each action group is backed by a Lambda function that receives the operation and parameters the agent chose and does the real work. The alternative is return of control, where Bedrock hands the chosen action back to your application to execute.
• Knowledge bases. Attaching one gives the agent a retrieval tool; the agent decides when to search and cites what it used.
• Guardrails. Amazon Bedrock Guardrails applied to the agent filter the user's input (denied topics, harmful content, prompt attack detection, PII) and the agent's output (word filters, contextual grounding checks against the retrieved sources).
• Memory. Session state carries the conversation within a session; optional long-term memory keeps summaries across sessions for the same user.
• Prompt templates and tracing. The pre-processing, orchestration, knowledge base response generation and post-processing prompts can be edited, and a trace shows every thought, action and observation, which is how an agent is debugged.
• Multi-agent collaboration. A supervisor agent can delegate to specialist agents, each with its own tools, when one agent's instructions would grow too long.

action group

A named set of operations an agent may call, described by an OpenAPI schema or function details and backed by a Lambda function or return of control.

return of control

An action group mode in which Bedrock returns the agent's chosen action and parameters to the calling application instead of invoking Lambda, so the application executes it.

contextual grounding check

A Bedrock Guardrails filter that scores whether the model's answer is supported by the retrieved sources and blocks answers that are not.

Business applications: where RAG and agents earn their keep

The exam expects you to recognise the four or five business shapes that RAG and agents fit, map each to its architecture, and pick the AWS service with the least operational overhead that meets it.

Recap