Observability and system security

Metrics, traces and logs: metrics show that something is wrong and how much, traces show where in the request path, logs show exactly what happened; they become observability when a trace id links them so you can move from an alert to a span to a log line.

• Metrics are cheap aggregates for dashboards and alerts.
• Traces are per-request trees of timed spans across services.
• Logs are the detailed events, the most expensive to store.

OpenTelemetry instruments all three with one SDK and one context, which is why the trace id ends up in metrics exemplars and log lines.

Because the average hides the tail: a mean of 80 ms is compatible with one in a hundred users waiting three seconds, and with fan-out those slow calls hit a large share of page loads.

• With 100 backends each 1 percent slow, about 63 percent of requests hit at least one slow call.
• Percentiles cannot be averaged across servers, so export histograms with shared buckets and compute the quantile at query time.
• Track p50, p95 and p99 together; the gap between them is a diagnosis in itself.

Prometheus's histogram_quantile over summed buckets is the standard way to get a fleet-wide p99.

Every distinct combination of label values is its own time series, so a label with many values multiplies series counts until the metrics store runs out of memory or money; high-cardinality identifiers belong in traces and logs, not metric labels.

• Ten endpoints times five statuses times a thousand customers is fifty thousand series for one metric.
• User ids, request ids and raw URLs are the usual offenders.
• Bound labels to enumerations: route template, method, status class, region.

Datadog and Prometheus both price or fail on series count, which is why teams learn this the hard way.

RED for the service, rate, errors and duration as p50, p95 and p99, plus USE for its resources: utilisation, saturation and errors of CPU, memory, connection pools and queues.

• RED is what users experience; USE is what will break next.
• Google's four golden signals (latency, traffic, errors, saturation) are the same set.
• Add the dependency's RED as seen from this service, since most incidents are downstream.

If the alerting is on the SLI, the dashboard exists to explain the alert, not to be watched.

The first service creates a trace id and root span, every operation opens a child span, outgoing calls carry the trace and span ids in a traceparent header so the next service continues the tree, spans are exported to a collector, and a backend stitches them into a waterfall.

• Context propagation is the fragile part: one service that drops the header splits the trace.
• Queues carry the context in message metadata; thread pools need the library's context handling.
• Head-based sampling is cheap and blind; tail-based keeps errors and slow traces.

Dapper at Google defined the model; OpenTelemetry with Jaeger, Tempo or X-Ray is the modern stack.

Structure (one JSON object with consistent fields), correlation (trace id, request id, tenant), a level, and a named event; never passwords, tokens, card numbers or full personal data.

• Structured fields make logs queryable and let the store index only what matters.
• Write to stdout and let an agent ship, buffer and enrich.
• Sample debug logs and cap retry-loop logging; volume is the cost driver.

Loki's design, indexing labels only and grepping the body, is a direct response to how expensive full-text indexing of logs became.

An SLI is a measurement of good events over total events, an SLO is the internal target for that SLI over a window, an SLA is the external contract with penalties and is always looser than the SLO; the error budget is the allowed failure implied by the SLO and is spent deliberately on velocity.

• 99.9 percent over 30 days is roughly 43 minutes of downtime or the equivalent failed requests.
• Budget remaining means ship; budget spent means releases pause and reliability work starts.
• A service cannot promise more than its dependencies deliver.

The policy is agreed in advance so the ship-versus-stabilise argument is settled by a number, not a meeting.

On burn rate: how fast the error budget is being consumed relative to the window, using several window lengths so fast burns page immediately and slow burns create tickets, with a short confirmation window so alerts resolve quickly.

• Page at 14.4× over one hour (2 percent of the monthly budget in an hour) and 6× over six hours.
• Ticket at 1× over three days.
• Alert on symptoms users feel, not causes such as CPU.

These defaults come from Google's SRE workbook and most SLO tools implement them as-is.

About 53 minutes of downtime a year, 4 minutes a month and 1 minute a week, which means no human can be in the recovery loop: detection, failover and rollback must be automatic.

• Three nines is 43 minutes a month and is where most web services sit.
• Five nines is 26 seconds a month and implies multi-region active-active with no manual steps.
• Each extra nine roughly multiplies cost and complexity, so pick the one users notice.

AWS's EC2 regional SLA is 99.99 percent, and reaching it as a customer means spreading across availability zones yourself.

Sessions for browsers talking to one backend because they are simple and instantly revocable; JWTs for mobile apps and service-to-service calls because they are stateless and carry identity across services; a JWT cannot be revoked before expiry, so keep it short-lived and revoke the refresh token instead.

• Access token lifetime of 5 to 15 minutes bounds the damage of a leak.
• A server-side denylist works for emergencies but reintroduces state.
• Store browser tokens in HttpOnly cookies, not localStorage.

Many systems combine both: a session cookie at the edge and a short JWT minted for internal calls.

OAuth 2.0 is a delegation protocol for obtaining an access token to call an API on a user's behalf; OpenID Connect adds an identity layer on top, the ID token, so the app also learns who the user is; the authorization code flow with PKCE keeps tokens out of the browser URL and proves the same client that started the login is finishing it.

• Roles: resource owner (user), client (your app), authorization server (the identity provider), resource server (your API).
• The implicit flow returned tokens in the URL fragment and is deprecated.
• PKCE replaced the client secret for public clients such as mobile apps and single-page apps.

"Sign in with Google" is OIDC; a "connect your calendar" permission screen is OAuth scopes.

With mutual TLS, where each service presents a certificate issued by an internal authority and the certificate is its identity, or with platform workload identities such as IAM roles and Kubernetes service accounts; never with shared static secrets.

• A service mesh issues and rotates short-lived certificates automatically.
• SPIFFE gives each workload a standard identity that the certificates encode.
• Short-lived credentials from the platform beat long-lived API keys in config files.

Istio turns on mTLS for every pod-to-pod call with one setting, which is the most common way teams get there.

RBAC for a small set of job roles, ABAC for contextual rules over attributes, ReBAC for sharing models built on relationships; enforce in the service at the point of action on the specific object, with policy decisions centralised in an engine.

• RBAC fails through role explosion; ABAC through unauditable policies; ReBAC needs a dedicated store to answer checks fast.
• Deny by default and least privilege apply to every model.
• Audit every sensitive allow and deny.

Google Zanzibar serves Drive's sharing checks with relationship tuples, and OpenFGA brings the same model to everyone else.

It is checking that a user is logged in but not that they may act on the specific object requested, so changing an id in the URL exposes someone else's data; prevent it by checking ownership or permission on every object access inside the service.

• It has topped the OWASP API security list because it is easy to miss and easy to exploit.
• Gateway authentication does not prevent it; only a check against the resource does.
• Scope database queries by the caller's identity rather than filtering after loading.

Non-guessable ids help slightly, but they are obscurity, not authorization.

The client sends supported ciphers and a key share, the server replies with its choice, its own key share and certificate, the client verifies the certificate chain and both derive the session key, all in one round trip; terminate at the edge for latency and certificate management, then re-encrypt inside with mTLS where the network is not trusted.

• Ephemeral key exchange gives forward secrecy.
• Resumed sessions can send data in the first flight (0-RTT).
• Automated certificate issuance (ACME, cloud certificate managers) prevents the expiry outages.

Cloud load balancers terminate TLS with managed certificates and can re-encrypt to backends in one setting.

Data is encrypted with a per-object data key, and the data key is encrypted with a master key that lives in a key management service and never leaves it; rotating the master key only re-encrypts the small data keys, never the data.

• The KMS logs and authorises every decrypt of a data key.
• Per-tenant or per-object data keys limit the blast radius of a leak.
• Server-side encryption protects against stolen disks; client-side protects against the storage provider too.

S3, EBS and RDS all use exactly this scheme with KMS keys, which is why enabling encryption there costs nothing noticeable.

Secrets live in a secrets manager, are injected at runtime, rotate automatically, and never sit in repositories, images or logs, with short-lived platform credentials preferred over long-lived keys; passwords are hashed with a slow salted algorithm such as Argon2id or bcrypt, never encrypted.

• Encryption implies a key that decrypts everything; a hash has no way back.
• Vault can issue per-service database credentials that expire.
• Scan repositories and images for leaked secrets as part of CI.

Most real breaches involve a long-lived credential in a place it should not have been, not a broken cipher.

Log in through an identity provider with the authorization code flow plus PKCE, hold a short-lived access token and a refresh token in secure device storage, validate the token at the API gateway and forward identity inward, run mTLS between services, and have each service check object-level permissions against a central policy before acting.

• Gateway: signature, issuer, audience, expiry, then rate limits per user.
• Services: trust the gateway's identity header only over mTLS; check permissions per resource; audit sensitive actions.
• Refresh tokens are revocable server-side, which is how logout and compromise are handled.

This is the shape Cognito or Auth0 plus an API gateway plus a service mesh gives you with little custom code.