Failure containment: design the blast radius before an incident does

Something will fail: a deployment with a bad configuration, a dependency that slows to a crawl, a credential that leaks, a query that deletes the wrong rows. The blast radius is how much of the system, and how many customers, that one event can affect. Every system has one. In most it was never chosen; it is simply "everything", because every component shares the same deployment, network, database and credentials.

Failure containment is the practice of choosing it. You decide in advance how far a single fault, compromise or change may spread, and you build boundaries that stop it there. The same structures that limit an outage also limit an attacker, which is why containment belongs in security architecture as much as in reliability.

It helps to separate three kinds, because different boundaries contain them.

• Failure. A bug, overload or dependency outage. Contained by isolating capacity and dependencies: cells, bulkheads, timeouts and load shedding.
• Compromise. An attacker controls a workload or credential. Contained by identity and network partitions: separate accounts, narrow roles, per-tenant keys. Our guide to identity and privilege chains covers this side.
• Change. A deployment or configuration change that is wrong. Contained by rolling changes out in waves, small enough that the first wave catches the mistake.

A design that contains only one of these still has a blast radius of "everything" for the other two.

The strongest containment is structural. In a cell-based architecture, the whole stack (services, queues, data stores) is replicated into independent cells, and a thin routing layer assigns each customer to one cell. A cell can fail, be compromised or receive a bad deployment, and only its customers notice.

Two rules keep cells honest. The router must stay simple, because anything shared by all cells is the new blast radius. And cells must not call each other on the request path; a cell that depends on its neighbours is not isolated.

Not every system needs full cells. Bulkheads apply the same idea inside one deployment: separate thread pools, connection pools or worker fleets for different tenants or features, so one noisy consumer cannot exhaust capacity for the rest. Shuffle sharding goes further by giving each customer a random combination of a few workers out of many, so the chance that two customers share all their workers becomes very small and a poisonous request takes down only the combination it lands on.

Outages grow through dependencies. A slow database makes requests queue, queued requests hold connections, retries multiply the load, and a single slow component becomes a site-wide failure. Google's SRE guidance describes these cascading failures in detail, and the defences are well known:

• Timeouts on every call, shorter than the caller's own deadline, so slow dependencies fail fast instead of holding resources.
• Retries with backoff, jitter and a budget, so a struggling dependency is not hammered by every client at once.
• Circuit breakers that stop calling a dependency that keeps failing, and probe it before resuming.
• Load shedding that rejects excess work early and cheaply, keeping the system useful for the requests it accepts.
• Static stability: design so that if a dependency fails, components keep doing what they were doing rather than needing that dependency to recover.

An attacker who lands in one workload inherits its identity, network position and data access. Partition all three:

• Separate cloud accounts or projects by environment and, where it matters, by cell or product line. An account boundary is the cheapest strong boundary in the cloud.
• Give each cell or service its own credentials, scoped to its own data. Shared database users turn any single compromise into a full one.
• Encrypt tenant data with per-tenant or per-cell keys, so access to storage alone does not reveal everything.
• Protect against destructive actions: deletion protection, versioned storage, and backups that the production identities cannot delete.

Many of the largest outages were caused by a change, often a configuration change rather than code. Treat every change as a possible fault and limit its blast radius the same way:

1. Deploy to one cell, region or small percentage first.
2. Let it bake long enough to see real traffic and real errors.
3. Compare health against the cells that did not change, and roll back automatically on regression.
4. Widen in waves, never all at once, and apply the same discipline to feature flags and configuration.

Containment improves when it is a stated requirement rather than an aspiration. Write a short budget for your system: no single deployment may affect more than a small, named share of customers before automated checks run; no single workload credential may read more than one cell's data; no shared component may be changed outside a maintenance process. Then test it. Game days that fail a cell, revoke a credential or roll out a deliberately broken configuration show whether the boundaries hold.

For the security side of the same question, how far an attacker can travel along real paths, read our field note on deriving trust boundaries and blast radius from architecture evidence.