Blog

Building Resilient Systems: Lessons from Distributed Architecture

2026-02-20

Modern software doesn't live on a single machine anymore. The moment you split a monolith into services — or even just rely on an external API — you've entered the world of distributed systems. And distributed systems fail. Not if, but when.

This post distills a few years of building, breaking, and rebuilding distributed architectures into actionable principles you can apply today.

Why Resilience Matters

The cost of downtime grows exponentially with scale. A five-minute outage for a hobby project is invisible; a five-minute outage for a payment system can cost millions. Resilience isn't about preventing failures — it's about designing systems that degrade gracefully when the inevitable happens.

"Everything fails, all the time."
— Werner Vogels, CTO of Amazon

This quote isn't pessimism. It's an engineering axiom. Once you accept it, you stop building for the happy path and start building for reality.

Core Principles

1. Circuit Breakers

The circuit breaker pattern is borrowed from electrical engineering. When a downstream service starts failing, instead of hammering it with retries (which makes things worse), you "open the circuit" and fail fast.

class CircuitBreaker {
  private failures = 0
  private lastFailure = 0
  private state: 'closed' | 'open' | 'half-open' = 'closed'

  constructor(
    private threshold: number = 5,
    private resetTimeout: number = 30_000
  ) {}

  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === 'open') {
      if (Date.now() - this.lastFailure > this.resetTimeout) {
        this.state = 'half-open'
      } else {
        throw new Error('Circuit is open — request blocked')
      }
    }

    try {
      const result = await fn()
      this.onSuccess()
      return result
    } catch (err) {
      this.onFailure()
      throw err
    }
  }

  private onSuccess() {
    this.failures = 0
    this.state = 'closed'
  }

  private onFailure() {
    this.failures++
    this.lastFailure = Date.now()
    if (this.failures >= this.threshold) {
      this.state = 'open'
    }
  }
}

Key implementation details:

• Threshold: how many failures before opening the circuit
• Reset timeout: how long to wait before trying again (half-open state)
• Monitoring: always track circuit breaker state changes in your observability layer

2. Bulkheads

Named after the compartments in a ship's hull, bulkheads isolate failure domains. If your payment processing goes down, your search service should keep working.

Practical implementations:

• Thread pool isolation: dedicate separate thread pools per downstream dependency
• Service mesh: use sidecar proxies (like Envoy) to enforce connection limits
• Queue-based decoupling: put an async boundary between critical paths

3. Backpressure

When a system receives more load than it can handle, it has three options:

1. Drop requests (load shedding)
2. Buffer them (queuing with bounded limits)
3. Signal the caller to slow down (backpressure)

Option 3 is the most elegant. TCP does this with flow control. Reactive Streams do it with the request(n) protocol. Your services should do it too.

# Example: rate limiting config for an API gateway
rate_limit:
  requests_per_second: 1000
  burst: 200
  strategy: token_bucket
  on_limit: return_429

4. Idempotency

In distributed systems, messages can be delivered more than once. Network retries, queue redeliveries, and failover mechanisms all contribute to duplicates. Every write operation should be idempotent — applying it twice produces the same result as applying it once.

Common strategies:

• Idempotency keys: the client sends a unique ID with each request; the server deduplicates by tracking seen IDs
• Conditional writes: use IF NOT EXISTS or version checks in your database operations
• Event sourcing: store the sequence of events rather than mutable state

Observability: The Missing Piece

You can't build resilient systems without seeing what's happening inside them. The three pillars:

Structured logging is non-negotiable. Every log line should include:

• Timestamp (ISO 8601)
• Request / correlation ID
• Service name and version
• Severity level
• Contextual fields (user ID, operation, etc.)

{
  "timestamp": "2026-02-20T14:30:00.000Z",
  "level": "error",
  "service": "payment-service",
  "version": "2.4.1",
  "correlationId": "abc-123-def",
  "message": "Charge failed after 3 retries",
  "userId": "usr_789",
  "amount": 4999,
  "currency": "USD",
  "error": "gateway_timeout"
}

Testing for Failure

You wouldn't ship code without unit tests. Don't ship infrastructure without failure tests.

• Chaos engineering: use tools like Chaos Monkey or Litmus to randomly inject failures in staging (and eventually production)
• Game days: scheduled exercises where teams simulate outages and practice incident response
• Load testing: tools like k6 or Gatling help you find breaking points before your users do
• Contract testing: verify that service interfaces haven't broken with tools like Pact

The Human Side

Resilience isn't just technical. It's organizational:

• Runbooks: documented step-by-step procedures for known failure scenarios
• Blameless postmortems: after every incident, focus on what broke and how to prevent it — never who caused it
• On-call rotations: distribute the burden and ensure knowledge sharing
• Error budgets: agree on acceptable failure rates (e.g., 99.9% uptime = ~8.7 hours/year of downtime) and use them to balance velocity vs. stability

Wrapping Up

Building resilient distributed systems is a journey, not a destination. Start with the basics:

1. Add circuit breakers to all external calls
2. Isolate failure domains with bulkheads
3. Make every write operation idempotent
4. Invest in observability from day one
5. Test your failure modes actively

The goal isn't perfection. It's graceful degradation — ensuring that when things break (and they will), your users barely notice.

Got questions or want to share your own resilience patterns? Reach out — I'd love to hear what's worked for your team.