7 Resilience and Reliability Patterns

This chapter presents patterns that combat network unreliability and service instability, both of which are inevitable in distributed systems. As such, it is the largest chapter in this work.

1. The Bulkheads pattern in § 7.1 limits the impact of a failing service on the rest of the system

2. The Queue-Based Load Levelling pattern in § 7.2 evens out the load between services

3. The Retry pattern in § 7.3 guards against transient failures

4. The Health Monitoring pattern in § 7.4 ensures the system is running as expected

5. The Rate Limiting pattern in § 7.5 protects services from misuse

6. The Leader and Followers pattern in § 7.6 coordinates services in a cluster without a natural leader

Other patterns that could be considered part of this category include [92]:

• Message brokers can improve communication reliability
• Circuit Breaker improves overall service reliability during failures of its dependencies
• Ambassador and Offload to Gateway present opportunities to implement these patterns
• Competing Consumers & Load Balancer provide natural backups for failed services
• Claim Check can help recover message contents in the event of a disaster and Partitioning or Identity Provider can limit the amount of information lost
• Chapter 10 discusses patterns to ensure consistency during failures
• Backend for Frontend (BFF) can be used to personalize reliability for the different requirements of clients
• Gateway Aggregation can improve reliability when a client is slow or on an unreliable network
• Strangler Fig improves reliability during transitional periods

Notable omissions in this category due to the methodology described in § 1 include:

• Cache-Aside and Throttling by Microsoft [93, 94], although common techniques, do not add anything new to the discussion in the context of distributed systems and are not recognised as patterns by other authors

• Compensating Transaction by Microsoft [95] is discussed as part of Sagas