Failure is normal in distributed software. Reliability comes from making failure observable, bounded, and recoverable.
Assume partial failure
A distributed system rarely fails as one unit. One dependency can be slow while another is healthy; a host can be reachable while an application is overloaded; a message can be duplicated while another is lost. Reliable designs start by assuming these partial failures are ordinary states rather than rare exceptions.
Use deadlines deliberately
Timeouts define how long a caller is willing to wait before taking another action. They should be chosen from the user-visible operation and dependency behavior, not copied from a generic template. A deadline that is too short creates false failures; one that is too long turns overload into a queue of stuck work.
Backpressure is a control surface
When producers can create work faster than consumers can process it, queues grow. The system needs a policy: reject excess work, shed optional work, slow producers, or increase capacity. Without backpressure, a transient spike can become resource exhaustion across the whole dependency graph.
Make retries scarce and informed
Retries can recover from transient faults, but uncontrolled retries amplify load during incidents. Use bounded attempts, exponential backoff with jitter, and idempotency where practical. A retry policy belongs to the operation semantics; it is not merely a transport setting.
Recovery is part of the design
Measure the time and steps required to restore service. Run failure exercises against the exact operational paths people will use during an incident. A recovery plan that depends on a person remembering an undocumented sequence is a hidden single point of failure.
Key takeaways
- Design around partial failure, not perfect networks.
- Treat deadlines and retries as application-level semantics.
- Use backpressure before queues become outages.
- Practice recovery paths before the incident arrives.