Reliability is a feature, not a phase

Teams rarely decide to build unreliable software. They decide to “make it work first” and “harden it later”. Later arrives as an incident.

Reliability is not a phase at the end of a project. It is a property of the design, and it is cheapest on the first day. Here are five habits we treat as non-negotiable.

1. Name the failure modes before you write the happy path

For every external call, ask three questions: what happens if it is slow, if it fails, and if it succeeds twice? Writing the answers down turns vague risk into concrete design: timeouts, retries with backoff, and a decision about what the user sees when the dependency is down.

2. Make operations idempotent

Networks retry. Users double-click. Workers restart halfway through a job. If running an operation twice can charge a customer twice or create a duplicate record, the bug already exists — it just has not happened yet. Idempotency keys and unique constraints in the database are the boring, reliable fix.

3. Let the database enforce what must never be false

Application code has many paths; a constraint has one. When a rule must hold — an amount is never negative, an email is unique, a state can only move forward — put it in a CHECK, a UNIQUE index or a trigger. The application should still validate, but the database is the last line that cannot be skipped.

4. Observe what you would want to know at 3 a.m.

Logs that nobody reads are not observability. Decide which few signals tell you the system is healthy — error rate, latency, queue depth, the age of the oldest unprocessed job — and make sure a person sees an alert when they go wrong. Then remove the noise, because an alert that fires every day is an alert that gets ignored.

5. Recover by design, not by heroics

When a process crashes, what state does it leave behind? A good system can be restarted at any moment and converge to the right state without guessing. That usually means persisting intent before acting, and on recovery checking reality instead of repeating the action.

The cost argument

Each of these habits costs hours at design time. Each one, skipped, costs days during an incident — plus the trust that is hardest to win back. That is why we say reliability is a feature: it is something users experience, even when they never notice it.