Skip to content

Latest commit

 

History

History
110 lines (64 loc) · 6.24 KB

Durability.md

File metadata and controls

110 lines (64 loc) · 6.24 KB

D — Durability

Does data stay safe forever?

Durability ensures that once data is saved, it won't be lost — even during failures.

To achieve high durability (often targeting "eleven nines" or $99.999999999%$), we use four main techniques:

Strategy Description
Replication Store copies across multiple servers
Backups Regular snapshots of data
Write-Ahead Logging Log changes before applying them
Geo-Redundancy Store data across different regions

In a system design context, Durability is the guarantee that once data is successfully written to the system, it will not be lost—even in the event of a total power failure, a server crash, or a literal natural disaster at a data center.

Note: While Availability is about seeing the data, Durability is about not losing the data.

Replication (The "Copies" Strategy)

Data is never kept in just one place.

Synchronous Replication: The system doesn't tell the user "Success" until the data is written to at least two different physical disks (usually in different Availability Zones).

Asynchronous Replication: Data is written to the primary disk first, then copied to others a few milliseconds later. (Faster, but carries a tiny risk of data loss if the primary fails before the copy finishes).

Backups

Adding Backups is the final, crucial piece of the Durability puzzle. While replication protects you against a hardware failure (a server dying), backups protect you against human failure or software bugs (someone accidentally running DROP TABLE or a bug corrupting data).

To ensure true durability, follow this industry standard:

3 copies of data: (1 Primary + 2 Backups).

2 different media: (e.g., Database snapshots + S3 Object storage).

1 off-site: (At least one copy must be in a different physical region, like moving a copy from AWS us-east-1 to us-west-2).

"Snapshotting" Microservices

For stateful microservices (like your Project X or HR Chatbot), we back up the entire environment:

EBS Snapshots: Taking a "photo" of the hard drive attached to your server.

Configuration as Code: Backing up your Kubernetes manifests and environment variables. Data is useless if you don't have the "instructions" on how to run the app that reads it.

Write-Ahead Logging (WAL)

Before a database updates a record, it first writes the change to a "Log" on a persistent disk. If the server crashes mid-update, the system can simply "replay" the log upon restart to recover the data.

Java Context: This is how Spring Data JPA/Hibernate interacts with ACID-compliant databases (like PostgreSQL) to ensure transactional integrity.

Redundany

At the hardware level, data is striped across multiple disks.Amazon S3: When you upload a file, AWS automatically replicates it across a minimum of three physically separated facilities. This is why S3 can offer $99.999999999%$ durability.


The Trade-off: Durability vs. Latency

There is a direct cost to durability: SPEED.

If you require your Java application to wait for three different data centers to confirm a write before moving on, your Latency will increase significantly.

Durability Level Risk Latency Use Case
Low High (In-memory only) Extreme Fast (<1ms) Real-time gaming scores, session cache.
Medium Low (Single Disk + Async) Fast (10-30ms) Social media posts, logs.
High Near Zero (Multi-AZ Sync) Slower (50ms+) Bank transfers, Medical records.

Interview Question: "How do you ensure zero data loss in case of a server crash?"

To ensure zero data loss, I rely on Write-Ahead Logging for local crash recovery and

Synchronous Replication across Availability Zones to protect against hardware failure.

In my Spring Boot services, I use Idempotent APIs to safely handle retries and

the Transactional Outbox pattern to ensure that state changes and downstream events are coupled atomically.

This ensures that once a transaction is acknowledged, it is permanent and recoverable.


For a "Zero Data Loss" guarantee, you need to prove that the data is "hardened" to persistent storage before the user receives a success message.

Atomic Commit & Write-Ahead Logging (WAL)

The first line of defense is ensuring the database itself is crash-resilient.

The Mechanism: When a Spring Boot application sends a "Save" command, the database (like PostgreSQL or MySQL) doesn't immediately update the data file. Instead, it writes the change to a Write-Ahead Log (WAL) on the disk.

The Recovery: If the server loses power a millisecond later, the data might not be in the main table, but it is in the log. Upon reboot, the database "replays" the log to restore the state.

Synchronous Replication (The "Witness" Strategy)

A single server's disk can fail. To ensure zero loss, the data must exist in two places at once.

Synchronous Commits: In your database configuration, you set the "Synchronous Commit" to on. The primary database will wait for at least one "Replica" (in a different Availability Zone) to confirm it has received and written the data before telling the Spring Boot app "Success."

Trade-off: This adds a few milliseconds of Latency, but it is the only way to guarantee that if Server A explodes, Server B has the exact same data.

Application-Level Retries & Idempotency

Sometimes the server crashes after saving the data but before sending the confirmation back to your Java app.

The Risk: Your app thinks the write failed, but it actually succeeded.

The Fix: Implement Idempotency Keys. If your Spring Boot service retries the request, the database recognizes the key and says, "I already have this," instead of creating a duplicate. This ensures data integrity during the recovery process.

Distributed Transaction Management (The Saga Pattern)

In a microservices environment (like your HR Chatbot architecture), data might be spread across multiple services.

Transaction Logs: Use a pattern like Transactional Outbox. You save the data and the "message to be sent" in the same atomic database transaction. If the server crashes, the message hasn't been sent yet, but because it's in the DB, a background process will find it and send it once the server is back up.