🧠 Analysis: Etcd Consensus in a Hybrid Optane/SATA Cluster

Architecture:

• Node 1 (CP1): Intel Optane (Ultra-Low Latency ~10µs)

• Node 2 (CP2): Intel Optane (Ultra-Low Latency ~10µs)

• Node 3 (CP3): SATA SSD rpool (Variable Latency 10ms - 500ms+)

1. The Physics of Consensus (Speed)

Etcd uses the Raft Consensus Algorithm. For a cluster of size $N=3$, the Quorum (votes required to commit data) is calculated as:

\(\text{Quorum} = \lfloor \frac{3}{2} \rfloor + 1 = 2\)

This means the Leader only needs acknowledgment from itself + 1 peer to consider a transaction "Saved."

The "Fastest Majority" Rule

When a write request arrives (e.g., creating a Pod), the speed of the cluster is determined by the two fastest nodes. The slowest node is mathematically irrelevant for write latency.

• Leader: CP1 (Optane)

• Flow:

  1. CP1 writes to its own Optane disk (0.01ms).

  2. CP1 broadcasts the entry to CP2 (Optane) and CP3 (SATA).

  3. CP2 writes to Optane and replies "Done" (0.01ms).

  4. QUORUM REACHED (2/2).

  5. CP1 commits the write and responds to the client (Kubernetes API).

• Result: The cluster operates at Optane Speed.

• Note: CP3 receives the commit message milliseconds later, after it finally finishes writing to its SATA disk. This "catch-up" happens asynchronously in the background.

2. Failure Scenarios

Scenario A: The "Slow" Follower (SATA Lags Behind)

Condition: The SATA drive is under load (e.g., database backups), causing write latency to spike to 200ms.

• Impact: None on cluster performance.

• Mechanism: The Leader (CP1) sends heartbeats and log entries to CP3. CP3 queues them and applies them as fast as its disk allows. Since CP1 and CP2 have already committed the data, CP3 is effectively "eventually consistent" (usually lagging by only a few milliseconds).

• Monitoring: You may see etcdserver: slow fdatasync warnings in CP3's logs, but as long as it sends heartbeats, the cluster remains healthy.

Scenario B: The "Dead" Follower (SATA Times Out)

Condition: The SATA drive saturates completely (latency > 15s), causing heartbeat timeouts.

• Impact: Minor degradation (Loss of Redundancy).

• Behavior:

  1. CP3 stops responding to heartbeats.

  2. The Leader marks CP3 as unreachable.

  3. Cluster size remains 3, but active members = 2.

  4. Availability: The cluster remains Online. Quorum is still met by CP1 + CP2.

  5. Risk: If one more node fails (an Optane node), the cluster goes down.

Scenario C: Single Optane Failure (Maintenance)

Condition: CP1 (Optane) dies or is rebooted.

• Impact: Performance Drop.

• Behavior:

  1. CP2 (Optane) and CP3 (SATA) are the survivors.

  2. They form a Quorum (2/3). The cluster stays Online.

  3. New Performance Profile: To commit a write, CP2 (Leader) needs a vote from CP3 (SATA).

  4. Latency: The cluster slows down to match the speed of the SATA drive (~10-100ms per write).

  5. Recovery: Once CP1 comes back, it catches up, and performance restores to Optane speeds.

Scenario D: Double Optane Failure (Catastrophe)

Condition: The PCIe adapter holding both Optane drives burns out. CP1 and CP2 vanish instantly.

• Impact: Total Cluster Outage (Read-Only).

• Status:

  • Active Members: 1 (CP3).

  • Votes Needed: 2.

  • Result: No Consensus. The API Server goes into Read-Only mode. No pods can be scheduled.

• Data Integrity: Safe. CP3 (SATA) contains a full copy of the cluster state (up to the last replicated millisecond).

• Recovery Action:

  • You cannot "restart" the cluster normally.

  • You must perform a Disaster Recovery (Force New Cluster) using the data on CP3.

  • Command: talosctl etcd snapshot (from CP3) -> talosctl etcd restore (to CP3 as a new single-node cluster).

3. Special Edge Cases

The "PreVote" Safety Mechanism

Question: If CP3 is slow, will it think the Leader is dead and start annoying elections?

Answer: No. Modern Etcd (v3.4+) uses a PreVote phase. Before CP3 declares "I want to be Leader," it asks CP1 and CP2: "Would you vote for me?"

• CP1 and CP2, being healthy and connected to each other, will reply: "No, we have a valid leader."

• CP3 cancels its election bid.

• Benefit: A slow/flaky SATA node cannot disrupt the stability of the healthy Optane core.

The "Split Brain" Protection

Question: What if the network partitions CP1 vs CP2+CP3?

Answer: The side with the majority wins.

• CP1 (Isolated): Cannot reach majority (1/3). Steps down. Becomes Read-Only.

• CP2+CP3 (Connected): Have majority (2/3). They elect a new leader and continue operations (at SATA speed, due to CP3's involvement).

4. Summary Table