Ordering events across 38 regions means agreeing on time, and machines are bad at agreeing on time. Here is how we keep order without pretending clocks are perfect.
Hybrid logical clocks
Every event gets a timestamp that combines wall-clock time with a logical counter. If a message arrives from the future, the receiver moves its clock forward instead of breaking causality.
A 3 ms budget
- Every node measures its offset against three time sources each second.
- A node more than 3 ms off stops accepting writes and drains.
- Reads stay available from replicas that are within budget.
The night Singapore drifted
In May a firmware update made one rack in Singapore drift 40 ms in six minutes. The budget caught it at 3.1 ms, the rack drained, and customers saw a slightly slower write path for eleven minutes instead of out-of-order data.
Sigrún leads the query engine team in Reykjavík. Before Basalt she worked on storage at a Nordic payments company, where she learned to distrust any latency number without a percentile next to it.
