Between this and the google spanner improvements I wonder whether we are living through a sea change in distributed system design, where mindsets like There is No Now become less important. For a long time now (and this is certainly how I practice & was trained) there has existed an aversion to relying on timestamps for anything in distributed systems other than timeout & retry loops. The thinking is you should design your distributed system so it would still function if one of the nodes were operating next to a black hole. While this certainly leads to robust systems, it is very difficult to come up with designs that work well within this constraint. I have however seen my share of clock skew/drift issues in the real world while working on-call shifts in Azure, so I readily believed this type of thinking was the way to do it. TLA+ made it easy to reason about such systems anyway.
But if clock skew & synchronization really can be solved at the hardware level, then perhaps I find myself in the position of experts in classical error correction schemes when transistors became reliable enough to make the field much less important within CPU design (of course error correction codes still enjoy vibrant life in many other domains). Distributed systems can start to rely on a global total ordering of events with timestamps. Concurrency will always be hard for one reason or another, but maybe this is a dragon of complexity that has been slain.
You can maybe get microsecond-level accuracy on a Raspberry Pi with GPS receiver, but NTP lacks the hardware support from the network that is needed for clients to sync that accurately. Consider that a 1500 byte packet takes 12 μs to transmit over gigabit ethernet, so on a busy network you can expect at least that level of jitter from contention between NTP and other traffic — and that’s without accounting for forwarding delays in switches and routers.
Amazon has deployed PTP, which synchronizes the clocks of every element in the network. That means every synchronization hop is over a point-to-point link (eg, server to switch) so it is not affected by forwarding delays. And PTP uses hardware-assisted packet timestamps, so it isn’t affected by delays between the time daemon and the ethernet card.
Between this and the google spanner improvements I wonder whether we are living through a sea change in distributed system design, where mindsets like There is No Now become less important. For a long time now (and this is certainly how I practice & was trained) there has existed an aversion to relying on timestamps for anything in distributed systems other than timeout & retry loops. The thinking is you should design your distributed system so it would still function if one of the nodes were operating next to a black hole. While this certainly leads to robust systems, it is very difficult to come up with designs that work well within this constraint. I have however seen my share of clock skew/drift issues in the real world while working on-call shifts in Azure, so I readily believed this type of thinking was the way to do it. TLA+ made it easy to reason about such systems anyway.
But if clock skew & synchronization really can be solved at the hardware level, then perhaps I find myself in the position of experts in classical error correction schemes when transistors became reliable enough to make the field much less important within CPU design (of course error correction codes still enjoy vibrant life in many other domains). Distributed systems can start to rely on a global total ordering of events with timestamps. Concurrency will always be hard for one reason or another, but maybe this is a dragon of complexity that has been slain.
Barbara Liskov has a 1991 (!) paper Practical uses of synchronized clocks in distributed systems
Murat Demirbas also covers it on his blog.
This is kind of funny. I was able to set up a network time server for $100 that does microsecond accuracy. https://grav.munn.me/my-blog/how-to-setup-a-stratum-1-secure-ntp-server-ntps
You can maybe get microsecond-level accuracy on a Raspberry Pi with GPS receiver, but NTP lacks the hardware support from the network that is needed for clients to sync that accurately. Consider that a 1500 byte packet takes 12 μs to transmit over gigabit ethernet, so on a busy network you can expect at least that level of jitter from contention between NTP and other traffic — and that’s without accounting for forwarding delays in switches and routers.
Amazon has deployed PTP, which synchronizes the clocks of every element in the network. That means every synchronization hop is over a point-to-point link (eg, server to switch) so it is not affected by forwarding delays. And PTP uses hardware-assisted packet timestamps, so it isn’t affected by delays between the time daemon and the ethernet card.