Strange Loop

How to Have your Causality and Wall Clocks, Too

Does it bother you when all your microservice logs show up out of order in your log analysis tool because your system clocks aren't synchronized? Did you ever wonder what time it was when the dotted version vector on a piece of data became 2.4.5? If so, then this is the talk for you!

Time is a notoriously difficult concept in distributed systems. Perfect clock synchronization is impossible, and most logical timestamp schemes (like Lamport clocks) don't bear a relationship to "wall clock time" that is meaningful to human operators.

In this talk, we'll learn about distributed monotonic clocks (DMC) whose timestamps can reflect causality but which have a component that stays close to wall clock time. This scheme builds on previous hybrid logical clock proposals by adding important operational hooks and by building in mechanisms to prevent a single runaway system clock from dragging a whole cluster's logical clocks forward into the future.

DMCs can be implemented as a "piggyback protocol" suitable for transport in existing application messages--for example, as additional HTTP headers. I'll describe a novel coordination protocol in this unusual piggyback style--a style that may also be useful in ad-hoc clusters formed by Internet-attached "things".

Jon Moore

Jon Moore


Jon Moore is a Senior Fellow at Comcast Cable, where he leads the Core Application Platforms group that focuses on building scalable, performant, robust software components for the company's varied software product development groups. His current interests include distributed systems, hypermedia APIs, and fault tolerance. Jon received his Ph.D. in Computer and Information Science from the University of Pennsylvania and currently resides in West Philadelphia.