Strange Loop

Distributed Commit Log: Application Techniques for Transaction Processing

Messaging systems built on the idea of a distributed commit log such as Apache Kafka and Amazon Kinesis offer a powerful approach for building robust, real-time systems. However, because of their unique nature there are particular techniques required to use them effectively.

This talk will catalog approaches used in practice to address topics such as: dealing with duplicate event delivery, merging & splitting of shards, poison events, exception handling, detecting failed workers, performing rapid failover, and providing facilities for debugging. These approaches provide a foundation for using distributed commit logs not just for applications such as click-stream analysis but for transaction processing.

David McNeil

David McNeil


David has been a developer for many years. He has worked on R&D project, enterprise application, and database implementations in C, Smalltalk, Java, .Net, etc. with a focus on distributed system design. He is currently a part of the LonoCloud team at ViaSat building cloud infrastructure in Clojure.