Strange Loop

Scaling Validation and Quality of Streaming Data Products at Twitter

Twitter data is invaluable for social, behavioral, and marketing research. The Data Products Quality Engineering team builds and maintains services, tools, and process to help feature teams provide a strong distributed streaming architecture to meet customers' social data needs.

This talk is about how, as the Twitter data business has grown, we have scaled up from simple QA for streaming systems to building tools and processes for identifying issues both pre-production and for live microservices. I'll cover some of the tools and processes that my team maintain and use now, and how they bring value to the other engineers that depend on us as well as Twitter's data customers. Some of our specialties include end-to-end stress testing, measuring pipeline data loss with respect to meeting product SLAs, and per-message data quality.

Kelly Kaoudis

Kelly Kaoudis


Kelly Kaoudis is a software engineer at Twitter. She also likes yoga, biking, reverse engineering x86 binaries, and hash functions.