Graph: composable production systems in Clojure
Prismatic’s newsfeed generation service has dozens of components including storage systems, in-memory caches, pub/sub messaging, indices, HTTP handlers, and dynamic relevance ranking algorithms, all of which work together in an intricate dance to serve personalized feeds. Designing an architecture of this complexity is itself a nontrivial task; and to run it in production, we also need to be able to monitor each component for performance and failures, reuse sub-components in other services, and test our services by mocking out components.
To help tame this complexity, we’ve developed Graph: a simple, declarative approach for specifying system compositions. A Graph specification is just a Clojure map from node names to functions with named arguments, each of which can refer to the output of a previous step or an input to the graph. This approach is very general; in addition to compiling Graphs down to production services (where nodes build components), we also use Graphs for real-time newsfeed generation and streaming web document analysis. And because Graphs are just Clojure maps, it is trivial to reason about them, mock out components (assoc), monitor individual nodes (map observer functions over leaves), and compose them with a variety of execution strategies to intelligently schedule real-time computation.
Jason Wolfe builds large systems, machine learning, and ranking algorithms for news personalization at Prismatic. Jason recently completed his PhD in Computer Science at UC Berkeley, specializing in Artificial Intelligence.