Strange Loop

Sept 30 - Oct 2, 2021

/

Union Station

/

St. Louis, MO

Unifying the Search Engine and NoSQL DBMS with a Universal Index

Unifying the Search Engine and NoSQL DBMS with a Universal Index In contrast to single-function architectures, MarkLogic Server takes an unusual approach to collapsing the usual hierarchies of types of servers that make up a complete application, combining Search, a NoSQL DBMS, and an application server in a single kernel. The computational foundation for this hybrid is the Universal Index.


 In this talk, we'll begin with the familiar text indexing data structures and algorithms that underlie search engine technologies. We'll extend that index to cover document structure and semantics, add scalar range indexing in one and two dimensions (including geospatial application), and then incorporate "reverse" indexing of queries. We will demonstrate a novel type of "matchmaking" query whose evaluation is based on a composition of forward and reverse index evaluation, in a true "strange loop" path through abstraction levels. Finally, we'll explore the means by which all of this indexing may efficiently run concurrently with querying, using MultiVersion Concurrency Control and Log-Structured Merge Trees, providing ACID transactions together with lock-free query evaluation, built-in sharding, terabyte-per-server scale-out, replication, and query distribution.


 We will conclude with examples of production applications built on this architecture for geospatial service discovery at warriorgateway.org, social networking at bx.businessweek.com, and knowledge management on a US Air Force application.

Chris Blow

Chris Blow