Pinot: Open Source Real-time Analytics Infrastructure

Pinot is a real-time analytics infrastructure that was recently open sourced by LinkedIn. It includes the XLNT A/B testing platform.

Pinot is a realtime distributed OLAP datastore used to deliver scalable real time analytics with low latency. It can ingest data from offline data sources (such as Hadoop and flat files) as well as online sources (such as Kafka). Pinot is designed to scale horizontally.

Pinot is well suited for analytical use cases on immutable append-only data that require low latency between an event being ingested and it being available to be queried.

Key Features

  • A column-oriented database with various compression schemes such as Run Length, Fixed Bit Length
  • Pluggable indexing technologies - Sorted Index, Bitmap Index, Inverted Index
  • Ability to optimize query/execution plan based on query and segment metadata .
  • Near real time ingestion from Kafka and batch ingestion from Hadoop
  • SQL like language that supports selection, aggregation, filtering, group by, order by, distinct queries on fact data.
  • Support for multivalued fields
  • Horizontally scalable and fault tolerant