Picnic Technologies

Project name:
Apache Calcite as a real-time customer data platform

Description project:
Our project provides a high performance query layer powered by Apache Calcite. It exposes our company’s disparate microservices and data stores as virtual tables in a unified data mesh. This allows for using standard SQL to query data across multiple services in real-time through a single entry-point, as if interacting with one large, cohesive database.

What specific problem does the project solve, and how does it go beyond existing solutions in the same domain?:
Picnic’s vision is to deliver highly personalized, data-driven features, which requires real-time aggregation of customer data points from diverse sources. Typically, this would require continuous backend development effort to create and integrate APIs, slowing down business iteration. Existing solutions often fall short: they are unwieldy for operators, or don’t offer real-time performance.

What makes this project innovative:
Our entire landscape of microservices is now queryable as a single, high-performance data source serving live customer traffic. We’ve extended Calcite to make it easier to introduce new data sources, support faster query execution through indexing, and guide query development with features like a query plan visualizer, query benchmarking tool, and execution statistics-based self optimization.

Used Technologies & Platforms:Apache Calcite with extensions built on top, such as improved index and JOIN support, instrumentation for continuous analysis, a query performance analyzer, cross-pod query plan caching via MongoDB, query results caching within sessions, real-time metrics tracking via Kafka and ClickHouse, and many other extensions.

What is the current phase of the project? Production (maintenance), Implementation or Inception?:
The first version of the project was deployed to production two years ago, exposing only a minimal schema with a very basic set of features. Now the project has grown to serve peak traffic of over 50k queries per second, returning millions of rows across hundreds of tables. Our Calcite platform is under continuous development, where we regularly onboard new services and add new features.

What results are/have been achieved by implementing this project: We are now able to quickly ideate and build new behaviors in our online supermarket. This allows us to deliver innovative new features such as the weekly meal planner and user-defined recipes, through real-time targeting of our customers as they are shopping. Calcite is now a core part of Picnic’s business as we seek to provide an even more personalized shopping experience.