July 21, 2016 in Product News


Over the past two decades, I have witnessed many massive shifts in the data technologies landscape. I have seen a maturity of the RDBMS market, the adoption of NoSQL technologies, and organizations of every size struggling with data management in one way or another. Throughout all of this change, one area is starting to become more and more intrinsic to the data backbone of many companies: real-time data. And for good reason, the current pattern of writing data to some flavor of database or file sink and then processing that data using either a b-tree index path or parallel processing frameworks, or both isn’t able to meet real-time needs. Some other mechanism must be engineered.

We founded to solve an increasingly pervasive problem; data needs to be more real-time.

Data in the here and now

The demand comes from an ever-evolving ecosystem that has figured out data matters. There is lots of it, and customers demand it to be contextually relevant, secured, and real-time. Companies have figured out that by using data they are more competitive and compelling. Real-time information is the cherry on top of the data cupcake. The demand is growing, and it’s growing rapidly.

Yet, there aren’t many options for these companies to be successful with low barriers to entry, high performance, default security, and a fast time to market. So we built the platform we dreamed of, and we called it

Let’s be clear here, real-time computing and real-time data processing aren’t new technologies. Real-time data processing works by processing small bits of data as they flow into the system, as they happen. Data is not stored then retrieved to answer questions, rather, it’s immediately mutated, aggregated, and routed to requesting applications through data pipelines. It’s a completely different paradigm. The difference today is the compute ecosystem has changed around these technologies enabling them to be more powerful and lower cost. The advent and adoption of the cloud as well as messaging systems like Apache Kafka mark a distinct apex in this area of computer science.

Real-time data pipelines using Apache Kafka

The platform uses Apache Kafka as its backbone. Kafka is a real-time messaging system written in Java. Kafka was invented at LinkedIn to solve real-time data problems, was open-sourced, and since has exploded in adoption. It’s now an Apache project and at the core of

Kafka excels at processing real-time events. It’s a messaging system, but it also has properties of a log structured database. Kafka is scalable, fault-tolerant, and durable. Because of this, it’s a perfect foundation for processing real-time data.

Amazing as Kafka is, it’s not in itself a complete real-time data solution. This is where steps in and extends Kafka to include important components like tight security, cloud-based deployment and scaling, a SQL query interface, a schema manager, and simply consume and produce endpoints. These components unlock the potential of Kafka and transform it into a true fast-data platform.

The beta platform is engineered to make deploying Kafka data pipelines really simple and useful right out of the box. We used our decades of experience building massive data systems at places like eBay/Paypal and ObjectRocket/Rackspace to build a completely unique and powerful platform. It’s the system that we would want to use.

We think that will be useful in a number of verticals including wearables, sensor data, supply chain/manufacturing, gaming, social, and real-time analytics and data sciences just to name a few. To that end, for our beta we focused on a few key foundational areas:

  • A high performance, highly redundant Apache Kafka implementation
  • SSL and an ACL based security model
  • Easy deployment to a dedicated VPC on AWS
  • REST-based consume/produce interfaces
  • Apache Avro based schema manager
  • Aggregations, analytics, and processing in PostgreSQL/PipelineDB
  • A web-based management and metrics console


We are having a blast building, and love making customers successful. Starting today is in private beta. We enjoy the opportunity to talk to customers and understand each unique use case and help build to solve these real-world problems. I am so proud of what the team has built so far, and excited to evolve it, so sign up now and see what I am talking about!

Leave a Reply

Your email address will not be published. Required fields are marked *