We built Eventador to solve a pervasive and tricky problem; It’s exceptionally challenging to build real-time data systems using existing technology. To address this problem, in July, we launched our real-time data pipeline service based on Apache Kafka. Since launch, we have been working feverishly on the platform. Iterating, adding features, and core functions.
Today I am excited to announce the Eventador Beta 0.2.0 release.
There are some big changes and massive improvements in the system that we think will make building streaming and real-time data projects easy. This release is comprised of features derived from early customer feedback, as well as some key core improvements:
- Native Kafka driver support
- Security enhancements
- Eventador Stacks
- User interface
- Performance metrics collection and reporting
Native Kafka driver access
The initial release of Eventador was built on the premise of using a REST interface to interact with the system. While good, we got feedback that native driver support was highly desirable. It was clear that access needed to be simple, ubiquitous, and asynchronous. Producing and consuming data via the native drivers provides for these requirements.
Beta 0.2.0 introduces native Kafka driver access. Kafka clusters are automatically deployed in the cloud and the broker list is directly exposed. You can use any native Kafka driver (supporting 0.9.0 or greater) in the ever-growing ecosystem to produce and consume directly from Kafka. Management of the cluster is done via the native drivers, as well as the Eventador UI. The new UI provides useful helper components such as topic, security, and cluster management.
Moving to this new connection method required changes to the Eventador cluster manager, deployment components, metadata, just about everything. We also introduced our own statistics capture component that keeps metadata about Kafka performance and powers graphs in the UI.
Check out our examples to see what’s new.
When we decided to open up native driver access to Kafka brokers, one of our biggest concerns was security. To that end, we implemented a couple of key security features to the service. The service requires white-listing source IP’s using ALLOW rules in CIDR format. The UI has a simple interface to help users add a white list, as well as a wizard (more on this later) that leads users through a step-by-step process. Allow rules are implemented as AWS VPC ACL’s for each of your dedicated VPCs. The ACL’s are provisioned for each deployed Kafka cluster, and you can have unique and independently secured clusters.
Additionally, Eventador has full SSL driver support. When a cluster is deployed, certificates and keys are generated and then provided to you to be used on the client-side when connecting. Again, keys are unique per deployed cluster, and you may have multiple deployed clusters each with unique SSL credentials.
Eventador Stacks are our newest feature; They provide the ability to process and mutate streams in-flight. Data projects are often a myriad of various components, use cases, and technical requirements. There needed to be a way to plug-in processing components to meet these requirements – and Eventador Stacks solves this problem. Stacks are built around a deployed cluster of Docker containers that allow for a highly scalable and robust processing framework. Beta 0.2.0 introduces our first stack – a PipelineDB instance for aggregation and database access. This stack is automatically created when a cluster is deployed, and there is literally no work on the customer side to provision the stack. Once provisioned, you can connect to PipelineDB (via any standard PostgeSQL-compatible client) and create continuous views on the stream. This makes creating reports, powering other applications, or moving data a snap. Future releases will expand upon our default stack and allow for even more functionality.
The user interface has been updated to allow for management of the new 0.2.0 components as well as display metadata about the system. There are 3 main components of the Eventador system:
- Deployments: the group of logical compute resources assigned to a cluster. This is your Kafka cluster with it’s associated components.
- Topics: A Kafka topic. This is the logical stream or pipeline of data.
- Stacks: The Eventador data processing framework. A stack operates on Topics.
The UI now allows for management of all these components as well as new statistics pane showing various graphs on the performance of your cluster.
Additionally, there is a new wizard for creating a data pipeline. When a customer first signs in, the wizard leads them through creating their first deployment, topic, and stack step by step.
We also made a ton of other improvements and fixes:
- A robust statistics collector service.
- AWS deployment and provisioning fixes
- Kafka server default configuration changes (performance/robustness)
- Kafka 0.10.0 update
- PipelineDB 0.9.5 update
- Bug fixes
- Docs: Getting Started Guide and Examples.
- Increased transparency for Kafka, exposing more details about the cluster.
We are rapidly iterating on the platform, adding great new features and functionality every day. As you use the product, please don’t be shy about sharing your feedback. We are having a great time building Eventador and we are just getting started!