Eventador 0.7 is out! It’s our first release after gathering customer feedback and lessons learned from the platform since our 0.5 release. We pushed a number of bug fixes and minor improvements that will make our platform more powerful and easy to use. Additionally, we launched a couple of really important new features: One-Click Scaling and Dashboard.
One-Click Kafka Scaling
One-Click scaling is the first in a series of scaling and cluster automation features we plan. With one-click scaling, you can add brokers to an existing Kafka cluster by simply clicking a button in the Eventador console. You may ask for as many as 12 more brokers in a single click. This feature allows for rapid scaling and sets the groundwork for some powerful features we have planned for the future.
As your workload grows or you need more performance, or both, you can easily scale your cluster to achieve better throughput. From the deployments page, click details->’add Kafka Broker’, then select the number of brokers to add. Click OK. The Eventador platform then initiates the following actions:
- A request is sent to our cloud builder daemon. This controller is in charge of coordinating the creation of your nodes and registering them with the existing cluster and security context.
- The appropriate number and type of AWS resources are added.
- Once running, they are tagged, DNS names are added, SSL configuration completed.
- Kafka is then started with the proper configurations for the existing cluster.
- Zookeeper then automatically registers the new brokers.
- Once ZK registration is confirmed, we acknowledge in console and the account metadata.
- A few seconds later, metrics for these brokers start to show in the dashboard.
This is all done seamlessly and without interruption to your service. This is the first in a series of automation functions we plan to add. It should be noted, at this release, that partitions don’t auto-rebalance when new brokers are added. Freshly created topics will partition across the new brokers, and existing topics will need to be rebalanced by our support team. In the future, we plan to add automatic scaling and rebalancing components to make this even more seamless.
The Eventador Dashboard
Introducing the Eventador Dashboard: A set of graphs and charts that shows you exactly what is going on inside Kafka in graphical format. You can select from various metrics, choose time-windows, select which deployments to show, configure color themes and more. This gives you new insights into the state and performance of your clusters.
The Dashboard is included with every plan level.
In order to build the Dashboard, we had to first build out a statistics monitoring and management framework. This framework underlays every deployment and provides data to the Dashboard, it also makes it simple for us to internally handle day to day management tasks, such as monitoring resource utilization, managing failed servers, and so on. To ensure we have immediate access to rich metrics, we’ve built the statistics framework around a number of open source components, as well as our own software. This includes:
- Java Management Extensions (JMX)
- Jetty Servlet Engine
- Prometheus exporters
- Python Pandas data analysis library
This framework allows us to provide rich data to users via the Eventador Dashboard, but this framework also is the foundation that enables future roadmap innovations based on the data it provides.
OK, Let’s Talk Roadmap
With the most recent batch of feedback and features, we are quickly approaching the release of 1.0. We have a short-list of features that we believe to be super exciting, namely:
- Auto-Scaling. Building on our statistics framework and auto-scale capabilities, we will expose controls to auto-scale the cluster based on simple threshold rules and performance goals.
- Kafka partition and topic level scaling componentry.
- More configuration and transparency for Kafka itself. Ensuring that you can see what’s happening with your cluster and can customize it to your needs.
- More metrics! We want to expose more and more metrics for full transparency.
- Dashboard features. We have a long list of work to make the dashboard even more useful, like resizing options and more chart types.
- More Notebook features. We plan to add more helpers and better (deeper) integrations to help everything from simple experimentation to hard-core data science to be easier.
But there is one thing we are really really excited about adding for 1.0:
- Stream Processing.
Today we have a very basic stream processor built into the Eventador platform, but we have already been working hard on a new release that includes a popular massively scalable stream processing engine. This new stream processing capability will unlock new workflows and use-cases in Machine Learning, AI, Analytics, routing, real-time ETL, intrusion detection, as well as additional persistent storage engine capabilities and integrations.
If you aren’t familiar; stream processing adds the ability to aggregate, mutate, enrich, inspect and take action on a stream in real-time.
We are excited to add this componentry to our stack, but beyond that, we realized we had to make it easy to reliably process unbounded streams of data inside our service, and we have to make it easy to do so. With 1.0 you will be able to, so stay tuned!