Apache Flink® On Eventador.Io

September 11, 2017 in Managing Apache Flink

Apache Flink® On Eventador.Io

With the addition of Apache Flink – Eventador.io has a true end-to-end enterprise-grade stream processing platform. We run the complex infrastructure and provide support, you can focus on your streaming code.

In July we launched a public beta for Apache Flink. Since then we have been making improvements and changes to ensure our platform has a killer implementation of Flink.

Just to recap – Apache Flink is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. It’s a natural fit with Apache Kafka. Flink can be used for a myriad of use cases from AI/ML to manufacturing, transportation, IoT, and much more. Flink has a thriving community-driven by Data Artisans.

Our mission was to bring the powerful Flink framework to our existing Kafka platform. If there was ever a match made in heaven, it’s Kafka + Flink. We had four initial design goals we wanted to achieve. As a team, we have a lot of experience building as-a-service platforms, thus, we have some pretty serious opinions about how services on our platform should be built.

A proper as-a-service offering will magnify the usefulness of a given software by merging ease-of-use, limitless and near instant cloud infrastructure, and fantastic support in a seamless and highly observable platform

Put simply, we wanted to:

  1. Ensure Flink users have a simple and powerful fully managed stream processing experience
  2. To ease application development and deployment headaches, yet integrate into existing SDLC workflows
  3. Take advantage of easy deployment and management in a modern cloud environment
  4. Provide the same awesome support our Kafka customers are accustomed to

Let’s break these down a bit:

Fully managed Flink as a Service

At its core, Eventador.io is a streaming data platform. It’s designed to be fully managed, and run in the cloud. If you look at companies like Netflix and Uber you will see they have similar internal services. The Eventador platform allows anyone to have a streaming data platform with just a few clicks.

We integrated Flink with our existing console application, including our team controls and the overall security context. When you choose to create a deployment it works just as a Kafka deployment would. A single-tenant Flink cluster with job manager and task managers – all created in just a few seconds. Both a Flink and Kafka cluster are ready for you to start building amazing streaming applications.

To be clear, our Flink (and Kafka) deployments are dedicated instances on AWS infrastructure. No multi-tenancy or over-provisioning. We have plans starting with a simple developer plan, to a production -eady plan, all the way up to our enterprise offering (which deploys into your AWS account). All fully managed and fully supported. Reach out to us and we can help you select the plan that will fit your use case.

Application Development and SDLC workflows

Eventador projects are a brand new component of our service. Projects allow developers to easily integrate existing SDLC workflows into a Flink project via Github. Eventador handles all the complexity of the build process (typically Maven) and the deploy process. This includes CI/CD components, Git Flow, or any other development lifecycle workflow you may have. When the code is merged (or via selection) the it gets deployed automatically and seamlessly to your Flink cluster. This makes development, test, and production management of jobs a snap. It also wraps the entire Github ecosystem around Flink. If you have a specific integration process, specific branching strategies, or whatever, all those workflows apply directly to your Flink code, and Flink deployment on Eventador.

Templates make building projects easy. Are you just getting started with Flink? Or perhaps you just need to get a simple filter deployed ASAP. Simply select a base template (or none) when you create your project. These templates serve to teach as well as reduce time to market. Templates are simply Github repositories – Eventador imports them when you create a project. A template includes all the components to build an entire Flink job – typically in Java. It’s a complete project with all the scaffolding and files required. You focus on the core logic, we’ll cover the rest.

Deployment and Management in the cloud

The Eventador Console has an integrated view of the Flink cluster, showing jobs, the execution plan, the state and more. This gives you full control over running jobs, and the overall topology of the cluster. The Flink UI is built into the Eventador Console – so it has the same team level access controls as the rest of the platform.

Killer Support

If we want to be known for anything, it’s for having amazing support. It’s a passion of ours. If you are just starting out with Flink or have a large production job, we are ready to help. Each of our plans comes with some level of support. We don’t care if it’s helping debug some Java code or making best use of your resources or even capacity planning, we are here for you.

Sign up and give it a whirl

Flink on Evenador.io is available now. You can register here and deploy your own Flink cluster in minutes. If you have questions please reach out and we will be happy to answer your questions or schedule a demo of the platform.

Leave a Reply

Your email address will not be published. Required fields are marked *