Getting Started with Kafka + Flink

Getting Started with Kafka + Flink


Eventador offers several plans ranging from development/testing plans to full production grade enterprise configurations. This guide will show you how to create a Kafka + Flink developer deployment, which is free for 30 days. If you’d like to deploy the Kafka only plan use this guide.

Background

Eventador is a high performance real-time data pipeline platform based on Apache Kafka and Apache Flink. Eventador makes it easy to perform analysis and build applications using real time streaming data. Some areas that can benefit from real time data are sensor networks, IoT, click-stream analysis, fraud detection, or anything that requires real-time data. Eventador is deployed to Amazon AWS, and delivered as a service.

A developer deployment consists of: – A Kafka cluster including Zookeeper nodes (the backbone of the service) – A Flink Job/Task manager with up to 4 job slots – Ability to create up to 5 topics – Access to the Eventador console to monitor and configure your deployment. – For more details visit our plans page.

Each account has an isolated VPC that your deployments live in. You must grant access to each deployment via the console in order for any traffic to be allowed through. More on this below.

Prerequisites

  • Create an account or login via GitHub. Note: to deploy Flink projects you must link to your GitHub account.
  • If you already have an account you can link with GitHub by visiting your profile and clicking the “Link this Account to GitHub” button.
  • Enter credit card info on the accounts page (required for production account and developer plans after trial period)

Create a Deployment

  • After creating an account, click the Deployments tab from the Eventador Console.
  • Click Create Deployment
  • Select Kafka + Flink Developer
  • Name the deployment. For this example we will use the name PrototypeCluster. Click create.
  • You will see a status bar as your deployment is provisioned.

Add an ACL

  • Once provisioning is complete you will be reminded to add an ACL. An ACL is how you set access permissions for your cluster. By default, your cluster will be closed to the outside world (including you).
  • Click the link to add an ACL. You can add/modify this list later by clicking the security button on the right side of your newly created cluster. This is an important step so pay close attention to the on screen help.

Create a Topic (optional)

A topic is a container to hold a stream of related data. We will use these topics for our tutorial on Flink projects.

  • From the Deployments tab inside your new cluster you should see a box containing the words Apache Kafka with a green check next to it. Click this box.
  • You should see an existing topic defaultsink. This is the default topic created during provisioning.
  • We want to create two new topics so click Add Topic
  • Name the topic input and set the number of partitions to 16. Replication factor is set to 1 for development cluster but can be increased in production deployments.
  • Click create
  • Repeat, naming the second topic output

Next step

Congratulations! You have setup a new Apache Kafka + Flink cluster. Now that your topics are created you’re ready to build a Flink project.

If you’d you’ve never used Kafka before you may want to try to send data with kafkacat.

Questions, feedback or find any of these steps unclear? Please contact us. We love helping people get started.

Leave a Reply

Your email address will not be published. Required fields are marked *