Flink Project | Eventador.io

Flink Projects

Projects allow developers to easily integrate existing SDLC workflows into a Flink project via Github. You check in your project and Eventador handles all the complexity of the build and deploy process. If you have a specific integration process or branching strategies all those workflows apply directly to your Flink code, and Flink deployments on Eventador.

Prerequisites

To create a project:

Now we will build and deploy your project. Select your new project from the projects page.

If you are deploying the FlinkReadWriteKafka template. Your form should look something like this:

flink_job

Switch over to the deployments tab and click “Apache Flink” box. You should see your Flink job running. The Eventador Console has an integrated view of the Flink cluster, showing jobs, the execution plan, the state and more. This gives you full control over running jobs, and the overall topology of the cluster. The Flink UI is built into the Eventador Console - so it has the same team level access controls as the rest of the platform.

flink_job

Since we know any message posted to input should be read and forwarded to output, we can use kafkacat to post a message then read from output to test our setup. If you have not used kafkacat before, you may want to start with this guide.

First, set your BROKERS environment variable BROKERS=<value from connections tab> e.g. BROKERS= xxxxxxxx-kafka0.pub.va.eventador.io:9092. This is covered further in the kafkacat guide.

Now we’ll validate output topic is empty (or see what’s already there).

kafkacat -C -b $BROKERS -t output -o beginning -e -q

Ok, lets post a message to input topic.

echo '{"hi":"there"}' | kafkacat -P -b $BROKERS -t input

And now we read output again to validate message was forwarded on by flink.

kafkacat -C -b $BROKERS -t output -o beginning -e -q
{"hi":"there"}

Indeed! It looks like Flink read our message to input and posted it to output. Of course this is a simple example but it lays foundation for a lot of awesome pipelines. Instead of forwarding every message you could hook into a log stream and only send alerts if you notice something awry or monitor thousands of warehouse sensors temperature rising above a threshold, the possibilities are endless.

Next Steps

To learn more about Flink you can check the flink docs. Here is a demo of Flink in action.

Please contact us if you experience any issues following this guide or have questions about Flink. We love helping people get started.