How To Setup Confluent with Snowflake

Overview:  Here is a practical guide to getting started with setting up Confluent with Snowflake

Pre-Requisites:

  1.  You must have a Snowflake Account (the good thing is that this is really easy!)
    – 30 Day free trial if you do not have a Snowflake Account yet.  Free Snowflake Account Setup
  2.  This setup requires using Docker.  (I’ll have separate instructions to do this without Docker later)
  3.  You also need git.

Here we go – there are really 3 main parts to this setup:

  1. Get this docker version of confluent/kafka up and running.
  2. Create a topic on it to generate data for moving data into Snowflake
  3. Setup the Kafka to Snowflake Connector as the Destination with the right Snowflake connectivity

Part 1 – Get docker version of Confluent/Kafka running.

Let’s go…execute these commands in sequence:

Copy to Clipboard

Okay….The first time it will take a few minutes downloading…and you will see output eventually like this:

Copy to Clipboard

If you want to verify that everything is up and running then execute this command:

Copy to Clipboard

Then you should see some output like this:

Copy to Clipboard

You can also see from the Confluent Dashboard/Visualizations of the work you are going to do below.  (sometimes it takes awhile for this to refresh)

http://localhost:9021/

Part 2 – Create a topic to send Data to Snowflake.  Generate data for it with the DataGen functionality.

Let’s go…execute these commands in sequence:

Create a topic:

Copy to Clipboard

Generate data for the topic:
Let’s first configure a json file for the data we want to create.
Use whatever editor you want and create this file u.config

I’m using vi u.config and paste this in there:

u.config details

Copy to Clipboard

Part 3 – Download the Kafka to Snowflake Connector and configure it.

Okay.  So you have Confluent/Kafka up and running.  You have data generating into a topic.

So now just download the magical Kafka to Snowflake connector here: https://mvnrepository.com/artifact/com.snowflake/snowflake-kafka-connector/0.3.2
I’m sure by the time I publish this the version will change but for now assume it’s this one.

Once you have the file in the same directory we have been using for everything then copy it to the connect virtual machine where it needs to be to work.

Copy to Clipboard

You now need to create the configuration file to setup the Connector and the Sink associated with it that connects to the Snowflake Database.  This does assume you have already setup your rsa key.  You do have to fill in 6 of the settings below to have this setup for your specific configuration.Again, use your favorite editor.  I’m using:
vi connector_snowflake.config and entering in my specific details.

connector_snowflake.config details

Copy to Clipboard

Okay.  Almost there.  Now use this configuration file to setup the sink.

Copy to Clipboard

Now in a few seconds or minutes if you setup everything correctly the topic should be writing to the Snowflake table. Go into the database and schema you connected to and you should be able to execute something like:

Copy to Clipboard

Now you should see data flowing from Kafka to Snowflake. Enjoy!

Find out more about all the benefits Snowflake has to offer you and your business. Sign up for a free proof of concept!