Go Language

Create Kafka producer with Golang and Sarama

There is a Kafka library for golang call sarama. We will be using it to build the producer.

Golang Community Moderator. Software Engineer - Open Source Enthusiast. Goper Lover

Published · Oct 19 2019

As promised, it's time to see Apache Kafka in action with a couple of microservices written in Go. You will be amazed how easy it's to have it up and running.

  
Meet Sarama
“Sarama is an MIT-licensed Go client library for Apache Kafka version 0.8 (and later)”. That’s from the Readme on Sarama github page (and arguable for a beginner, that might be the only thing that makes sense at first there).

Aside being robust, that Sarama implements the recent version of Kafka makes it the golang library of choice to use with Kafka. The consumer groups feature in recent Kafka versions finds great use in horizontally scaled applications because it ensures that for replicated consumers, duplicate execution of published message doesn’t occur.
. . .
Golang producer
There is a Kafka library for golang call sarama. I’m using sarama library to build the producer. Read more about sarama library from here.

Sarama has two kinds of producers, the SyncProducer and AsyncProducer. As you might have guessed, the SyncProducer publishes messages, blocking until they have been acknowledged while the AsyncProducer publishes messages using a non-blocking API. The choice will largely depend on the design of your overall architecture. However you should note that

The SyncProducer is a very thin wrapper around the AsyncProducer — the SendMessage method simply pushes the message onto the underlying async queue, then waits for the response. As such, you should get effectively identical performance with either.

Here’s a sample code snippet to create a Sarama producer:

Code is simple and self explanatory. What I’m doing here is initializing thesarama.SyncProducer. Then start to read the command line inputs viabufio.Reader. Finally publishing read data in to Kafka topic senz.

To create an AsyncProducer we have instead
producer, err := sarama.NewAsyncProducer(brokers, config)

Check this out for more producer configuration options.

To produce a message, wrap the message in the saram.ProducerMessage struct and then pass it on to the producer instance.

For all available ProducerMessage configuration, including how to specify the partition to send message to using partition key, checkout the documentation.

SendMessage returns the partition the message was sent to, the offset of the message on Kafka and error information.

Note: be sure to shutdown the producer to avoid memory leaks by calling the Close() function on the producer instance.
To ensure a high chance for data consistency, it’s often
. . .
Conclusion
Getting started with Sarama might seem overwhelming at first.

But underneath all the available configurations and options, it is basically, all about two components producers and consumers and of course, how they interact with each other. 