gRPC

Working with Protocol Buffers in Golang

This tutorial provides a basic Go programmer's introduction to working with protocol buffers

Golang Community Moderator. Software Engineer - Open Source Enthusiast. Goper Lover

Published · Aug 2 2020

Previous tutorial: Introduction To Protocol Buffers Interface Definition Language - Ii

This tutorial provides a basic Go programmer's introduction to working with protocol buffers, using the proto3 version of the protocol buffers language. By walking through creating a simple example application, it shows you how to
Define message formats in a .proto file.
Use the protocol buffer compiler.
Use the Go protocol buffer API to write and read messages.

  

This isn't a comprehensive guide to using protocol buffers in Go. For more detailed reference information, see the Protocol Buffer Language Guide.
. . .
Defining your protocol format
To create your address book application, you'll need to start with a .proto file. The definitions in a .proto file are simple: you add a message for each data structure you want to serialize, then specify a name and a type for each field in the message. In our example, the .proto file that defines the messages is addressbook.proto.

The .proto file starts with a package declaration, which helps to prevent naming conflicts between different projects.
syntax = "proto3";
package tutorial;

import "google/protobuf/timestamp.proto";

The go_package option defines the import path of the package which will contain all the generated code for this file. The Go package name will be the last path component of the import path. For example, our example will use a package name of "tutorialpb".
option go_package = "tutorialpb/addressbookpb";

Next, you have your message definitions. A message is just an aggregate containing a set of typed fields. Many standard simple data types are available as field types, including bool, int32, float, double, and string. You can also add further structure to your messages by using other message types as field types.
Let's define our messages in addressbook.proto file:
// go-pb/blob/master/addressbook.proto

syntax = "proto3";
package tutorial;

import "google/protobuf/timestamp.proto";
option go_package = "tutorialpb/addressbookpb";

message Person {
  string name = 1;
  int32 id = 2;  // Unique ID number for this person.
  string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    string number = 1;
    PhoneType type = 2;
  }

  repeated PhoneNumber phones = 4;

  google.protobuf.Timestamp last_updated = 5;
}

// Our address book file is just one of these.
message AddressBook {
  repeated Person people = 1;
}

In the above example, the Person message contains PhoneNumber messages, while the AddressBook message contains Person messages. You can even define message types nested inside other messages – as you can see, the PhoneNumber type is defined inside Person. You can also define enum types if you want one of your fields to have one of a predefined list of values – where you want to specify that a phone number can be one of MOBILE, HOME, or WORK.

You'll find a complete guide to writing .proto files – including all the possible field types – in the Protocol Buffer Language Guide. Don't go looking for facilities similar to class inheritance, though – protocol buffers don't do that.
. . .
Compiling your protocol buffers
Now that you have a .proto, the next thing you need to do is generate the classes you'll need to read and write AddressBook (and hence Person and PhoneNumber) messages. To do this, you need to run the protocol buffer compiler protoc on your .proto:
the 
If you haven't installed the compiler, then run following commands  If you want to use latest stable version of the compiler, simply do:
apt-get install libprotobuf-dev
apt-get install protobuf-compiler

Otherwise, if you want to build from source, don't forget to replace the URL with the latest version of the protocol buffers compiler.
apt-get install build-essential
wget https://protobuf.googlecode.com/svn/rc/protobuf-2.6.0.tar.gz 
tar xvfz protobuf-2.6.0.tar.gz
cd protobuf-2.6.0 $ ./configure && make install

Run the following command to install the Go protocol buffers plugin:go install 
go get -u -v github.com/golang/protobuf/proto
go get -u -v github.com/golang/protobuf/protoc-gen-go

The compiler plugin protoc-gen-go will be installed in $GOBIN, defaulting to $GOPATH/bin. It must be in your $PATH for the protocol compiler protoc to find it.

Now run the compiler, specifying the source directory (where your application's source code lives – the current directory is used if you don't provide a value), the destination directory (where you want the generated code to go; often the same as $SRC_DIR), and the path to your .proto. In this case, you would invoke:
protoc -I=. --go_out=. ./addressbook.proto

Because you want Go code, you use the --go_out option – similar options are provided for other supported languages. 

This generates  tutorialpb/addressbookpb/addressbook.pb.go  in your specified destination directory.

Generating addressbook.pb.go gives you the following useful types:
An AddressBook structure with a People field.
A Person structure with fields for Name, Id, Email and Phones.
A Person_PhoneNumber structure, with fields for Number and Type.
The type Person_PhoneType and a value defined for each value in the Person.PhoneType enum.

You can read more about the details of exactly what's generated in the Go Generated Code guide, but for the most part, you can treat these as perfectly ordinary Go types.
. . .

Marshaling and Unmarshalling a Message
The whole purpose of using protocol buffers is to serialize your data so that it can be parsed elsewhere. In Go, you use the proto library's Marshal function to serialize your protocol buffer data. 

A pointer to a protocol buffer message's struct implements the proto.Message interface. Calling proto.Marshal returns the protocol buffer, encoded in its wire format.

To parse an encoded message, you use the proto library's Unmarshal function. Calling this parses the data in buf as a protocol buffer and places the result in pb.

We’ve finally got everything in place to start writing our go code. Let’s start by defining a new Person and then we can marshal this object into a protobuf object.
// go-pb/master/main.go

package main

import (
	"fmt"
	"log"

	"github.com/golang/protobuf/proto"
	pb "github.com/gufranmirza/go-pb/tutorialpb/addressbookpb"
)

func main() {
	// Create Addressbook Object with few persons
	book := &pb.AddressBook{
		People: []*pb.Person{
			{
				Id:    1234,
				Name:  "John Doe",
				Email: "jdoe@example.com",
				Phones: []*pb.Person_PhoneNumber{
					{Number: "555-4321", Type: pb.Person_HOME},
				},
			},
			{
				Id:    1235,
				Name:  "Alex",
				Email: "alex@example.com",
				Phones: []*pb.Person_PhoneNumber{
					{Number: "1234-5678", Type: pb.Person_HOME},
				},
			},
		},
	}

	// let's unmarshal our Address object into byte array so
	// that we can use it transfer the data across services
	data, err := proto.Marshal(book)
	if err != nil {
		log.Fatal("marshaling error: ", err)
	}

	// printing out our raw protobuf object
	fmt.Println("Raw data", data)

	// let's go the other way and unmarshal
	// our byte array into an object we can modify
	// and use
	addresssBook := pb.AddressBook{}
	err = proto.Unmarshal(data, &addresssBook)
	if err != nil {
		log.Fatal("unmarshaling error: ", err)
	}

	for _, person := range addresssBook.People {
		// print out our `person` object
		// for good measure
		fmt.Println("===========================")
		fmt.Println("ID: ", person.GetId())
		fmt.Println("Name: ", person.GetName())
		fmt.Println("Email: ", person.GetEmail())
		fmt.Println("Phones: ", person.GetPhones())
	}
}

When we run this
go run main.go 
Raw data [10 45 10 8 74 111 104 110 32 68 111 101 16 210 9 26 16 106 100 111 101 64 101 120 97
 109 112 108 101 46 99 111 109 34 12 10 8 53 53 53 45 52 51 50 49 16 1 10 42 10 4 65 108 101 1
20 16 211 9 26 16 97 108 101 120 64 101 120 97 109 112 108 101 46 99 111 109 34 13 10 9 49 50 
51 52 45 53 54 55 56 16 1]
===========================
ID:  1234
Name:  John Doe
Email:  jdoe@example.com
Phones:  [number:"555-4321"  type:HOME]
===========================
ID:  1235
Name:  Alex
Email:  alex@example.com
Phones:  [number:"1234-5678"  type:HOME]

Congratulations! You are now using protocol buffers from Go.
. . .
Conclusion
So, in this tutorial, we had a good look at how you can get up and running with the protocol buffer data format within your own Go-based applications.

Hopefully, you found this tutorial useful, if you have any further questions or comments then please feel free to let me know in the comments section below!
Final code for this tutorial can be found here : https://github.com/gufranmirza/go-pb