gRPC

Working with Protocol Buffers in Javascript/NodeJS

This tutorial provides a basic JavaScript programmer's introduction to working with protocol buffers

Golang Community Moderator. Software Engineer - Open Source Enthusiast. Goper Lover

Published · Aug 2 2020

This tutorial provides a basic Python programmer's introduction to working with protocol buffers. By walking through creating a simple example application, it shows you how to
Define message formats in a .proto file.
Use the protocol buffer compiler.
Use the Python protocol buffer API to write and read messages.

  
This isn't a comprehensive guide to using protocol buffers in Go. For more detailed reference information, see the Protocol Buffer Language Guide.
. . .
Defining Your Protocol Format
To create your address book application, you'll need to start with a .proto file. The definitions in a .proto file are simple: you add a message for each data structure you want to serialize, then specify a name and a type for each field in the message. In our example, the .proto file that defines the messages is addressbook.proto.

The .proto file starts with a package declaration, which helps to prevent naming conflicts between different projects
syntax = "proto3"; package tutorial;  
import "google/protobuf/timestamp.proto";

Next, you have your message definitions. A message is just an aggregate containing a set of typed fields. Many standard simple data types are available as field types, including bool, int32, float, double, and string. You can also add further structure to your messages by using other message types as field types. Lets create addressbook.proto file and start writing the schema
// javascript-pb/blob/master/addressbook.proto

syntax = "proto3";
package tutorial;

import "google/protobuf/timestamp.proto";

message Person {
  string name = 1;
  int32 id = 2;  // Unique ID number for this person.
  string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    string number = 1;
    PhoneType type = 2;
  }

  repeated PhoneNumber phones = 4;

  google.protobuf.Timestamp last_updated = 5;
}

// Our address book file is just one of these.
message AddressBook {
  repeated Person people = 1;
}

In the above example, the Person message contains PhoneNumber messages, while the AddressBook message contains Person messages. You can even define message types nested inside other messages – as you can see, the PhoneNumber type is defined inside Person. You can also define enum types if you want one of your fields to have one of a predefined list of values – where you want to specify that a phone number can be one of MOBILE, HOME, or WORK.  

You'll find a complete guide to writing .proto files – including all the possible field types – in the Protocol Buffer Language Guide. Don't go looking for facilities similar to class inheritance, though – protocol buffers don't do that.
. . .
Compiling your protocol buffers
Now that you have a .proto, the next thing you need to do is generate the classes you'll need to read and write AddressBook (and hence Person and PhoneNumber) messages. To do this, you need to run the protocol buffer compiler protoc on your .proto:  

If you haven't installed the compiler, then run following commands  If you want to use latest stable version of the compiler, simply do:
apt-get install libprotobuf-dev
apt-get install protobuf-compiler

 Otherwise, if you want to build from source, don't forget to replace the URL with the latest version of the protocol buffers compiler.
apt-get install build-essential
wget https://protobuf.googlecode.com/svn/rc/protobuf-2.6.0.tar.gz 
tar xvfz protobuf-2.6.0.tar.gzcd protobuf-2.6.0 .
/configure && make install

Run the following command to install the Python protocol buffers plugin:python install 
npm install google-protobuf

The protocol buffer compiler produces JavaScript output when invoked with the --js_out= command-line flag. The parameter to the --js_out= option is the directory where you want the compiler to write your JavaScript output. The exact output depends on whether you want to use Closure-style imports or CommonJS-style imports; the compiler supports both.
Closure Imports
CommonJS Imports
The protocol buffer compiler for JavaScript has many options to customize its output in addition to the library and import_style options mentioned above. For example:
binary: Using this option generates code that lets you serialize and deserialize your proto from the protocol buffers binary wire format. Its recommend that you enable this option.--js_out=library=myprotos_lib.js,binary:.

error_on_name_conflict: Using this option means a compiler error is returned if there are two types in your input that would generate output files with the same name.

As in the above, multiple options can be specified, separated by commas. You can see a complete list of available options in js_generator.h.

For this tutorial, we are assuming that we want the CommonJS imports. Now run the compiler,  before invoking compiler make sure you have created build 
directory where you want the output of protocol buffers, you would invoke:
protoc -I=. --js_out=import_style=commonjs,binary:build ./addressbook.proto

 Because you want Javascript classes, you use the --js_out option – similar options are provided for other supported languages. 

This generates build/addressbook_pb.js   in your specified destination directory

Your generated class has accessors for all its fields (which we'll look at in the following sections) and the following methods that apply to the entire message:
toObject(): Returns an object representation of the message, suitable for use in Soy templates. This method comes in static and instance versions. Field names that are reserved in JavaScript are renamed to pb_name. If you don't want to generate this method (for instance, if you're not going to use it and are concerned about code size), set jspb.Message.GENERATE_TO_OBJECT to false before code generation. Note that this representation is not the same as proto3's JSON representation.
cloneMessage(): Creates a deep clone of this message and its fields.

The following methods are also provided if you have enabled the binary option when generating your code:
deserializeBinary(): Static method. Deserializes a message from protocol buffers binary wire format and returns a new populated message object. Does not preserve any unknown fields in the binary message.
deserializeBinaryFromReader(): Static method. Deserializes a message in protocol buffers binary wire format from the provided BinaryReader into the provided message object. Does not preserve any unknown fields in the binary message.
serializeBinary(): Serializes this message to protocol buffers binary wire format.
serializeBinaryToWriter(): Serializes this message in protocol buffers binary wire format to the specified BinaryWriter. This method has a static variant where you can serialize a specified message to the BinaryWriter.
. . .
Serializing and Deserializing Messages
Finally, each protocol buffer class has methods for writing and reading messages of your chosen type using the protocol buffer binary format. These include:
deserializeBinary(): Static method. Deserializes a message from protocol buffers binary wire format and returns a new populated message object. Does not preserve any unknown fields in the binary message.
serializeBinary(data): Serializes this message to protocol buffers binary wire format.
These are just a couple of the options provided for parsing and serialization. Again, see the Message API reference for a complete list.

Now let's try using your protocol buffer classes. We will create few Person objects, and then use them to serialize and deserialize objects 
// javascript-pb/master/index.js

var pb = require('./build/addressbook_pb');

var phone_book = new pb.AddressBook()

// Person 1
var person1 = new pb.Person()
person1.setId(1001)
person1.setName("John Doe")
person1.setEmail("jdoe@example.com")
var phone_number1 = new pb.Person.PhoneNumber();
phone_number1.setNumber("12345-67890")
phone_number1.setType(pb.Person.PhoneType.WORK)
person1.setPhonesList([phone_number1])

// Person 2
var person2 = new pb.Person()
person2.setId(1002)
person2.setName("Alex")
person2.setEmail("alex@example.com")
var phone_number2 = new pb.Person.PhoneNumber();
phone_number2.setNumber("100122-5889")
phone_number2.setType(pb.Person.PhoneType.WORK)
person2.setPhonesList([phone_number2])

// add persons to the phonebook
phone_book.setPeopleList([person1, person2])

// let's stringity our Address object so binary array
// that we can use it transfer the data across services
data = phone_book.serializeBinary()

// printing out our raw protobuf object
console.log("Raw data: ", data)

// let's go the other way and parse our raw protobuf binary array 
// we can modify and use
var address_book  = pb.AddressBook.deserializeBinary(data)
var person_list = address_book.getPeopleList()

for (var i=0; i < person_list.length ; i++) {
    var person = person_list[i]
    var phones = person.getPhonesList()
    console.log("===========================")
    console.log("Person ID:", person.getId())
    console.log("Name:", person.getName())
    console.log("E-mail:", person.getEmail())
    for (var k=0; k < phones.length; k++) {
        var phone = phones[k]
        console.log("Phone:", phone.getType() == 2 ? "WORK" : "", phone.getNumber())
    }
}

 Now lets run our index.js file which will output following
node index.js 
Raw data:  Uint8Array [
   10,  48,  10,   8,  74, 111, 104, 110,  32,  68, 111, 101,
   16, 233,   7,  26,  16, 106, 100, 111, 101,  64, 101, 120,
   97, 109, 112, 108, 101,  46,  99, 111, 109,  34,  15,  10,
   11,  49,  50,  51,  52,  53,  45,  54,  55,  56,  57,  48,
   16,   2,  10,  44,  10,   4,  65, 108, 101, 120,  16, 234,
    7,  26,  16,  97, 108, 101, 120,  64, 101, 120,  97, 109,
  112, 108, 101,  46,  99, 111, 109,  34,  15,  10,  11,  49,
   48,  48,  49,  50,  50,  45,  53,  56,  56,  57,  16,   2
]
===========================
Person ID: 1001
Name: John Doe
E-mail: jdoe@example.com
Phone: WORK 12345-67890
===========================
Person ID: 1002
Name: Alex
E-mail: alex@example.com
Phone: WORK 100122-5889

Congratulations! You are now using protocol buffers from Javascript.
. . .
Conclusion
So, in this tutorial, we had a good look at how you can get up and running with the protocol buffer data format within your own Javascript-based applications.  

Hopefully, you found this tutorial useful, if you have any further questions or comments then please feel free to let me know in the comments section below!
Final code for this tutorial can be found here: https://github.com/gufranmirza/javascript-pb