gRPC

Managing Microservice Schema and Interfaces in Distributed Environments With Protocol Buffers - [Part - I]

Defining the Microservice Schema, Interfaces, Objects, and strategies that can be used to manage Microservice API Interfaces.

Founder @ednsquare | Continiously Designing Testing Coding at ednsquare | Any query? shoot an email at ednsquare@gmail.com
Published · Oct 10 2020
This article is all about managing the Microservice Schema, Interfaces, Objects, and communication between the Microservices using the same. It describes some of the strategies that can be used to manage Microservice API Interfaces.

  

Prerequisites
To follow this article it is highly recommended that you first go through the links given below. These links provide the base for this article and we will be using a couple of concepts described in the articles given below.
Consumer-Driven Contracts: A Service Evolution Pattern
Microservices
Protocol Buffers
. . .

Disclaimer
I am not a microservice expert, these views on Microservice API Interfaces are personal of my own and may not match exactly with your views/opinions. These describe the terminology I use when building and managing Microservice API Interfaces. End of the day-use your brain to build and manage things in a way that makes more sense to you.
. . .

Target Audience
This article is all about managing the Microservice Schema, Interfaces, Objects, and communication between the Microservices using the same. This article will make more sense to you if:
You have a couple of Microservices and they communicate with each other via some Messaging-Bus or over HTTP.
You are looking for a batter way to manage the Microservice API Interfaces they expose to the consumers in a more predictable and efficient way.
You agree that  Microservice API Interfaces will evolve over time and more objects, fields, and interfaces will be added and removed as Microservice evolves, while doing so you want to make sure you don't break any existing consumers.
. . .

Terminology
Below is the explanation of the keywords used throughout in this article and what they mean.

Object: This is a custom data object exposed by the Microservice for example:
Type Product {
  ID : string
  Name: string
  Description: string
}

Microservice API Interface: This is an API exposed by Microservice which accepts or returns some data for example:
api.xyz.com/v1/products/productID/1 [GET] [HTTP1.1]
Accepts productID string in PATH
Returns Product JSON in BODY

Mcirosrvice Schema: A schema written in Interface Definition Language which describes the overall Objects and interfaces exposed by a Microservice.
syntax = "proto3";

package v1.product;

// Product message structure
message Product  {
    string id = 1; // id of the product
    string name = 2; // name of the product
    ...
}

// Service Interface for the Product Microservice API.
service ProductService {
    rpc GetProductByID(google.protobuf.Empty) returns (Product) {
        option (google.api.http) = {
            additional_bindings {
                get: "/v1/products/{productID}"
            }
        };
    }
}
. . .

Defining The Problem
Microservices exposes APIS and methods to access the data objects and manipulate them. So before we deep deep dive into the solution, first let's try to understand what are the problems we are trying to solve here, without further due let's write down the problems:
#1 Microservice API Interfaces are generally defined with loosely coupled schema such as JSON or swagger file or language-specific data structure. It's not reusable( if the consumer is written in different languages) consumers may need to write its own logic to serialize the data. Swagger is just a documentation of Microservice API Interfaces. Consumers need to depend on documentation to integrate with the system. 

#2 Changing the Microservice API Interfaces may result in breaking consumers due to inconsistent Microservice API Interfaces. Remember consumers may have their own logic to serialize the data. There is just some documentation of Microservice API Interfaces and consumers need to make changes according to it.

#3 No immediate feedback of consumers on changes in the Microservice  API Interfaces. One will know only when consumers start breaking then consumers need to check whether some changes have been made in Microservice API Interface. It's hard to identify changes in the Microservice API interface that will affect the current consumers.

#4 Managing the Microservice API Interfaces documentation is often overlooked and many times you will find that documentation is not up to date. And it needs to be automated so that any changes in the Microservice API Interfaces will also trigger the update into the API documentation automatically.
. . .

Solving Problem #1
We can solve this issue by using the Microservice Schema. Each microservice will have its own Microservice Schema. It represents the overall functionality of the Microservice itself. It exposes objects and interfaces to communicate with the Microservice.

A Microservice Schema expresses a service provider's business function capabilities in terms of the set of exportable elements necessary to support that functionality. From a service evolution point of view, a Microservice Schema is a container for a set of exportable business function elements. 

Microservice Schema contains the object definitions and interfaces, there could be multiple versions of Microservice Schema.  And changes in Microservice Schema should be forward and backward compatible. 

After compiling the Microservice Schema it will generate the code/classes/stubs (in the language of your choice) which will be used by the Provider and Consumers to communicate with Microservice.

Any consumers which want to communicate with the microservice will refer to a specific version of the Microservice Schema. Providers and Consumers can directly import the generated code in their programs.

More details at Consumer-Driven Contracts: A Service Evolution Pattern

In the next section, we will look at how we can define Microservice Schema with the help of Interface Definition language.
. . .

1.1 Defining Microservice Schema with Protocol Buffers
Microservice Schema can be defined with the help of an Interface Definition Language of your choice. Here for defining the Microservice Schema we will use the protocol buffers. 

What protocol buffers can provide to us:
Defining the Microservice Schema,  Interfaces, and Objects.
Forward and backward compatibility of Microservice Schema and Versioning.
Code generation of Microservice Schema and interfaces into the language of your choice. 
Protocol buffers are in binary format but provide the ability to serialize/de-serialize objects into JSON format.

Now its time to define a Microservice Schema with the help of protocol buffers,
this is how a Microservice Schema will look like written in Protocol Buffers
// microservice-proto/blob/main/proto/v1/product/product.proto

syntax = "proto3";

package v1.product;
option go_package = "github.com/gufranmirza/microservice-proto/proto/v1/product/v1product";

// Product message structure
message Product  {
    string id = 1; // id of the product
    string name = 2; // name of the product
    string description = 3; // product description
    string manufacturer = 4; // product manufacturer 
    string price = 5;  // product price
    bool in_stock = 6; // product is in stock or not
    ProductCategory category = 7; // category of the product it belogs to 
}

// ProductCategory is the category a product belogs to
enum ProductCategory {
    UNSPECIFIED = 0; // Product has not any category unspecified
    BOOKS_AND_LITERATURE = 1; // Books and literature
    FOOD_AND_DRINK = 2; // Food and drink
    COMPUTERS_AND_ELECTONICS = 3; // Computers and electronic
}

You can add and remove objects and interfaces to the schema as per requirements. You must follow the guidelines provided to working with Protocol Buffers. Below is the link to learn more about the Protocol Buffers.
You can read more about how to define the protocol buffers schema here  https://developers.google.com/protocol-buffers


If you look closer to the Microservice Schema we only have written the Object definitions and we have not defined any Microservice API Interface. There are different ways to define the Microservice API Interfaces as follows
#1 Defining Microservice API interfaces inside the Microservice Schema itself
// Service Interface for the Product Microservice API.
service ProductService {
    rpc GetProductByID(google.protobuf.Empty) returns (Product) {
        option (google.api.http) = {
            additional_bindings {
                get: "/v1/products/{productID}"
            }
        };
    }
}

This approach makes more sense if your Microservice is going to expose both the HTTP interface and gRPC interface to the consumers. For more details on this check https://github.com/grpc-ecosystem/grpc-gateway
#2 Or you can annotate each of the HTTP request handler function inside your code.  Then swagger documentation can be generated from annotated comments
provider-microservice/blob/master/src/web/services/v1/product/get_by_id.go#L37

// @Summary Get a Product
// @Description It allows to reterive a Product by ID
// @Tags product
// @Accept json
// @Produce json
// @Param productID path string true "Product ID"
// @Success 200 {object} v1product.Product{}
// @Failure 400 {object} v1error.ErrorResponse{}
// @Failure 404 {object} v1error.ErrorResponse{}
// @Failure 500 {object} v1error.ErrorResponse{}
// @Router /products/productID [GET]
func (p *product) GetByID(w http.ResponseWriter, r *http.Request) {
	...
}


Few things to remember
Each microservice defines a Microservice Shcmea which contains Interfaces, Objects with the help of a Protocol Buffer.
Each Consumer and Provider will refer to the same Microservice Schema to communicate with each other, making Consumers and Providers bounded by a Microservice Schema.
Protocol buffers or Microservice Schema need to be stored in a different repository,  also generated code may be stored there.
Any changes to the Objects and Interfaces of the Microservice  Schema will create a new version of the Microservice Schema, this is how versioning works.
. . .

1.2 Compiling Protocol Buffers
Now we have already defined our Microservice Schema with the help of the protocol buffers. Now we can use this Microservice Shema to generate the Stubs/Classes into the language of our choice, its language agnostic.

Let's go ahead and generate the code in the #golang, we can run the following commands to generate the structs in go.
// microservice-proto/blob/main/Makefile

protoc -I=./proto/v1/product/ --go_out="${GOPATH}/src/" ./proto/v1/product/*.proto 	

This will generate structs into the folder here,  now Microservice Provider and Consumers can import this generated code inside their #golang programs. Generated code contains Objects and Interfaces and methods to work with.

Developer documentation of the Microservice Schema contains the language-specific types and conversion and description of each object/field and interfaces. Run the following command to generate documentation.
// microservice-proto/blob/main/Makefile

protoc -I=./proto/v1/product/ --doc_out=markdown,doc.md:./proto/v1/product ./proto/v1/product/*.proto 

This will generate documentation into the folder here, you can choose the format of documentation you want.
. . .

1.3 Writing the Provider Microservice
Now we are ready to write a Microservice that will provide the functionality we have described in the Microservice Schema. We already know how to compile the Microservice Schema and generate the Staubs/Classes in the language we want. 

For writing the provider Microservice we will be using the #golang, you can choose any language you want.  We have already generated the Stubs/Structs in  #golang  and we can start using it in our programs

We are going to expose a very simple Microservice API Interface which allows the consumers to Get the product by its productID. 

So let's write some code that will allow us to retrieve product by id
// provider-microservice/blob/master/src/web/services/v1/product/get_by_id.go#L48

package product

import (
	"fmt"
	"net/http"

	"github.com/go-chi/chi"
	"github.com/go-chi/render"
	_ "github.com/gufranmirza/microservice-proto/proto/v1/error/v1error" //used in swag
	#1
        "github.com/gufranmirza/microservice-proto/proto/v1/product/v1product"
	"github.com/gufranmirza/provider-microservice/web/renderers"
)

#2
var products = &v1product.Products{
	Products: []*v1product.Product{
		{
			Id:           "1000",
			Name:         "Dell Ultrasharp Monitor",
			Description:  "Dell 24 inch (60.9 cm) Widescreen Ultrasharp LED Backlit Computer Monitor - Full HD, IPS Panel with VGA, DVI, Display, USB Ports - U2412M (Black)",
			Manufacturer: "Dell",
			Price:        "$200",
			InStock:      true,
			Category:     3,
		},
		...
	},
}

#3
func (p *product) GetByID(w http.ResponseWriter, r *http.Request) {
	txID := r.Header["transaction_id"][0]
	productID := chi.URLParam(r, "productID")

	for _, p := range products.Products {
		if productID == p.Id {
                        #4
			render.JSON(w, r, p)
			return
		}
	}

	p.logger.Info(txID).Infof("No product found with the given productID %v", productID)
	render.Render(w, r, renderers.ErrorNotFound(fmt.Errorf("No product found with the given productID %v", productID)))
	return
} 

We have imported the generated code which is obtained by compiling the Microservice Schema. It contains the language-specific implementation of the Objects we have defined into the Microservice Schema.
We have created a list of the products which contain different products.
Handler function to handle the request GET /products/productID request
We are filtering the products and returning the product if found. Its serialized into the JSON format and sent to the consumers, consumer can use the same generated code to deserialize the JSON response. 

There could be multiple versions of Microservice Schema that can be referred by the consumers they can upgrade the Microservice Schema version as per their convenience. None the less Microservice Schema written in Protocol Buffers is forward and backward compatible.
. . .

1.4  Writing the Consumers/Clients
Now we have our Provider Microservice up and running which exposes the certain functionality to the consumers. We can look at the Microservice Schema to see what are the interfaces and available to us and what data they are exchanging. 

Again consumers can be written in any language, we need to compile the Microservice Schema provided by the Microservice in the language we are using to write to the consumers. It will generate the code for Stubs/Classes which we can use in our consumer to deserialize the data.

Now its time to write the consumer code that will call the Provider Microservice API to retrieve the product details:
// consumer-microservice/blob/main/main.go
package main

import (
	...
	"github.com/gogo/protobuf/jsonpb"
        #1
	"github.com/gufranmirza/microservice-proto/proto/v1/product/v1product"
)

func main() {
        #2 
	resp, err := http.Get("http://localhost:8001/provider-api/v1/products/1000")
	if err != nil {
		fmt.Printf("Failed to reterive product data from API with err: %v \n", err)
		os.Exit(1)
	}

	defer func() {
		...
	}()

        #3
	product := &v1product.Product{}
	err = jsonpb.Unmarshal(resp.Body, product)
	if err != nil {
		fmt.Printf("Failed to deserialize product data %s from API with err: %v \n", resp.Body, err)
		os.Exit(1)
	}

	fmt.Println("#ProductID: ", product.Id)
	fmt.Println("#Name: ", product.Name)
	fmt.Println("#Description: ", product.Description)
	fmt.Println("#Manufacturer: ", product.Manufacturer)
	fmt.Println("#Price: ", product.Price)
	fmt.Println("#InStock: ", product.InStock)
	fmt.Println("#Category: ", product.Category)
}

We have imported the generated code which is obtained by compiling the Microservice Schema which is provided by the Provider Microservice. It contains the language-specific implementation of the Objects we have defined into the Microservice Schema.
We are making an HTTP API call on the Provider Microservice and it will return the product details in JSON format.
We have created a new product object and we need to deserialize the JSON response to the language-specific object. After deserialization object fields can be accessed.

Now Provider and consumers are referring to the same Microservice Schema, serialization and deserialization is the part of the Microservice Schema, consumers need not to write boilerplate code for serialization and deserialization. Any changes in the Microservice Schema will have to affect both the Provider and Consumer. Microservice Provider and Consumers are bounded by the Microservice Schema.
. . .
1.5 Summary
Now let's summarize the things we have done so far to solve the problem #1 we defined earlier in this article

At this point,
We have a single Microservice Schema which will be used by the Microservice Provider and Consumers. Consumers and Providers are bounded by the Schema. 
Microservice Schema is the only source of truth about Microservice API Interfaces and Objects it exposes to the outside world.
Anyone who wants to communicate with microservice should refer to the Microservice Schema, Provider and Consumers can compile Microservice Schema to generate Stubs/Objects/Classed in the language of their choice and import the generated code in their programs.
. . .

Solving Problem #2
While modifying the Microservice API INterfaces we want to make sure we do not break the existing consumers. We have some questions that need to be answered.
What happens if one of the fields in the Product Object has been removed? 
How is it going to affect consumers? 
How consumers will know if some fields have been modified or removed from the object?
The answer is simple, do schema validation at the consumer side 

Let's assume you have two consumers which are consuming the Product API /v1/products/{productID}. 

The first one is a mobile application that only requires the few fields of the Product object such as
{
  Id string required
  Name string required
  Price string required
  InStock bool required
}
The second one is a web application it requires few additional fields as compared to the previous one
{
  Id string required
  Name string required
  Price string required
  InStock bool required
  Description required
  Category required
}

Consumer needs to write the validation on the Product Object and need to verify the Product Object on receiving. If these fields are missing in the received message it will stop working. 

However, fields that are not required, and changes in those fields will not affect the consumers. No need to write validations for the fields you are not using. Validation is performed at the runtime.

In the proto2 version of Protocol Buffers allowed to define the fields to be marked as required in the schema itself but in proto3 every field is optional. Which makes more sense because consumers should be able to decide what is required and not required and write validations for it. more details here 

Validation of the Microservice Schema at the consumer's side is itself a big problem to be solved. We will be discussing this issue in detail and possible solution in the next article [ Part - II ]
. . .

Solving Problem #3
When you start modifying the Microservice API Interface you want to know how it will affect the current microservice consumers immediately in advance that will help to make well-informed decisions.

The only way to ensure that deploy the latest version of Microservice Schema and check if Consumers will work correctly together is by using expensive and brittle integration tests.

 https://pactflow.io/ 

An efficient way to solve this problem using the consumer-driven contract where each consumer must define a consumer contract that contains the Microservice API Interfaces, Objects, and Fileds they are using and validations for the same.

In general, a contract is between a consumer (for example, a client that wants to receive some data) and a provider (for example, an API on a server that provides the data the client needs).

It contains the validation for fields required by the consumers. One popular tool is Pact. Pact is a code-first consumer-driven contract testing tool.

When some changes are made into the Microservice Schema, then the consumer contracts provided by each consumer as executed against it, and consumers test will start breaking immediately if they are incompatible with the latest Microservice Schema. 

The best part is these tests run as the part to build which provides the immediate feedback of consumers to the changes. The same validation rules are used which we use at runtime to validate an object.
We will be discussing this issue in detail and possible solution in the next article [ Part - III ]
. . .
Solving Problem #4
Documenting Microservice  API interfaces can be automated easily with the protocol buffers and annotating comments inside the code itself.

Object fields and Interfaces can be easily documented by adding comments next to them when you are defining them inside your code
// microservice-proto/blob/main/proto/v1/product/product.proto

syntax = "proto3";

package v1.product;
option go_package = "github.com/gufranmirza/microservice-proto/proto/v1/product/v1product";

// Product message structure
message Product  {
    string id = 1; // id of the product
    string name = 2; // name of the product
    string description = 3; // product description
}

Developer documentation of the Microservice Schema contains the language-specific types and conversion and description of each object/field and interfaces. Run the following command to generate documentation.
// microservice-proto/blob/main/Makefile

protoc -I=./proto/v1/product/ --doc_out=markdown,doc.md:./proto/v1/product ./proto/v1/product/*.proto 

This will generate documentation into the folder here, you can choose the format of documentation you want. This command should run with the build itself to make sure documentation is in sync with changes.

You need to annotate each of the HTTP request handler function inside your code.  Then swagger documentation can be generated from annotated comments
provider-microservice/blob/master/src/web/services/v1/product/get_by_id.go#L37

// @Summary Get a Product
// @Description It allows to reterive a Product by ID
// @Tags product
// @Accept json
// @Produce json
// @Param productID path string true "Product ID"
// @Success 200 {object} v1product.Product{}
// @Failure 400 {object} v1error.ErrorResponse{}
// @Failure 404 {object} v1error.ErrorResponse{}
// @Failure 500 {object} v1error.ErrorResponse{}
// @Router /products/productID [GET]
func (p *product) GetByID(w http.ResponseWriter, r *http.Request) {
	...
}

The API documentation of the Microservice API interfaces can be obtained by running the following command.
// provider-microservice/blob/master/src/Makefile#L21

swag init --output=./web/docs

This will generate documentation into the folder here. This command should run with the build itself to make sure documentation is in sync with changes.

You may need to the following changes if you want to host the API documentation  on the server 
// provider-microservice/blob/master/src/web/router/router.go#L64

// ================= API Documentation ====================
r.Get(v1Prefix+"/swagger/*", swagger.Handler())

After these changes API, documentation is completely automated and documentation will be updated with each build itself if needed.
. . .
GitHub repository links
https://github.com/gufranmirza/microservice-proto
https://github.com/gufranmirza/provider-microservice
https://github.com/gufranmirza/consumer-microservice

Please feel free to comment below if you have any questions or concerns.
. . .