gRPC on Node.js with Buf and TypeScript — Part 1

As gRPC and protocol buffers start to gain serious traction across the industry, more and more teams will likely start to explore what the different ecosystems (think Go, JavaScript/Node.js, Python, etc.) have to offer. Many will need to think about support, and potentially partial or full adoption of these technologies in the coming years.

In this chapter, we will explore Buf and use it to configure a modern build process for creating, maintaining, and consuming Protobuf APIs, thus laying strong foundations for working with gRPC and protocol buffers — in this example on Node.js.

In Part 2, we will build on these learnings and implement a gRPC server in TypeScript. And then finally, in Part 3, we will close off with useful tips and tricks on topics around code organisation, testing, deployments, further improving developer experience, and more.

Project configuration

Let’s start by creating a new project (you may optionally want to git init).

$ mkdir grpc-node-buf-typescript
$ cd grpc-node-buf-typescript

Protobuf messages and service definitions

Everything we do in Part 1 will take place in proto (within the project).

$ mkdir proto
$ cd proto
$ pwd/Users/me/src/grpc-node-buf-typescript/proto

Before we jump into tooling, let’s define our “common” messages as well as our service, aptly named HelloService.

In proto/com/language/language.proto:

And in proto/services/hello_service.proto:

Quality control with Buf

Now that we’ve defined our protos, let’s install and configure Buf to see what we can improve…

If you’re on macOS, you can use brew. For other installation options, please refer to the official guide.

$ brew tap bufbuild/buf
$ brew install buf

As per the official guide, we’ll first create our Buf config (buf.yaml). For the purpose of this tutorial we will use the strictest possible lint setting.

First, let’s verify that we’ve configured our roots correctly.

$ buf ls-filescom/language/language.proto
services/hello/hello_service.proto

Great! Now, let’s run the lint check.

$ buf check lintcom/language/language.proto:3:1:Package name "com.language" should be suffixed with a correctly formed version, such as "com.language.v1".
com/language/language.proto:10:5:Enum value name "EN" should be prefixed with "CODE_".
com/language/language.proto:10:5:Enum zero value name "EN" should be suffixed with "_UNSPECIFIED".
services/hello/hello_service.proto:5:1:Package name "services.hello" should be suffixed with a correctly formed version, such as "services.hello.v1".

The error messages are straight-forward, and we can apply the required fixes fairly quickly.

In proto/com/language/language.proto, we’ll update the Language.Code Enum values as suggested and the package will now need to include a version (v1).

In proto/services/hello_service.proto, we’ll update the reference to Language.Code to include v1 and the package will now need to include a version (v1) as well.

Running buf check lint fails again, this time because we are told that the directory structure must match the package name-spacing.

$ buf check lintcom/language/language.proto:3:1:Files with package "com.language.v1" must be within a directory "com/language/v1" relative to root but were in directory "com/language".
services/hello/hello_service.proto:5:1:Files with package "services.hello.v1" must be within a directory "services/hello/v1" relative to root but were in directory "services/hello".

All we need is to move the protos into their respective .VERSION directories, in our case v1. To verify:

$ buf ls-filescom/language/v1/language.proto
services/hello/v1/hello_service.proto

Our lint check should now pass.

$ buf check lint# No output means errors :)

FileDescriptorSets, Images, and detecting breaking changes

As you probably know…

FileDescriptorSets are the primitive used throughout the Protobuf ecosystem to represent a compiled Protobuf schema. They are also the primary artifact that protoc produces.

Read more: https://docs.buf.build/build-images

You probably also know that working with protoc and associated plugins can be cumbersome, some might even say painful.

Buf aims to alleviate these pains with what it calls Images. An Image is essentially Buf’s custom extension to FileDescriptorSets. And because it’s an extension, Images are FileDescriptorSets, and FileDescriptorSets are Images.

Building an Image with Buf is as simple as:

buf build -o image.json

You can, of course, choose from a variety of formats but for the purpose of this tutorial we’ll stick to json.

Now that we’ve successfully built an image, we can use it as a baseline to ensure that any new changes won’t break backwards compatibility.

Before we move on, however, we need to configure breaking change detection in buf.yaml:

Let’s see what happens if we try to introduce a breaking change in one of our messages. For example, we could change language_code from position 2 to position 3.

To run breaking change detection:

$ buf check breaking --against image.jsonservices/hello/v1/hello_service.proto:11:1:Previously present field "2" with name "language_code" on message "GreetRequest" was deleted without reserving the name "language_code".
services/hello/v1/hello_service.proto:11:1:Previously present field "2" with name "language_code" on message "GreetRequest" was deleted without reserving the number "2".

This can be especially useful in CI/CD contexts. You could, for example, store your “baseline” Image artefact somewhere and reject any potential changes that don’t pass the breaking change detection, thus only ever allowing the “baseline” Image to be updated on the condition the no breaking changes are introduced.

We will cover the process aspect in more detail in Part 3.

Protobuf generation for JavaScript

While it’s certainly possible to use the @grpc/proto-loader library to load .proto files directly into the runtime, for those wanting to use TypeScript this would effectively mean losing all typing. What’s more, if our aim is to create a consistent process for end-to-end Protobuf API management, deviating from the standards does not help us.

Luckily, Buf comes out-of-the-box with generation capability, which reduces the complexity of using and configuring protoc and associated plugins and further streamlines the process. Let’s give it a try!

⚠️ NOTE: The below assumes you have a working installation of protoc (version 3.12.3 or above).

$ protoc --versionlibprotoc 3.13.0

As per the official guide, let’s create our Buf generation config (buf.gen.yaml).

Notes:

• You can change or completely omit opt if you like, but for the purpose of this tutorial we’ll use the recommended options. See https://developers.google.com/protocol-buffers/docs/reference/javascript-generated#compiler-options for more details.

Running buf generate will generate the expected JavaScript code.

$ tree build/nodejsbuild/nodejs
├── com
│   └── language
│       └── v1
│           └── language_pb.js
└── services
    └── hello
        └── v1
            └── hello_service_pb.js6 directories, 2 files

Stub generation for gRPC on Node.js

For this to work, we’ll need to install grpc-tools to get hold of the grpc_tools_node_protoc plugin.

$ npm install grpc-tools@1.9.1 --global
$ which grpc_tools_node_protoc_plugin/Users/me/.nvm/versions/node/v14.15.0/bin/grpc_tools_node_protoc_plugin

Now, we can add the plugin to buf.gen.yaml:

Notes:

• The out option is identical to that of the js plugin— we want the JavaScript and gRPC generated modules to be colocated.
• You can change or completely omit the opt if you like, but for the purpose of this tutorial we’ll use the grpc_js option which instructs the plugin to generate code that uses @grpc/grpc-js (as opposed to the soon-to-be-deprecated grpc). See grpc-tools for more details.

After we’ve run buf generate again, we will see the new _grpc_pb.js files.

$ tree build/nodejsbuild/nodejs
├── com
│   └── language
│       └── v1
│           ├── language_grpc_pb.js
│           └── language_pb.js
└── services
    └── hello
        └── v1
            ├── hello_service_grpc_pb.js
            └── hello_service_pb.js6 directories, 4 files

Code generation for TypeScript (d.ts)

This last bit is easy… :)

First, we install the required plugin (protoc-gen-ts). Funnily enough, this is the only plugin we’ve used so far that follows the protoc-gen-NAME convention.

$ npm install grpc_tools_node_protoc_ts@5.0.1 --global
$ which protoc-gen-ts~/.nvm/versions/node/v14.15.0/bin/protoc-gen-ts

Buf will by default look for protoc-gen-NAME (NAME being ts as per example below) on $PATH so we don’t need to use the path option for this plugin.

Notes:

• The out option is identical to those of the grpc and js plugins — we need all TypeScript definitions to be colocated with their JavaScript counterparts.
• As is the case with the grpc_tools_node_protoc plugin, you can change or completely omit the opt if you like. For the purpose of this tutorial we’ll use the grpc_js option to instruct the plugin to generate code that uses @grpc/grpc-js (as opposed to the soon-to-be-deprecated grpc). See grpc_tools_node_protoc_ts for more details.

Let’s buf build for one last time. We should now see the TypeScript definitions.

$ tree build/nodejsbuild/nodejs
├── com
│   └── language
│       └── v1
│           ├── language_grpc_pb.js
│           ├── language_pb.d.ts
│           └── language_pb.js
└── services
    └── hello
        └── v1
            ├── hello_service_grpc_pb.d.ts
            ├── hello_service_grpc_pb.js
            ├── hello_service_pb.d.ts
            └── hello_service_pb.js6 directories, 7 files

Congrats! You’ve made it through Part 1 and you can use these modules in your server and client implementations.

In Part 2 we will expand on what we’ve built and implement a Node.js gRPC server in TypeScript.

You can find the completed project on GitHub.

Credit to Peter Edge and the team behind Buf.