Replace protoc with buf

The buf CLI acts as a build system for all your .proto compilation and code generation needs. This guide describes how to migrate your existing protoc setup and migrate to using buf.

This guide assumes that you've installed buf and generate code by callingprotoc manually from scripts or a tool like make. Other guides are available for users currently using Protolock or Prototool.

We'll cover these common use cases:

• Compile .proto files to detect build failures.
• Generate code with protoc plugins.

Consider this file layout:

.
├── proto
│   └── acme
│       └── weather
│           └── v1
│               └── weather.proto
└── vendor
    └── protoc-gen-validate
        └── validate
            └── validate.proto

This protoc command is used to generate Go/gRPC client and server stubs:

$ protoc \
    -I proto \
    -I vendor/protoc-gen-validate \
    --go_out=. \
    --go_opt=paths=source_relative \
    --go-grpc_out=. \
    --go-grpc_opt=paths=source_relative \
    $(find proto -name '*.proto')

With protoc, each -I flag represents a directory used to search for imports. For example, given the above protoc invocation, the proto/acme/weather/v1/weather.proto and vendor/protoc-gen-validate/validate/validate.proto files are imported as acme/weather/v1/weather.proto and validate/validate.proto, respectively.

The placement of the buf.yaml is analogous to a protoc include (-I) path. With buf, there is no -I flag - each protoc -I path maps to a directory that contains a buf.yaml (called a module in Buf parlance), and multiple modules are stitched together with a buf.work.yaml, which defines a workspace.

The example shown above can be adapted to buf by adding a buf.yaml config file to each of the -I directories and creating a buf.work.yaml that specifies both directories:

.
├── buf.work.yaml
├── proto
│   ├── acme
│   │   └── weather
│   │       └── v1
│   │           └── weather.proto
│   └── buf.yaml
└── vendor
    └── protoc-gen-validate
        ├── buf.yaml
        └── validate
            └── validate.proto

version: v1
directories:
  - proto
  - vendor/protoc-gen-validate

version: v1
breaking:
  use:
    - FILE
lint:
  use:
    - DEFAULT

version: v1
breaking:
  use:
    - FILE
lint:
  use:
    - DEFAULT

The default buf.yaml configuration files shown above are created with this command:

$ buf mod init

With this, you can verify that the workspace compiles with this command:

$ buf build

The buf build command:

• Discovers the buf.work.yaml file found in the current directory.
• Collects all Protobuf files for each buf.yaml configuration.
• Copies the Protobuf files into memory.
• Compiles all Protobuf files.
• Outputs the compiled result to a configurable location (defaults to /dev/null)

The buf.yaml files aren't actually required in this case. You can run buf build without the buf.yaml configuration files and buf treats each directory specified in the buf.work.yaml as a module by default. We do, however, strongly recommend defining a buf.yamlfile.

Now that we've migrated the file layout to buf, we can simplify the protoc invocation used to generate Go/gRPC code with this buf.gen.yaml template:

version: v1
plugins:
  - plugin: go
    out: .
    opt:
      - paths=source_relative
  - plugin: go-grpc
    out: .
    opt:
      - paths=source_relative

The buf.gen.yaml file is typically placed next to the buf.work.yaml, so that your file layout looks like this:

.
├── buf.gen.yaml
├── buf.work.yaml
├── proto
│   ├── acme
│   │   └── weather
│   │       └── v1
│   │           └── weather.proto
│   └── buf.yaml
└── vendor
    └── protoc-gen-validate
        ├── buf.yaml
        └── validate
            └── validate.proto

With this, you can generate the Go/gRPC client and server stubs with this command:

$ buf generate

Most users only need a single buf.gen.yaml code generation template. If your project has more complex code generation requirement, however, you can use the --template flag to use more than one buf.gen.yaml templates.

For example, if you need different buf.gen.yaml configurations for your public and private API definitions, you might consider a setup like this, where the public directory contains your public APIs and the private directory contains your private APIs:

$ buf generate public --template buf.public.gen.yaml
$ buf generate private --template buf.private.gen.yaml