Sign inSign up
HomeGuidesManualsReference
Manuals / The Buf Schema Registry (BSR) / Automate code generation / Integrate with Remote Packages
/

Migrating from alpha

The remote generation alpha included a feature called remote code execution, which we now simply refer to as remote package. This guide walks you through what has changed, and outlines exactly how to migrate from remote code execution to remote packages.

Alpha deprecation

We've deprecated the remote generation alpha, but it will continue to work until April 30, 2023, at which time you'll need to migrate to remote packages.

Plugin changes

Public plugins are now solely maintained by the Buf team

In the alpha, public plugins could be uploaded by individual users with no verification. This caused a subpar experience for users who discovered plugins on their own, as well as caused a security headache for some of our customers. All public remote plugins are now maintained and verified by the Buf team directly.

To see all publicly-available plugins, go to buf.build/plugins. We think we've covered the vast majority of use cases, however if you find a useful plugin that should be added, please file an issue!

Private plugins available for enterprise and team customers

The BSR will still allow you to upload your custom, private plugins. This feature is already available for our enterprise users, and we'll be rolling this out to our buf.build users in the coming weeks as a paid feature. Contact us if you are interested in working with us!

Disable plugin uploads

Existing plugins and templates on the public BSR will remain available, but will be removed at a later date. We take breaking changes very seriously and want to provide ample opportunity for our valued users to migrate and request plugins which have not yet been added.

Templates removed

Templates were a complex concept for users to understand when interacting with the BSR, so we removed them. Instead, you reference plugins directly by name (and plugins can depend on the output of other plugins). A list of publicly-available plugins can be found at buf.build/plugins. Note that not all plugins can be used with Remote Packages - check the individual plugin for more details.

Versioning

The synthetic versioning scheme has been replaced by a more explicit versioning scheme comprised of the plugin version, module reference (datetime+short commit name) and revision. Example:

0.4.0-20220908151351-622fe7149695.1

This new semver-compatible versioning scheme can be pinned in lock files and always references a specific plugin + module for reproducibility.

Most users fetch the @latest version and will be unaffected by the versioning change.

Go module proxy

There are a couple of key changes from the alpha:

  • The base URL has changed from go.buf.build to buf.build/gen/go.
  • The path has changed to begin with the module name.
  • The template reference in the path has been replaced with plugins and moved to the end.
  • The version has changed to include plugin version and module commit information.

The new format is:

buf.build/gen/go/{moduleOwner}/{moduleName}/{pluginOwner}/{pluginName}

- go.buf.build/protocolbuffers/go/acme/petapis
+ buf.build/gen/go/acme/petapis/protocolbuffers/go

This means you'll need to search and replace the old import path with the new one and run go mod tidy.

The versioning has also changed to a more descriptive form:

{pluginVersion}-{moduleCommitTimestamp}-{moduleCommitName}.{pluginRevision}

Instead of relying on the commit sequence it now relies directly on commits. For ways to pin to a commit and other documentation please see the new Go proxy docs.

connect-go template

If you've used the connect-go template you'll need to update all connect imports to the generated code of the connect plugin.

The go.mod will now require two different imports, one for the go plugin and the other for the connect-go plugin.

go.mod
- go.buf.build/bufbuild/connect-go/acme/petapis
+ buf.build/gen/go/acme/petapis/protocolbuffers/go
+ buf.build/gen/go/acme/petapis/bufbuild/connect-go

Example:

package main

import (
-  petv1 "go.buf.build/bufbuild/connect-go/acme/petapis/pet/v1"
-  petv1connect "go.buf.build/bufbuild/connect-go/acme/petapis/pet/v1/petv1connect"
+  petv1 "buf.build/gen/go/acme/petapis/protocolbuffers/go/pet/v1"
+  petv1connect "buf.build/gen/go/acme/petapis/bufbuild/connect-go/pet/v1/petv1connect"
)

grpc-go template

If you've used the grpc-go template you'll need to update all grpc imports to the generated code of the grpc plugin.

The go.mod will now require two different imports, one for the go plugin and the other for the grpc-go plugin.

go.mod
- go.buf.build/grpc/go/acme/petapis
+ buf.build/gen/go/acme/petapis/protocolbuffers/go
+ buf.build/gen/go/acme/petapis/grpc/go

We patched the grpc-go plugin to generate code to a sub package. Earlier it used to generate code to the same package as the go plugin. The new import path is a subpackage that is named in the format: {goPackageName}grpc

Example:

package main

import (
-  petv1 "go.buf.build/grpc/go/acme/petapis/pet/v1"
+  petv1 "buf.build/gen/go/acme/petapis/protocolbuffers/go/pet/v1"
+  petv1grpc "buf.build/gen/go/acme/petapis/grpc/go/pet/v1/petv1grpc"
)

func main() {
  ...
-  client := petv1.NewPetStoreServiceClient(conn)
+  client := petv1grpc.NewPetStoreServiceClient(conn)
  res, err := client.GetPet(ctx, &petv1.GetPetRequest{})
  ...
}

protoc-gen-validate plugin

If you've used a custom template that included the protoc-gen-validate plugin as of now there is no direct migration path. We've taken stewardship of protoc-gen-validate from the Envoy team, and will continue to work to improve it, however protoc-gen-validate generated code is required to be generated to the same package as protoc-gen-go code, which does not fit cleanly into the remote packages model. In the meantime, switch to remote plugins using buf generate.

BSR NPM registry

Base URL

The base URL for the BSR NPM registry has changed, you'll want to update your .npmrc:

- @buf:registry=https://npm.buf.build
+ @buf:registry=https://buf.build/gen/npm/v1

Naming convention

The naming convention has changed because templates have been removed in favor of plugins. The new format is:

{moduleOwner}_{moduleName}.{pluginOwner}_{pluginName}

Note the dot (.) delimiter is used to break up the module and the plugin components.

This means you'll need to do 2 things:

  1. npm remove the old package and npm install the new package
  2. Search and replace application imports
- npm install @buf/bufbuild_es_bufbuild_eliza
+ npm install @buf/bufbuild_eliza.bufbuild_es

New documentation is available at NPM registry.

connect-web template

If you consumed connect-web template you'll need to update all imports for base types within your application code. This plugin now outputs plugin dependencies, namely protobuf-es, into a separate package.

Also, please note that the connect-web plugin has been renamed to connect-es. As a result, this plugin now outputs files with a _connect suffix rather than _connectweb.

One package (old behavior)

- node_modules
- └── @buf
-     └── bufbuild_connect-es_bufbuild_eliza
-         ├── buf
-         │   └── connect
-         │       └── demo
-         │           └── eliza
-         │               └── v1
-         │                   ├── eliza_connectweb.d.ts
-         │                   ├── eliza_connectweb.js
-         │                   ├── eliza_pb.d.ts
-         │                   └── eliza_pb.js
-         └── package.json

Two packages (new behavior)

+ node_modules
+ └── @buf
+     ├── bufbuild_eliza.bufbuild_connect-es
+     │   ├── buf
+     │   │   └── connect
+     │   │       └── demo
+     │   │           └── eliza
+     │   │               └── v1
+     │   │                   ├── eliza_connect.d.ts
+     │   │                   └── eliza_connect.js
+     │   └── package.json
+     └── bufbuild_eliza.bufbuild_es
+         ├── buf
+         │   └── connect
+         │       └── demo
+         │           └── eliza
+         │               └── v1
+         │                   ├── eliza_pb.d.ts
+         │                   └── eliza_pb.js
+         └── package.json

Using this example, there are two important things to note, assuming you have run npm uninstall and npm install based on the naming change mentioned above.

  • If your application code imported eliza_pb.js from @buf/bufbuild_connect-es_bufbuild_eliza/eliza_connectweb.js, then you'll want to update that import within your application code to reference the base types from @buf/bufbuild_eliza.bufbuild_es/eliza_pb.js.
  • If your application code imported eliza_connectweb.js from @buf/bufbuild_connect-es_bufbuild_eliza/eliza_connectweb.js, then you'll want to update that import within your application code to reference the Connect artifacts from @buf/bufbuild_eliza.bufbuild_connect-es/eliza_connect.js.
  1. Alpha deprecation
  2. Plugin changes
    1. Public plugins are now solely maintained by the Buf team
    2. Private plugins available for enterprise and team customers
    3. Disable plugin uploads
  3. Templates removed
  4. Versioning
  5. Go module proxy
    1. connect-go template
    2. grpc-go template
    3. protoc-gen-validate plugin
  6. BSR NPM registry
    1. Base URL
    2. Naming convention
    3. connect-web template
    4. One package (old behavior)
    5. Two packages (new behavior)