OpenTelemetry for connect-go: Observability out of the box

Authored by Joshua Carpeggiani

Observability is a critical element for any robust system and monitoring incoming calls, error scenarios, and traces is a foundational component of it. As part of building a better RPC protocol, it’s important to provide a level of observability that customers need to successfully run their services. Today, we’re launching connect-opentelemetry-go to provide out-of-the-box support for OpenTelemetry instrumentation with connect-go.

connect-opentelemetry-go conforms to the OpenTelemetry specification for both the gRPC and Connect protocols. Connect can now be used with observability platforms like Grafana, Datadog, Zipkin, Jaeger, and many others!

The library supports:

• Tracing and metrics: The full stable OpenTelemetry API is supported out of the box.
• gRPC, gRPC-Web, and Connect: Traces and logs will reflect the correct information regardless of the protocol being used.
• Unary and streaming endpoints: Get the important Connect-specific data that simple HTTP instrumentation cannot provide.
• Client and servers: Observability is available for the entire end-to-end system.

We chose OpenTelemetry (OTel) because of its compatibility with a wide range of tools and platforms, as well as its strong focus on vendor neutrality. Unlike some proprietary observability solutions, OpenTelemetry is not tied to any specific vendor or cloud platform. It provides the freedom to choose the tools and platforms that best meet one’s needs, without being locked into a specific solution. With OpenTelemetry enabled in a connect-go project, the backend observability platform can be replaced without any changes to the application code.

Example integration

To get started with connect-opentelemetry-go, enable telemetry tracing and metrics using the otelconnect.NewInterceptor() option as shown in the example below:

func main() {
	// Set up OpenTelemetry globals
	setupOtel()
	// Construct a handler with otelconnect.NewInterceptor
	mux := http.NewServeMux()
	mux.Handle(
		pingv1connect.NewPingServiceHandler(
			&Pingservice{},
			// Add the otelconnect interceptor
			connect.WithInterceptors(otelconnect.NewInterceptor()),
		),
	)
	log.Fatal(http.ListenAndServe(":8080", mux))
}

func setupOtel() {
	// Exporting to different platforms can be configured here
	otel.SetTracerProvider(trace.NewTracerProvider())
	global.SetMeterProvider(metric.NewMeterProvider())
	otel.SetTextMapPropagator(propagation.TraceContext{})
}

On the client side, the same option can be passed into the constructor:

func makeRequest() {
	// Set up OpenTelemetry globals
	setupOtel()
	client := pingv1connect.NewPingServiceClient(
		http.DefaultClient,
		"http://localhost:8080",
		// Add the otelconnect interceptor
		connect.WithInterceptors(otelconnect.NewInterceptor()),
	)
	resp, err := client.Ping(
		context.Background(),
		connect.NewRequest(&pingv1.PingRequest{}),
	)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(resp)
}

This yields traces and metrics such as those shown below:

# Example trace:
{
  "Name": "observability.ping.v1.PingService/Ping",
  "SpanContext": {
          "TraceID": "69b452e89c49ecb385d08ea29454a26c",
          "SpanID": "0ba9117c170f8eeb",
          "TraceFlags": "01",
          "TraceState": "",
          "Remote": false
        }, # ...
#...
# Example metrics:
"Metrics": [
    {
      "Name": "rpc.server.duration",
      "Description": "",
      "Unit": "ms",
      "Data": {
              "DataPoints": [
# ...

With just these few lines of code, tracing and metrics can be added to connect-go applications. To see more examples and tutorials on how to set up metrics and tracing dashboards, head over to the OpenTelemetry website. More information about implementing OpenTelemetry for connect-go can be found by reading the docs and visiting the connect-opentelemetry-go GitHub repository.

What about existing observability?

Since connect-go uses the standard net/http library, it automatically works with any observability tools that are designed for net/http. On top of the information that HTTP observability exposes, OpenTelemetry answers some important RPC-specific questions:

• rpc.system: Was this call gRPC, gRPC-Web, or Connect?
• rpc.service / rpc.method: What service and method was called?
• responses_per_rpc: How many responses is each RPC service getting?
• error_code: What specific gRPC or Connect error was returned?

This type of information isn’t available at the HTTP level, but connect-opentelemetry-go fills that gap and allows for the creation of more powerful and effective observability tooling. To see a full list of supported attributes, see the OpenTelemetry specifications for Connect metrics and tracing.

Support and versioning

connect-opentelemetry-go adheres to both the gRPC OpenTelemetry metrics and tracing specifications and the Connect OpenTelemetry metrics and tracing specifications, so any dependency on the library will be backed up by a specification.

Get started with connect-go and OpenTelemetry

We’d love for you to try OpenTelemetry with connect-go! Check out the Connect observability docs, the connect-go repository, and connect-opentelemetry-go. If you have any questions or feedback, you can reach us through the Buf Slack or by filing a GitHub issue; we’d be more than happy to chat!