1# Go support for Protocol Buffers
2
3[](https://pkg.go.dev/google.golang.org/protobuf)
4[](https://travis-ci.org/protocolbuffers/protobuf-go)
5
6This project hosts the Go implementation for
7[protocol buffers](https://protobuf.dev), which is a
8language-neutral, platform-neutral, extensible mechanism for serializing
9structured data. The protocol buffer language is a language for specifying the
10schema for structured data. This schema is compiled into language specific
11bindings. This project provides both a tool to generate Go code for the
12protocol buffer language, and also the runtime implementation to handle
13serialization of messages in Go. See the
14[protocol buffer developer guide](https://protobuf.dev/overview)
15for more information about protocol buffers themselves.
16
17This project is comprised of two components:
18
19* Code generator: The
20 [`protoc-gen-go`](https://pkg.go.dev/google.golang.org/protobuf/cmd/protoc-gen-go)
21 tool is a compiler plugin to `protoc`, the protocol buffer compiler. It
22 augments the `protoc` compiler so that it knows how to
23 [generate Go specific code for a given `.proto` file](https://protobuf.dev/reference/go/go-generated).
24
25* Runtime library: The
26 [`protobuf`](https://pkg.go.dev/mod/google.golang.org/protobuf) module
27 contains a set of Go packages that form the runtime implementation of
28 protobufs in Go. This provides the set of interfaces that
29 [define what a message is](https://pkg.go.dev/google.golang.org/protobuf/reflect/protoreflect)
30 and functionality to serialize message in various formats (e.g.,
31 [wire](https://pkg.go.dev/google.golang.org/protobuf/proto),
32 [JSON](https://pkg.go.dev/google.golang.org/protobuf/encoding/protojson),
33 and
34 [text](https://pkg.go.dev/google.golang.org/protobuf/encoding/prototext)).
35
36See the
37[developer guide for protocol buffers in Go](https://protobuf.dev/getting-started/gotutorial)
38for a general guide for how to get started using protobufs in Go.
39
40This project is the second major revision of the Go protocol buffer API
41implemented by the
42[`google.golang.org/protobuf`](https://pkg.go.dev/mod/google.golang.org/protobuf)
43module. The first major version is implemented by the
44[`github.com/golang/protobuf`](https://pkg.go.dev/mod/github.com/golang/protobuf)
45module.
46
47## Package index
48
49Summary of the packages provided by this module:
50
51* [`proto`](https://pkg.go.dev/google.golang.org/protobuf/proto): Package
52 `proto` provides functions operating on protobuf messages such as cloning,
53 merging, and checking equality, as well as binary serialization.
54* [`encoding/protojson`](https://pkg.go.dev/google.golang.org/protobuf/encoding/protojson):
55 Package `protojson` serializes protobuf messages as JSON.
56* [`encoding/prototext`](https://pkg.go.dev/google.golang.org/protobuf/encoding/prototext):
57 Package `prototext` serializes protobuf messages as the text format.
58* [`encoding/protowire`](https://pkg.go.dev/google.golang.org/protobuf/encoding/protowire):
59 Package `protowire` parses and formats the low-level raw wire encoding. Most
60 users should use package `proto` to serialize messages in the wire format.
61* [`reflect/protoreflect`](https://pkg.go.dev/google.golang.org/protobuf/reflect/protoreflect):
62 Package `protoreflect` provides interfaces to dynamically manipulate
63 protobuf messages.
64* [`reflect/protoregistry`](https://pkg.go.dev/google.golang.org/protobuf/reflect/protoregistry):
65 Package `protoregistry` provides data structures to register and lookup
66 protobuf descriptor types.
67* [`reflect/protodesc`](https://pkg.go.dev/google.golang.org/protobuf/reflect/protodesc):
68 Package `protodesc` provides functionality for converting
69 `descriptorpb.FileDescriptorProto` messages to/from the reflective
70 `protoreflect.FileDescriptor`.
71* [`reflect/protopath`](https://pkg.go.dev/google.golang.org/protobuf/reflect/protopath):
72 Package `protopath` provides a representation of a sequence of
73 protobuf reflection operations on a message.
74* [`reflect/protorange`](https://pkg.go.dev/google.golang.org/protobuf/reflect/protorange):
75 Package `protorange` provides functionality to traverse a protobuf message.
76* [`testing/protocmp`](https://pkg.go.dev/google.golang.org/protobuf/testing/protocmp):
77 Package `protocmp` provides protobuf specific options for the `cmp` package.
78* [`testing/protopack`](https://pkg.go.dev/google.golang.org/protobuf/testing/protopack):
79 Package `protopack` aids manual encoding and decoding of the wire format.
80* [`testing/prototest`](https://pkg.go.dev/google.golang.org/protobuf/testing/prototest):
81 Package `prototest` exercises the protobuf reflection implementation for
82 concrete message types.
83* [`types/dynamicpb`](https://pkg.go.dev/google.golang.org/protobuf/types/dynamicpb):
84 Package `dynamicpb` creates protobuf messages at runtime from protobuf
85 descriptors.
86* [`types/known/anypb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/anypb):
87 Package `anypb` is the generated package for `google/protobuf/any.proto`.
88* [`types/known/timestamppb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/timestamppb):
89 Package `timestamppb` is the generated package for
90 `google/protobuf/timestamp.proto`.
91* [`types/known/durationpb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/durationpb):
92 Package `durationpb` is the generated package for
93 `google/protobuf/duration.proto`.
94* [`types/known/wrapperspb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/wrapperspb):
95 Package `wrapperspb` is the generated package for
96 `google/protobuf/wrappers.proto`.
97* [`types/known/structpb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/structpb):
98 Package `structpb` is the generated package for
99 `google/protobuf/struct.proto`.
100* [`types/known/fieldmaskpb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/fieldmaskpb):
101 Package `fieldmaskpb` is the generated package for
102 `google/protobuf/field_mask.proto`.
103* [`types/known/apipb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/apipb):
104 Package `apipb` is the generated package for
105 `google/protobuf/api.proto`.
106* [`types/known/typepb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/typepb):
107 Package `typepb` is the generated package for
108 `google/protobuf/type.proto`.
109* [`types/known/sourcecontextpb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/sourcecontextpb):
110 Package `sourcecontextpb` is the generated package for
111 `google/protobuf/source_context.proto`.
112* [`types/known/emptypb`](https://pkg.go.dev/google.golang.org/protobuf/types/known/emptypb):
113 Package `emptypb` is the generated package for
114 `google/protobuf/empty.proto`.
115* [`types/descriptorpb`](https://pkg.go.dev/google.golang.org/protobuf/types/descriptorpb):
116 Package `descriptorpb` is the generated package for
117 `google/protobuf/descriptor.proto`.
118* [`types/pluginpb`](https://pkg.go.dev/google.golang.org/protobuf/types/pluginpb):
119 Package `pluginpb` is the generated package for
120 `google/protobuf/compiler/plugin.proto`.
121* [`compiler/protogen`](https://pkg.go.dev/google.golang.org/protobuf/compiler/protogen):
122 Package `protogen` provides support for writing protoc plugins.
123* [`cmd/protoc-gen-go`](https://pkg.go.dev/google.golang.org/protobuf/cmd/protoc-gen-go):
124 The `protoc-gen-go` binary is a protoc plugin to generate a Go protocol
125 buffer package.
126
127## Reporting issues
128
129The issue tracker for this project is currently located at
130[golang/protobuf](https://github.com/golang/protobuf/issues).
131
132Please report any issues there with a sufficient description of the bug or
133feature request. Bug reports should ideally be accompanied by a minimal
134reproduction of the issue. Irreproducible bugs are difficult to diagnose and fix
135(and likely to be closed after some period of time). Bug reports must specify
136the version of the
137[Go protocol buffer module](https://github.com/protocolbuffers/protobuf-go/releases)
138and also the version of the
139[protocol buffer toolchain](https://github.com/protocolbuffers/protobuf/releases)
140being used.
141
142## Contributing
143
144This project is open-source and accepts contributions. See the
145[contribution guide](https://github.com/protocolbuffers/protobuf-go/blob/master/CONTRIBUTING.md)
146for more information.
147
148## Compatibility
149
150This module and the generated code are expected to be stable over time. However,
151we reserve the right to make breaking changes without notice for the following
152reasons:
153
154* **Security:** A security issue in the specification or implementation may
155 come to light whose resolution requires breaking compatibility. We reserve
156 the right to address such issues.
157* **Unspecified behavior:** There are some aspects of the protocol buffer
158 specification that are undefined. Programs that depend on unspecified
159 behavior may break in future releases.
160* **Specification changes:** It may become necessary to address an
161 inconsistency, incompleteness, or change in the protocol buffer
162 specification, which may affect the behavior of existing programs. We
163 reserve the right to address such changes.
164* **Bugs:** If a package has a bug that violates correctness, a program
165 depending on the buggy behavior may break if the bug is fixed. We reserve
166 the right to fix such bugs.
167* **Generated additions**: We reserve the right to add new declarations to
168 generated Go packages of `.proto` files. This includes declared constants,
169 variables, functions, types, fields in structs, and methods on types. This
170 may break attempts at injecting additional code on top of what is generated
171 by `protoc-gen-go`. Such practice is not supported by this project.
172* **Internal changes**: We reserve the right to add, modify, and remove
173 internal code, which includes all unexported declarations, the
174 [`protoc-gen-go/internal_gengo`](https://pkg.go.dev/google.golang.org/protobuf/cmd/protoc-gen-go/internal_gengo)
175 package, the
176 [`runtime/protoimpl`](https://pkg.go.dev/google.golang.org/protobuf/runtime/protoimpl?tab=doc)
177 package, and all packages under
178 [`internal`](https://pkg.go.dev/google.golang.org/protobuf/internal).
179
180Any breaking changes outside of these will be announced 6 months in advance to
181[protobuf@googlegroups.com](https://groups.google.com/forum/#!forum/protobuf).
182
183Users should use generated code produced by a version of
184[`protoc-gen-go`](https://pkg.go.dev/google.golang.org/protobuf/cmd/protoc-gen-go)
185that is identical to the runtime version provided by the
186[protobuf module](https://pkg.go.dev/mod/google.golang.org/protobuf). This
187project promises that the runtime remains compatible with code produced by a
188version of the generator that is no older than 1 year from the version of the
189runtime used, according to the release dates of the minor version. Generated
190code is expected to use a runtime version that is at least as new as the
191generator used to produce it. Generated code contains references to
192[`protoimpl.EnforceVersion`](https://pkg.go.dev/google.golang.org/protobuf/runtime/protoimpl?tab=doc#EnforceVersion)
193to statically ensure that the generated code and runtime do not drift
194sufficiently far apart.
195
196## Historical legacy
197
198This project is the second major revision
199([released in 2020](https://blog.golang.org/a-new-go-api-for-protocol-buffers))
200of the Go protocol buffer API implemented by the
201[`google.golang.org/protobuf`](https://pkg.go.dev/mod/google.golang.org/protobuf)
202module. The first major version
203([released publicly in 2010](https://blog.golang.org/third-party-libraries-goprotobuf-and))
204is implemented by the
205[`github.com/golang/protobuf`](https://pkg.go.dev/mod/github.com/golang/protobuf)
206module.
207
208The first version predates the release of Go 1 by several years. It has a long
209history as one of the first core pieces of infrastructure software ever written
210in Go. As such, the Go protobuf project was one of many pioneers for determining
211what the Go language should even look like and what would eventually be
212considered good design patterns and “idiomatic” Go (by simultaneously being
213both positive and negative examples of it).
214
215Consider the changing signature of the `proto.Unmarshal` function as an example
216of Go language and library evolution throughout the life of this project:
217
218```go
219// 2007/09/25 - Conception of Go
220
221// 2008/11/12
222export func UnMarshal(r io.Read, pb_e reflect.Empty) *os.Error
223
224// 2008/11/13
225export func UnMarshal(buf *[]byte, pb_e reflect.Empty) *os.Error
226
227// 2008/11/24
228export func UnMarshal(buf *[]byte, pb_e interface{}) *os.Error
229
230// 2008/12/18
231export func UnMarshal(buf []byte, pb_e interface{}) *os.Error
232
233// 2009/01/20
234func UnMarshal(buf []byte, pb_e interface{}) *os.Error
235
236// 2009/04/17
237func UnMarshal(buf []byte, pb_e interface{}) os.Error
238
239// 2009/05/22
240func Unmarshal(buf []byte, pb_e interface{}) os.Error
241
242// 2011/11/03
243func Unmarshal(buf []byte, pb_e interface{}) error
244
245// 2012/03/28 - Release of Go 1
246
247// 2012/06/12
248func Unmarshal(buf []byte, pb Message) error
249```
250
251These changes demonstrate the difficulty of determining what the right API is
252for any new technology. It takes time multiplied by many users to determine what
253is best; even then, “best” is often still somewhere over the horizon.
254
255The change on June 6th, 2012 added a degree of type-safety to Go protobufs by
256declaring a new interface that all protobuf messages were required to implement:
257
258```go
259type Message interface {
260 Reset()
261 String() string
262 ProtoMessage()
263}
264```
265
266This interface reduced the set of types that can be passed to `proto.Unmarshal`
267from the universal set of all possible Go types to those with a special
268`ProtoMessage` marker method. The intention of this change is to limit the
269protobuf API to only operate on protobuf data types (i.e., protobuf messages).
270For example, there is no sensible operation if a Go channel were passed to the
271protobuf API as a channel cannot be serialized. The restricted interface would
272prevent that.
273
274This interface does not behaviorally describe what a protobuf message is, but
275acts as a marker with an undocumented expectation that protobuf messages must be
276a Go struct with a specific layout of fields with formatted tags. This
277expectation is not statically enforced by the Go language, for it is an
278implementation detail checked dynamically at runtime using Go reflection. Back
279in 2012, the only types with this marker were those generated by
280`protoc-gen-go`. Since `protoc-gen-go` would always generate messages with the
281proper layout of fields, this was deemed an acceptable and dramatic improvement
282over `interface{}`.
283
284Over the next 10 years,
285[use of Go would skyrocket](https://blog.golang.org/10years) and use of
286protobufs in Go would skyrocket as well. With increased popularity also came
287more diverse usages and requirements for Go protobufs and an increased number of
288custom `proto.Message` implementations that were not generated by
289`protoc-gen-go`.
290
291The increasingly diverse ecosystem of Go types implementing the `proto.Message`
292interface led to incompatibilities, which often occurred when:
293
294* **Passing custom `proto.Message` types to the protobuf APIs**: A concrete
295 message implementation might work with some top-level functions (e.g.,
296 `proto.Marshal`), but cause others (e.g., `proto.Equal`) to choke and panic.
297 This occurs because the type only had partial support for being an actual
298 message by only implementing the `proto.Marshaler` interface or having
299 malformed struct field tags that happened to work with one function, but not
300 another.
301
302* **Using Go reflection on any `proto.Message` types**: A common desire is to
303 write general-purpose code that operates on any protobuf message. For
304 example, a microservice might want to populate a `trace_id` field if it is
305 present in a message. To accomplish this, one would use Go reflection to
306 introspect the message type, and assume it were a pointer to a Go struct
307 with a field named `TraceId` (as would be commonly produced by
308 `protoc-gen-go`). If the concrete message type did not match this
309 expectation, it either failed to work or even resulted in a panic. Such was
310 the case for concrete message types that might be backed by a Go map instead
311 of a Go struct.
312
313Both of these issues are solved by following the idiom that _interfaces should
314describe behavior, not data_. This means that the interface itself should
315provide sufficient functionality through its methods that users can introspect
316and interact with all aspects of a protobuf message through a principled API.
317This feature is called _protobuf reflection_. Just as how Go reflection provides
318an API for programmatically interacting with any arbitrary Go value, protobuf
319reflection provides an API for programmatically interacting with any arbitrary
320protobuf message.
321
322Since an interface cannot be extended in a backwards compatible way, this
323suggested the need for a new major version that defines a new `proto.Message`
324interface:
325
326```go
327type Message interface {
328 ProtoReflect() protoreflect.Message
329}
330```
331
332The new
333[`proto.Message`](https://pkg.go.dev/google.golang.org/protobuf/proto?tab=doc#Message)
334interface contains a single `ProtoReflect` method that returns a
335[`protoreflect.Message`](https://pkg.go.dev/google.golang.org/protobuf/reflect/protoreflect?tab=doc#Message),
336which is a reflective view over a protobuf message. In addition to making a
337breaking change to the `proto.Message` interface, we took this opportunity to
338cleanup the supporting functionality that operate on a `proto.Message`, split up
339complicated functionality apart into manageable packages, and to hide
340implementation details away from the public API.
341
342The goal for this major revision is to improve upon all the benefits of, while
343addressing all the shortcomings of the old API. We hope that it will serve the
344Go ecosystem well for the next 10 years and beyond.
View as plain text