1# Package Documentation for github.com/ugorji/go/codec
2
3Package codec provides a High Performance, Feature-Rich Idiomatic Go 1.4+
4codec/encoding library for binc, msgpack, cbor, json.
5
6Supported Serialization formats are:
7
8 - msgpack: https://github.com/msgpack/msgpack
9 - binc: http://github.com/ugorji/binc
10 - cbor: http://cbor.io http://tools.ietf.org/html/rfc7049
11 - json: http://json.org http://tools.ietf.org/html/rfc7159
12 - simple:
13
14This package will carefully use 'package unsafe' for performance reasons
15in specific places. You can build without unsafe use by passing the safe or
16appengine tag i.e. 'go install -tags=codec.safe ...'.
17
18This library works with both the standard `gc` and the `gccgo` compilers.
19
20For detailed usage information, read the primer at
21http://ugorji.net/blog/go-codec-primer .
22
23The idiomatic Go support is as seen in other encoding packages in the standard
24library (ie json, xml, gob, etc).
25
26Rich Feature Set includes:
27
28 - Simple but extremely powerful and feature-rich API
29 - Support for go 1.4 and above, while selectively using newer APIs for later
30 releases
31 - Excellent code coverage ( > 90% )
32 - Very High Performance. Our extensive benchmarks show us outperforming Gob,
33 Json, Bson, etc by 2-4X.
34 - Careful selected use of 'unsafe' for targeted performance gains.
35 - 100% safe mode supported, where 'unsafe' is not used at all.
36 - Lock-free (sans mutex) concurrency for scaling to 100's of cores
37 - In-place updates during decode, with option to zero value in maps and slices
38 prior to decode
39 - Coerce types where appropriate e.g. decode an int in the stream into a
40 float, decode numbers from formatted strings, etc
41 - Corner Cases: Overflows, nil maps/slices, nil values in streams are handled
42 correctly
43 - Standard field renaming via tags
44 - Support for omitting empty fields during an encoding
45 - Encoding from any value and decoding into pointer to any value (struct,
46 slice, map, primitives, pointers, interface{}, etc)
47 - Extensions to support efficient encoding/decoding of any named types
48 - Support encoding.(Binary|Text)(M|Unm)arshaler interfaces
49 - Support using existence of `IsZero() bool` to determine if a value is a zero
50 value. Analogous to time.Time.IsZero() bool.
51 - Decoding without a schema (into a interface{}). Includes Options to
52 configure what specific map or slice type to use when decoding an encoded
53 list or map into a nil interface{}
54 - Mapping a non-interface type to an interface, so we can decode appropriately
55 into any interface type with a correctly configured non-interface value.
56 - Encode a struct as an array, and decode struct from an array in the data
57 stream
58 - Option to encode struct keys as numbers (instead of strings) (to support
59 structured streams with fields encoded as numeric codes)
60 - Comprehensive support for anonymous fields
61 - Fast (no-reflection) encoding/decoding of common maps and slices
62 - Code-generation for faster performance, supported in go 1.6+
63 - Support binary (e.g. messagepack, cbor) and text (e.g. json) formats
64 - Support indefinite-length formats to enable true streaming (for formats
65 which support it e.g. json, cbor)
66 - Support canonical encoding, where a value is ALWAYS encoded as same
67 sequence of bytes. This mostly applies to maps, where iteration order is
68 non-deterministic.
69 - NIL in data stream decoded as zero value
70 - Never silently skip data when decoding. User decides whether to return an
71 error or silently skip data when keys or indexes in the data stream do not
72 map to fields in the struct.
73 - Detect and error when encoding a cyclic reference (instead of stack overflow
74 shutdown)
75 - Encode/Decode from/to chan types (for iterative streaming support)
76 - Drop-in replacement for encoding/json. `json:` key in struct tag supported.
77 - Provides a RPC Server and Client Codec for net/rpc communication protocol.
78 - Handle unique idiosyncrasies of codecs e.g. For messagepack,
79 configure how ambiguities in handling raw bytes are resolved and provide
80 rpc server/client codec to support msgpack-rpc protocol defined at:
81 https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md
82
83# Extension Support
84
85Users can register a function to handle the encoding or decoding of their custom
86types.
87
88There are no restrictions on what the custom type can be. Some examples:
89
90```go
91 type BisSet []int
92 type BitSet64 uint64
93 type UUID string
94 type MyStructWithUnexportedFields struct { a int; b bool; c []int; }
95 type GifImage struct { ... }
96```
97
98As an illustration, MyStructWithUnexportedFields would normally be encoded as
99an empty map because it has no exported fields, while UUID would be encoded as a
100string. However, with extension support, you can encode any of these however you
101like.
102
103There is also seamless support provided for registering an extension (with a
104tag) but letting the encoding mechanism default to the standard way.
105
106# Custom Encoding and Decoding
107
108This package maintains symmetry in the encoding and decoding halfs. We determine
109how to encode or decode by walking this decision tree
110
111 - is there an extension registered for the type?
112 - is type a codec.Selfer?
113 - is format binary, and is type a encoding.BinaryMarshaler and
114 BinaryUnmarshaler?
115 - is format specifically json, and is type a encoding/json.Marshaler and
116 Unmarshaler?
117 - is format text-based, and type an encoding.TextMarshaler and
118 TextUnmarshaler?
119 - else we use a pair of functions based on the "kind" of the type e.g. map,
120 slice, int64, etc
121
122This symmetry is important to reduce chances of issues happening because the
123encoding and decoding sides are out of sync e.g. decoded via very specific
124encoding.TextUnmarshaler but encoded via kind-specific generalized mode.
125
126Consequently, if a type only defines one-half of the symmetry (e.g.
127it implements UnmarshalJSON() but not MarshalJSON() ), then that type doesn't
128satisfy the check and we will continue walking down the decision tree.
129
130# RPC
131
132RPC Client and Server Codecs are implemented, so the codecs can be used with the
133standard net/rpc package.
134
135# Usage
136
137The Handle is SAFE for concurrent READ, but NOT SAFE for concurrent
138modification.
139
140The Encoder and Decoder are NOT safe for concurrent use.
141
142Consequently, the usage model is basically:
143
144 - Create and initialize the Handle before any use. Once created, DO NOT modify
145 it.
146 - Multiple Encoders or Decoders can now use the Handle concurrently. They only
147 read information off the Handle (never write).
148 - However, each Encoder or Decoder MUST not be used concurrently
149 - To re-use an Encoder/Decoder, call Reset(...) on it first. This allows you
150 use state maintained on the Encoder/Decoder.
151
152Sample usage model:
153
154```go
155 // create and configure Handle
156 var (
157 bh codec.BincHandle
158 mh codec.MsgpackHandle
159 ch codec.CborHandle
160 )
161
162 mh.MapType = reflect.TypeOf(map[string]interface{}(nil))
163
164 // configure extensions
165 // e.g. for msgpack, define functions and enable Time support for tag 1
166 // mh.SetExt(reflect.TypeOf(time.Time{}), 1, myExt)
167
168 // create and use decoder/encoder
169 var (
170 r io.Reader
171 w io.Writer
172 b []byte
173 h = &bh // or mh to use msgpack
174 )
175
176 dec = codec.NewDecoder(r, h)
177 dec = codec.NewDecoderBytes(b, h)
178 err = dec.Decode(&v)
179
180 enc = codec.NewEncoder(w, h)
181 enc = codec.NewEncoderBytes(&b, h)
182 err = enc.Encode(v)
183
184 //RPC Server
185 go func() {
186 for {
187 conn, err := listener.Accept()
188 rpcCodec := codec.GoRpc.ServerCodec(conn, h)
189 //OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h)
190 rpc.ServeCodec(rpcCodec)
191 }
192 }()
193
194 //RPC Communication (client side)
195 conn, err = net.Dial("tcp", "localhost:5555")
196 rpcCodec := codec.GoRpc.ClientCodec(conn, h)
197 //OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h)
198 client := rpc.NewClientWithCodec(rpcCodec)
199```
200
201# Running Tests
202
203To run tests, use the following:
204
205```
206 go test
207```
208
209To run the full suite of tests, use the following:
210
211```
212 go test -tags alltests -run Suite
213```
214
215You can run the tag 'codec.safe' to run tests or build in safe mode. e.g.
216
217```
218 go test -tags codec.safe -run Json
219 go test -tags "alltests codec.safe" -run Suite
220```
221
222# Running Benchmarks
223
224```
225 cd bench
226 go test -bench . -benchmem -benchtime 1s
227```
228
229Please see http://github.com/ugorji/go-codec-bench .
230
231# Caveats
232
233Struct fields matching the following are ignored during encoding and decoding
234
235 - struct tag value set to -
236 - func, complex numbers, unsafe pointers
237 - unexported and not embedded
238 - unexported and embedded and not struct kind
239 - unexported and embedded pointers (from go1.10)
240
241Every other field in a struct will be encoded/decoded.
242
243Embedded fields are encoded as if they exist in the top-level struct, with some
244caveats. See Encode documentation.
245
246## Exported Package API
247
248```go
249const CborStreamBytes byte = 0x5f ...
250const GenVersion = 28
251var SelfExt = &extFailWrapper{}
252var GoRpc goRpc
253var MsgpackSpecRpc msgpackSpecRpc
254func GenHelper() (g genHelper)
255type BasicHandle struct{ ... }
256type BincHandle struct{ ... }
257type BytesExt interface{ ... }
258type CborHandle struct{ ... }
259type DecodeOptions struct{ ... }
260type Decoder struct{ ... }
261 func NewDecoder(r io.Reader, h Handle) *Decoder
262 func NewDecoderBytes(in []byte, h Handle) *Decoder
263 func NewDecoderString(s string, h Handle) *Decoder
264type EncodeOptions struct{ ... }
265type Encoder struct{ ... }
266 func NewEncoder(w io.Writer, h Handle) *Encoder
267 func NewEncoderBytes(out *[]byte, h Handle) *Encoder
268type Ext interface{ ... }
269type Handle interface{ ... }
270type InterfaceExt interface{ ... }
271type JsonHandle struct{ ... }
272type MapBySlice interface{ ... }
273type MissingFielder interface{ ... }
274type MsgpackHandle struct{ ... }
275type MsgpackSpecRpcMultiArgs []interface{}
276type RPCOptions struct{ ... }
277type Raw []byte
278type RawExt struct{ ... }
279type Rpc interface{ ... }
280type Selfer interface{ ... }
281type SimpleHandle struct{ ... }
282type TypeInfos struct{ ... }
283 func NewTypeInfos(tags []string) *TypeInfos
284```
View as plain text