Golang JSON Serialization With Interfaces

I've been at Uber since September of last year, doing almost exclusively back-end work in Python. As of recently, Uber is moving more towards building new services in Go and Java. Even though my team is still using Python exclusively (for now), I thought it was time to get my feet wet with Go.

There are a number of great resources for getting introduced to Go, so I won't give an introduction here. What I will cover, however, is an interesting JSON serialization and deserialization example. JSON serialization / deserialization in Python and JavaScript is easy, but since Go is statically typed, it can be a bit more tricky.

Here are some starter resources on JSON serialization in Golang:

• The json package documentation
• A basic overview of json / golang
• Go By Example: JSON

And here's the example I'll cover:

You have a ColorfulEcosystem, filled with Plants and Animals, both of which implement the ColoredThing interface. All plants and animals are stored in the same place: in a slice of ColoredThings.

Also, the only thing our code will store about Plants and Animals is their color. So we will need to be clever about our JSON serialization / deserialization to make it easy to encode and decode our ecosystem's contents.

The struct and interface definitions:

type ColorfulEcosystem struct {  
    Things []ColoredThing

}

# Both Plant and Animal implement this interface
type ColoredThing interface {  
    Color() string
}

type Plant struct {  
    MyColor string
}

type Animal struct {  
    MyColor string
}

// Necessary for Plant to implement the ColoredThing interface
func (p *Plant) Color() string {  
    return p.MyColor
}

// Necessary for Animal to implement the ColoredThing interface
func (a *Animal) Color() string {  
    return a.MyColor
}

Here are a couple of reasons it may be challenging to serialize and deserialize a ColorfulEcosystem struct:

• it contains a slice of ColoredThings, which is an interface.
• the ColoredThing interface will be implemented by two different structs, both of which have the same underlying properties.

Serializing / Deserializing a simple struct

Before we jump into our example, let's start with something simpler: If our task was to serialize / deserialize a single, basic struct, that's actually pretty easy. Let's take a look at how this works with Plant right now (or you can check out the intro links at the top of the page):

package main

import (  
    "encoding/json"
    "fmt"
)

type Plant struct {  
    MyColor string
}

func main() {  
    p := Plant{MyColor: "green"}
    // json.Marshal returns a byte slice and an error (if any)
    byteSlice, _ := json.Marshal(p)

    // Prints out '{"MyColor":"green"}'
    fmt.Println(string(byteSlice))

    // Now let's deserialize it...

    newP := Plant{}

    // json.Unmarshal returns an error, if any (which we aren't even bothering to check)
    json.Unmarshal(byteSlice, &newP)

    // Prints "green"
    fmt.Println(newP.MyColor)
}

Boom. Pretty straightforward -- the json package handles it all for us.

Some things to keep in mind:

• Only exported (capitalized) fields will be automatically serialized / deserialized! So, if we used myColor instead of MyColor, the above would not work.
• Let's say we wanted to have this serialize into a slightly different byte slice (eg. {"color":"green"} instead of {"MyColor":"green"}). To change the output name for a deserialized field, you can simply add a "tag" after the property in your struct, like so:

type Plant struct {  
    MyColor string `json:"color"`
}

Note: Don't forget the quotes around the field name!

But what about Animal? Won't it serialize to the same thing as Plant?

Indeed it will! If we have them both defined as:

type Plant struct {  
    MyColor string `json:"color"`
}

type Animal struct {  
    MyColor string `json:"color"`
}

Then they will both serialize into something that looks like {"color":"green"} or {"color":"red"}, etc. Impossible to tell apart...

Let's make the deserialized strings more specific to their structs

Ok, to fix this, let's implement some methods on our structs that will make the serialization a bit more specific.

Specifically, we're going to define a custom MarshalJSON method on Plant, which indicates to json.Marshal that it should serialize the JSON in a certain way. Let's check it out:

func (p *Plant) MarshalJSON() ([]byte, error) {  
    m := make(map[string]string)
    m["type"] = "plant"
    m["color"] = p.MyColor
    return json.Marshal(m)
}

In the above method, we now create a blank "map" instance under the hood, add a type variable and a color variable to it, then tell json to marshal and return the map, instead.

This allows us to see the type of the object in addition to the color:

package main

import (  
    "encoding/json"
    "fmt"
)

type Plant struct {  
    MyColor string
}

func (p *Plant) MarshalJSON() ([]byte, error) {  
    m := make(map[string]string)
    m["type"] = "plant"
    m["color"] = p.MyColor
    return json.Marshal(m)
}

func main() {  
    p := Plant{MyColor: "green"}
    // Note that we now use a pointer to `p` instead of `p`
    // This is because our MarshalJSON method, above, is on the
    // pointer, not on the struct itself.
    // So, for `json.Marshal` to "see" the MarshalJSON method,
    // we must pass `json.Marshal` the pointer.
    byteSlice, _ := json.Marshal(&p)


    // Prints out '{"color":"green","type":"plant"}'
    fmt.Println(string(byteSlice))

    // Now let's deserialize it...

    newP := Plant{}

    json.Unmarshal(byteSlice, &newP)

    // Prints "green"
    fmt.Println(newP.MyColor)
}

Some things to note:

• We had to change to passing a Plant pointer to the json.Marshal method above. The reason why is explained in the code comment. (Try it out with and without the pointer to see what happens!)
• We didn't have to write any custom deserialization logic -- since type is not a property on the struct, json.Unmarshal just ignores it and uses only the field it recognizes (color)
• Obviously, this would be easier if we just added an exported "Type" field to the struct definition, but that wouldn't be as educational.

Ok, let's serialize an entire ColorfulEcosystem

Now that we know how to write code that will serialize Plant and Animal differently, let's try serializing an entire ColorfulEcosystem object.

package main

import "encoding/json"  
import "fmt"

type ColorfulEcosystem struct {  
    Things []ColoredThing `json:"things"`
}

type ColoredThing interface {  
    Color() string
}

type Plant struct {  
    MyColor string `json:"color"`
}

type Animal struct {  
    MyColor string `json:"color"`
}

func (p *Plant) Color() string {  
    return p.MyColor
}
func (p *Plant) MarshalJSON() (b []byte, e error) {  
    return json.Marshal(map[string]string{
        "type":  "plant",
        "color": p.Color(),
    })
}

func (a *Animal) Color() string {  
    return a.MyColor
}
func (a *Animal) MarshalJSON() (b []byte, e error) {  
    return json.Marshal(map[string]string{
        "type":  "animal",
        "color": a.Color(),
    })
}

func main() {  
    // First let's create some things to live in the ecosystem
    fern := &Plant{MyColor: "green"}
    flower := &Plant{MyColor: "purple"}

    panther := &Animal{MyColor: "black"}
    lizard := &Animal{MyColor: "green"}

    // Then let's create a ColorfulEcosystem
    colorfulEcosystem := ColorfulEcosystem{
        Things: []ColoredThing{
            fern,
            flower,
            panther,
            lizard,
        },
    }

    // prints out:
    // {"things":[{"color":"green","type":"plant"},{"color":"purple","type":"plant"},{"color":"black","type":"animal"},{"color":"green","type":"animal"}]}
    byteSlice, _ := json.Marshal(colorfulEcosystem)
    fmt.Println(string(byteSlice))
}

Now for the tricky bit: let's deserialize from JSON back into a ColorfulEcosystem struct

We just learned how to serialize our complex object into a json byte string. However, how do we make this work for deserialization (going from JSON back to a struct)?

If we just tried to use json.Unmarshal on ColorfulEcosystem right now, it wouldn't work, because we need to know what type of struct to use for each item. We are outputting the "type" fields in the JSON, but there is no code yet that tells Golang how to interpret the "type" field when de-serializing (not to mention, ColorfulEcosystem stores all Plant and Animal together as a slice of ColoredThings)

So, what's the solution? In the code examples above, we've already seen how to define custom serialization behavior using MarshalJSON, now let's define some custom deserialization behavior using UnmarshalJSON.

Defining a custom UnmarshalJSON method on ColorfulEcosystem

We have a fairly complex structure for our ColorfulEcosystem, so the easiest thing to do here is to use another part of the json package to break it down bit by bit.

For this, we will take advantage of the json.RawMessage type. From the json package documentation, we see that RawMessage "can be used to delay JSON decoding" (this stack overflow answer also gives a good example).

So, we will use RawMessage to only partially deserialize our JSON, so we can add custom processing at each step:

Step 1: Get a RawMessage for the things key

func (ce *ColorfulEcosystem) UnmarshalJSON(b []byte) error {  
    var objMap map[string]*json.RawMessage
    // We'll store the error (if any) so we can return it if necessary
    err := json.Unmarshal(b, &objMap)
    if err != nil {
        return err
    }
    fmt.Println(objMap)

    // More to come here ...
}

This first step will switch our raw JSON into a map, so this will print out something along the lines of: map[things:0xc82000e700]. "things" is the only key on the top level of the JSON, and objMap["things"] now points to the JSON-serialized slice of Plants and Animals.

Boom. By using RawMessage, we're now one step closer to unraveling the JSON and creating a ColorfulEcosystem.

• Why are we using *json.RawMessage instead of just json.RawMessage in the second line?
  • Great question! The reason is that if we look at the documentation, we see that json.MarshalJSON and json.UnmarshalJSON have pointer receivers, which roughly means that these methods will only be "visible" to Go if we make objMap values be pointers to RawMessage.

Step 2: Get a RawMessage for each ColoredThing

Let's keep going...

func (ce *ColorfulEcosystem) UnmarshalJSON(b []byte) error {  
    var objMap map[string]*json.RawMessage
    err := json.Unmarshal(b, &objMap)
    if err != nil {
        return err
    }

    var rawMessagesForColoredThings []*json.RawMessage
    err = json.Unmarshal(*objMap["things"], &rawMessagesForColoredThings)
    if err != nil {
        return err
    }
    fmt.Println(rawMessagesForColoredThings)

    // More to come here ...
}

Now we're one step closer: we have a slice of RawMessages now, one for each of our ColoredThing objects in our JSON.

• Wait, why did we use *objMap["things"] instead of just objMap["things"]?!
  • Remember above how we used *json.RawMessage? Well, that meant that all the values in our objMap were pointers to json.RawMessage.
  • json.Unmarshal expects an actual slice of bytes (or a RawMessage), not a pointer to it, so we have to de-reference the pointer and pass in the value instead of the pointer
  • Note that this principle will continue to apply in the examples below...

Step 3: De-serialize each ColoredThing into the appropriate underlying struct

Nearly there! Now let's parse the RawMessages in the slice:

func (ce *ColorfulEcosystem) UnmarshalJSON(b []byte) error {  
    // First, deserialize everything into a map of map
    var objMap map[string]*json.RawMessage
    err := json.Unmarshal(b, &objMap)
    if err != nil {
        return err
    }

    var rawMessagesForColoredThings []*json.RawMessage
    err = json.Unmarshal(*objMap["things"], &rawMessagesForColoredThings)
    if err != nil {
        return err
    }

    var m map[string]string
    for _, rawMessage := range rawMessagesForColoredThings {
        err = json.Unmarshal(*rawMessage, &m)
        if err != nil {
            return err
        }
        fmt.Println(m)
    }

    // More to come here ...
}

This prints out:

map[color:green type:plant]  
map[color:purple type:plant]  
map[color:black type:animal]  
map[color:green type:animal]  

Woooo! Now we're seeing the raw data that will make up each of our Plants and Animals. Almost done!

Step 4: Based on the "type" field, deserialize into the appropriate structs

At this point, we have a choice. Either we can create the Plant and Animal structs manually, or we can let json.Unmarshal create them for us.

Let's do the second option, otherwise we'll be re-implementing logic for de-serializing Plants and Animals. From our first simple Plant example at the beginning of this post, we already know that Plant knows how to deserialize itself using json.Unmarshal.

So, now that we know what "type" of struct to use, let's create and store all of our coloredThings.

func (ce *ColorfulEcosystem) UnmarshalJSON(b []byte) error {  
    // First, deserialize everything into a map of map
    var objMap map[string]*json.RawMessage
    err := json.Unmarshal(b, &objMap)
    if err != nil {
        return err
    }

    var rawMessagesForColoredThings []*json.RawMessage
    err = json.Unmarshal(*objMap["things"], &rawMessagesForColoredThings)
    if err != nil {
        return err
    }

    // Let's add a place to store our de-serialized Plant and Animal structs
    ce.Things = make([]ColoredThing, len(rawMessagesForColoredThings))

    var m map[string]string
    for index, rawMessage := range rawMessagesForColoredThings {
        err = json.Unmarshal(*rawMessage, &m)
        if err != nil {
            return err
        }

        // Depending on the type, we can run json.Unmarshal again on the same byte slice
        // But this time, we'll pass in the appropriate struct instead of a map
        if m["type"] == "plant" {
            var p Plant
            err := json.Unmarshal(*rawMessage, &p)
            if err != nil {
                return err
            }
            // After creating our struct, we should save it
            ce.Things[index] = &p
        } else if m["type"] == "animal" {
            var a Animal
            err := json.Unmarshal(*rawMessage, &a)
            if err != nil {
                return err
            }
            // After creating our struct, we should save it
            ce.Things[index] = &a
        } else {
            return errors.New("Unsupported type found!")
        }
    }

    // That's it!  We made it the whole way with no errors, so we can return `nil`
    return nil
}

Ok, let's break this down:

• ce was the ColorfulEcosystem pointer receiver, so this was the place that we were trying to store our JSON data
• The first change we made from the above is to initialize ce.Things to be a slice of ColoredThings, and we gave it the correct length (since we knew the length of RawMessageForColoredThingss`)
• From our work in Step 3, we had access to the "type" of each struct. With that, we were able to use json.Unmarshal on the same json.RawMessage, but this time we put it into the correct struct instead of into a map.
• Once we finished this all up, we could return nil, indicating that the method finished with no errors.
  • (Note that if the format didn't match what we were expecting, there would have been an error returned earlier in the function)

You made it!

Hope that was helpful! Please leave any comments or questions below.

For those interested, here's the full code for serializing and deserializing a ColorfulEcosystem.

package main

import "encoding/json"  
import (  
    "fmt"
    "errors"
)

type ColorfulEcosystem struct {  
    Things []ColoredThing `json:"things"`
}

type ColoredThing interface {  
    Color() string
}

type Plant struct {  
    MyColor string `json:"color"`
}

type Animal struct {  
    MyColor string `json:"color"`
}

func (p *Plant) Color() string {  
    return p.MyColor
}
func (p *Plant) MarshalJSON() (b []byte, e error) {  
    return json.Marshal(map[string]string{
        "type":  "plant",
        "color": p.Color(),
    })
}

func (a *Animal) Color() string {  
    return a.MyColor
}
func (a *Animal) MarshalJSON() (b []byte, e error) {  
    return json.Marshal(map[string]string{
        "type":  "animal",
        "color": a.Color(),
    })
}

func (ce *ColorfulEcosystem) UnmarshalJSON(b []byte) error {  
    var objMap map[string]*json.RawMessage
    err := json.Unmarshal(b, &objMap)
    if err != nil {
        return err
    }

    var rawMessagesForColoredThings []*json.RawMessage
    err = json.Unmarshal(*objMap["things"], &rawMessagesForColoredThings)
    if err != nil {
        return err
    }

    // Let's add a place to store our de-serialized Plant and Animal structs
    ce.Things = make([]ColoredThing, len(rawMessagesForColoredThings))

    var m map[string]string
    for index, rawMessage := range rawMessagesForColoredThings {
        err = json.Unmarshal(*rawMessage, &m)
        if err != nil {
            return err
        }

        // Depending on the type, we can run json.Unmarshal again on the same byte slice
        // But this time, we'll pass in the appropriate struct instead of a map
        if m["type"] == "plant" {
            var p Plant
            err := json.Unmarshal(*rawMessage, &p)
            if err != nil {
                return err
            }
            // After creating our struct, we should save it
            ce.Things[index] = &p
        } else if m["type"] == "animal" {
            var a Animal
            err := json.Unmarshal(*rawMessage, &a)
            if err != nil {
                return err
            }
            // After creating our struct, we should save it
            ce.Things[index] = &a
        } else {
            return errors.New("Unsupported type found!")
        }
    }

    // That's it!  We made it the whole way with no errors, so we can return `nil`
    return nil
}

func main() {  
    // First let's create some things to live in the ecosystem
    fern := &Plant{MyColor: "green"}
    flower := &Plant{MyColor: "purple"}

    panther := &Animal{MyColor: "black"}
    lizard := &Animal{MyColor: "green"}

    // Then let's create a ColorfulEcosystem
    colorfulEcosystem := ColorfulEcosystem{
        Things: []ColoredThing{
            fern,
            flower,
            panther,
            lizard,
        },
    }

    // prints:
    // {"things":[{"color":"green","type":"plant"},{"color":"purple","type":"plant"},{"color":"black","type":"animal"},{"color":"green","type":"animal"}]}
    byteSlice, _ := json.Marshal(colorfulEcosystem)
    fmt.Println(string(byteSlice))

    // Now let's try deserializing the JSON back to a new struct
    newCE := ColorfulEcosystem{}
    err := json.Unmarshal(byteSlice, &newCE)
    if err != nil {
        panic(err)
    }

    for _, colorfulThing := range newCE.Things {
        fmt.Println(colorfulThing.Color())
    }

}

Tag