What Could Not Be Fixed

Originally published in Japanese at https://zenn.dev/ymotongpoo/books/go-json-v2-history/viewer/10-unfixable.

The first issue

On March 10, 2016, an issue titled “encoding/json: parser ignores the case of member names” was filed. It pointed out that the JSON parser ignores the case of member names.

The same day, Russ Cox replied.

This has been the behavior at least as far back as Go 1.2 … The docs also seem to state quite clearly that this is what happens … I understand there are security implications if JSON is used in security contexts, and I was a little surprised too, but the docs are very clear[.]

The decision was to change nothing, on the grounds that “it’s in the documentation.” Why that was decisive becomes clear once you look at what Go’s backward compatibility guarantee actually says.

What the backward compatibility guarantee says

Go’s backward compatibility promise boils down to a single sentence.

It is intended that programs written to the Go 1 specification will continue to compile and run correctly, unchanged, over the lifetime of that specification.

This is the foundation of the Go project.

There are exceptions, though. Security problems, unspecified behavior, errors in the specification itself, obvious bugs—the document explicitly states that these are subject to change.

Case-insensitive matching fits none of them. Since it is documented, it is not “unspecified behavior.” Since it is explicitly stated, it is not a “bug” either. Security concerns were raised, but the behavior was framed as a dangerous default rather than a vulnerability in itself.

In other words, this behavior fell through the net of exceptions. Fixing it would break programs that depend on it. However undesirable it was, the promise of backward compatibility took priority.

Deferred to Go 2

The same judgment was repeated across other issues.

golang/go#4712 is an issue about the JSON representation of time.Duration. encoding/json emits time.Duration as an integer number of nanoseconds. 90 * time.Second becomes 90000000000. The unit is written nowhere, so the receiving side has no choice but to count digits.

On February 17, 2017, the issue was closed. Here is Russ Cox’s comment.

If you want a custom duration marshaling, define a type that implements json.Marshaler/json.Unmarshaler. At this point we’re not going to change this fundamental detail of the json package.

Brad Fitzpatrick added briefly.

Everything will be considered anew for any Go 2.

As of 2017, this class of problem was filed away as “something today’s Go cannot handle,” entrusted to some future thing.

An attempted fix, and its abandonment

The story does not end with “the Go team didn’t fix it.” Someone tried.

On February 26, 2020, Daniel Martí (mvdan), a maintainer of encoding/json, posted to the issue.

I’ve come to realise that pretty much all of my previous comments in this thread were wrong :) … I do think that many parts of the json package could be designed better, and I think the edge cases concerning missing, repeated, or case-insensitive-matching keys are some of them.

And he actually wrote a fix. From his March 19 comment:

Decoding structs is ~1% slower, but we get the benefit we want. … 1% performance loss is unfortunate, but I can’t figure out a way around it.

There was a working patch, and the performance cost had been measured down to a concrete figure: 1%. Even so, the problem of breaking programs that depended on the existing behavior remained. Imposing a 1% regression on all users, and on top of that breaking some users’ code—the proposal was not approved.

The patch from that effort, CL224079, was abandoned unmerged in 2024.

This was the turning point. “Can’t be fixed” was not resignation; it was a conclusion reached after writing the code, running it, and measuring it. And in the latter half of the year that conclusion was reached, the first drafts of v2 began to be written.

The behavioral difference between v1 and v2

In go1.27, both encoding/json and encoding/json/v2 are available. Let’s feed the same input to both.

• https://go.dev/play/p/vtzRCS8x2iP

package main

import (
	jsonv1 "encoding/json"
	jsonv2 "encoding/json/v2"
	"fmt"
)

type User struct {
	Name string `json:"name"`
}

func main() {
	in := []byte(`{"NAME":"gopher"}`)

	var a User
	err1 := jsonv1.Unmarshal(in, &a)
	fmt.Printf("v1: %+v  err=%v\n", a, err1)

	var b User
	err2 := jsonv2.Unmarshal(in, &b)
	fmt.Printf("v2: %+v  err=%v\n", b, err2)
}

v1: {Name:gopher}  err=<nil>
v2: {Name:}  err=<nil>

v2 does not match. Note that v2 is not returning an error here. NAME is simply ignored as an unknown member name, and the Name field is left empty. If you want an error, specify RejectUnknownMembers.

Duplicate keys and invalid UTF-8

Case sensitivity is not the whole story. Let’s also look at duplicate keys and invalid UTF-8.

• https://go.dev/play/p/hKjc4kyhrvF

dup := []byte(`{"name":"alice","role":"user","role":"admin"}`)
bad := []byte("{\"name\":\"go\xffpher\"}")

dup is JSON in which the key role appears twice. bad contains the byte 0xff, invalid as UTF-8, inside a string.

v1 dup: {Name:alice Role:admin}  err=<nil>
v2 dup: err=jsontext: duplicate object member name "role"
v1 utf8: err=<nil> -> "go�pher"
v2 utf8: err=jsontext: invalid UTF-8 within "/name" after offset 11

v1 accepts both without an error. Duplicate keys are overwritten last-wins, and invalid UTF-8 is replaced with the Unicode replacement character. The caller has no way of knowing the input was broken.

Duplicate keys become a problem when more than one party reads the JSON. Think of an authentication proxy that sees "role":"user" and lets the request through, while the application behind it reads "role":"admin". Even if both follow the same specification, if one is first-wins and the other last-wins, their decisions diverge. Discussion #63397, opened in 2023, describes this as something that “can be exploited by attackers and has been exploited in the past with severe consequences.”

The v2 error messages above begin with jsontext:. They come from a separate package, encoding/json/jsontext. In v2, the layer that handles JSON syntax is split from the layer that maps JSON to and from Go values. Both duplicate keys and invalid UTF-8 are rejected at the syntax stage, before Go types are ever touched. So why was the layer that reads and writes JSON carved out into its own package? Performance seems like the natural guess, but that was not the reason.

The four categories of v1’s problems

Discussion #63397 opens by sorting v1’s problems into four categories. With the issue numbers of the time attached, it doubles as a useful record of the 14 years.

Listed as missing functionality:

• A way to specify the format of time.Time (#21990)
• A way to omit particular values from the output (#22480, #50480, and others)
• A way to emit nil slices and maps as [] and {} instead of null (#37711, #27589)
• An inline tag to flatten a struct without using embedding (#6213)

Listed as API deficiencies:

• json.NewDecoder(r).Decode(v) silently succeeds even when garbage remains at the end of the input (#36225)
• Options cannot be passed to Marshal or Unmarshal, and thus cannot reach deep into nested types (#41144)
• Compact, Indent, and HTMLEscape write only to a *bytes.Buffer; you cannot pass a []byte or an io.Writer

The performance limits are rooted in the design. MarshalJSON returns a []byte, so every implementation necessarily allocates a byte slice. The caller then re-parses the returned bytes to validate them and re-indent them. UnmarshalJSON is worse: because it must be handed a complete value, the entire value is parsed before the call, then parsed again inside the method. When nested types each have their own UnmarshalJSON, this double work multiplies with every level of nesting. The discussion cites a case where this became a real problem loading Kubernetes’ OpenAPI specification. This category also has five items; the remaining three concern the lack of a streaming API. The fact that Encoder and Decoder accept an io.Writer or io.Reader yet still buffer the entire value in memory is among them (#33714 and others).

And then there were the behavioral flaws. Five items are listed; this book covers the following three.

• Accepting invalid UTF-8 and accepting duplicate keys (#43664)
• Case-insensitive matching (#14750)
• MarshalJSON being called or not called depending on whether the value is addressable (#22967 and others)

The last item can be reproduced with a short program.

• https://go.dev/play/p/PDbqd1aQCcC

package main

import (
	jsonv1 "encoding/json"
	jsonv2 "encoding/json/v2"
	"fmt"
	"strings"
)

type Tag struct {
	Name string
}

// Defined with a pointer receiver
func (t *Tag) MarshalJSON() ([]byte, error) {
	return []byte(`"` + strings.ToUpper(t.Name) + `"`), nil
}

func main() {
	slice := []Tag{{Name: "go"}}              // elements are addressable
	m := map[string]Tag{"lang": {Name: "go"}} // values are not addressable

	b1, _ := jsonv1.Marshal(slice)
	b2, _ := jsonv1.Marshal(m)
	fmt.Printf("v1 スライス: %s\n", b1)
	fmt.Printf("v1 マップ  : %s\n", b2)

	b3, _ := jsonv2.Marshal(slice)
	b4, _ := jsonv2.Marshal(m)
	fmt.Printf("v2 スライス: %s\n", b3)
	fmt.Printf("v2 マップ  : %s\n", b4)
}

v1 スライス: ["GO"]
v1 マップ  : {"lang":{"Name":"go"}}
v2 スライス: ["GO"]
v2 マップ  : {"lang":"GO"}

In v1, if the value is a slice element, MarshalJSON is called and you get "GO"; if it is a map value, the method is not called and it falls back to the default struct representation. The same value of the same type produces different results depending on where it sits. v2 calls the method in both cases.

This last item comes with a caveat worth noting.

This could arguably be considered a bug and be fixed in the current ‘json’ package. However, previous attempts at fixing this resulted in the changes being reverted because it broke too many targets implicitly depending on the inconsistent calling behavior.

Even a “bug” that fell within the exception clause had become unfixable once dependencies piled up on it.

These behavioral flaws of ‘json’ cannot be changed without being a breaking change. Options could be added to specify different behavior, but that would be unfortunate since the desired behavior is not the default behavior. Changing the default behavior suggests the need for a v2 ‘json’ package.