json.Encoder/json.Decoder: streaming JSON processing

1. Why Encoder/Decoder are needed

If you’ve previously worked with json.Marshal, the feeling is usually: “I get []byte, and then I do whatever I want.” And that’s true… until you try to process large input, work with stdin, write output to stdout, or (the funniest part) stop copying data in circles “string → bytes → string → bytes”. At that point you suddenly want JSON to be readable and writable directly “into the pipe”.

The main idea: Marshal/Unmarshal is about “everything into memory”, while Encoder/Decoder is about “working on top of a stream”.

Imagine that your input is not one nice JSON document, but many small objects one after another (for example, tasks one per line). The “read everything into []byte” option forces you either to build a huge buffer or to invent manual parsing. But json.Decoder can read values one at a time.

Quick refresher: io.Reader and io.Writer as “input” and “output”

It’s easy to get confused by io.Reader/io.Writer if you treat them as “something abstract from a textbook”. In practice, it’s a simple idea:

• io.Reader — something you can read bytes from
• io.Writer — something you can write bytes to

A file, network, memory, stdin/stdout — all of these can be unified under the same interface.

A very useful effect: if your code accepts an io.Reader, you can feed it os.Stdin, strings.NewReader(...), and bytes.Buffer. That means you can test and debug the logic without files and without “the outside world”.

In this lecture, we’ll evolve a learning app for tasks (a conditional “task manager”). We’ll add “export tasks to JSON” and “import tasks from JSON” functions so that they work with any data source/sink, not just files or strings.

2. json.Encoder: write JSON straight to a stream

json.NewEncoder(w) and Encoder.Encode(v)

When you write JSON to a stream, you want it to be as “non-magical” as possible: take an io.Writer, create an encoder, and say “Encode this value”. From an API point of view, it’s almost like Marshal, except that instead of []byte the result is written straight to the Writer.

There are two details that beginners usually don’t notice right away:

• Encode writes one JSON value per call (object, array, number, string — any valid JSON value).
• Encode usually adds a newline \n at the end — this is convenient for the “one object per line” format.

Let’s start with a small step: learn how to write one task.

package main

import (
	"encoding/json"
	"io"
)

type Task struct {
	ID    int    `json:"id"`
	Title string `json:"title"`
	Done  bool   `json:"done"`
}

func writeTask(w io.Writer, t Task) error {
	enc := json.NewEncoder(w)
	return enc.Encode(t)
}

Notice: no []byte, no string(...), and no back-and-forth conversions. We simply write to w, and what exactly w is (a file, stdout, a buffer) is decided by the calling code.

Encoder.SetIndent for readable JSON

In a real application, you sometimes need two modes: “machine” JSON without extra spaces (compact) and “human” JSON for debugging or exporting to a user (pretty, indented). Encoder has a SetIndent(prefix, indent) method that configures formatting.

It’s important to catch the style: formatting is an encoder setting, not a separate “magic function somewhere nearby”. We create an encoder, configure it, and then write.

package main

import (
	"encoding/json"
	"io"
)

func writePrettyTask(w io.Writer, t Task) error {
	enc := json.NewEncoder(w)
	enc.SetIndent("", "  ")
	return enc.Encode(t)
}

Yes, the JSON will be longer now, but at least you can read it with your eyes without pain.

Exporting tasks: as an array and as a stream

If you export a list of tasks, you have two practical options:

• export the entire list as one JSON array
• export as a stream of objects (multiple Encode calls in a row)

Here’s how this might look in the taskio layer.

package taskio

import (
	"encoding/json"
	"io"
)

type Task struct {
	ID    int    `json:"id"`
	Title string `json:"title"`
	Done  bool   `json:"done"`
}

func WriteTasksAsArray(w io.Writer, tasks []Task) error {
	enc := json.NewEncoder(w)
	enc.SetIndent("", "  ")
	return enc.Encode(tasks)
}

func WriteTasksAsStream(w io.Writer, tasks []Task) error {
	enc := json.NewEncoder(w)
	for _, t := range tasks {
		if err := enc.Encode(t); err != nil {
			return err
		}
	}
	return nil
}

3. json.Decoder: read JSON from a stream

json.NewDecoder(r) and Decoder.Decode(&v)

The decoder is the “other half of the heart”. It reads from an io.Reader and fills your variable. The most important thing here is not to forget what we already reinforced: decoding requires a pointer, because the function must write the result somewhere.

One more point: Decode also works “one value at a time”. Call it once — you read one JSON value. Call it a second time — you attempt to read the next one. That’s how you get the ability to read a “stream of values”.

package main

import (
	"encoding/json"
	"io"
)

func readTask(r io.Reader) (Task, error) {
	dec := json.NewDecoder(r)

	var t Task
	if err := dec.Decode(&t); err != nil {
		return Task{}, err
	}
	return t, nil
}

If you accidentally write dec.Decode(t) (without &), the compiler or runtime will quickly explain that you’re trying to fill a copy. It’s like trying to pour tea into a photo of a mug: the mug looks nice, but the tea ends up on the table.

A stream of JSON values: read until io.EOF

A stream of values usually looks like this:

{"id":1,"title":"Read Go book","done":false}
{"id":2,"title":"Buy milk","done":true}
{"id":3,"title":"Sleep","done":false}

Each line is a valid JSON object. The entire file as a whole is not a valid JSON document, because JSON does not allow “three objects in a row” without a wrapper. But for Decoder this is fine, because it reads one value at a time.

The reading pattern is always the same: run a loop, call Decode(&t), and stop on io.EOF.

package main

import (
	"encoding/json"
	"io"
)

func readTaskStream(r io.Reader) ([]Task, error) {
	dec := json.NewDecoder(r)

	var out []Task
	for {
		var t Task
		err := dec.Decode(&t)
		if err == io.EOF {
			break
		}
		if err != nil {
			return nil, err
		}
		out = append(out, t)
	}
	return out, nil
}

This code is simple, but very powerful: it works the same way with strings.NewReader(...), with stdin, with a network connection, and with a file.

To make it click visually, you can imagine the streaming flow like this:

flowchart TD
    A["Decoder.Decode(&t)"] -->|success| B["append(out, t)"]
    B --> A
    A -->|err == io.EOF| C[end of stream: return out]
    A -->|err != nil| D[error: return nil, err]

JSON array: read as one value

A JSON array looks like this:

[
  {"id":1,"title":"Read Go book","done":false},
  {"id":2,"title":"Buy milk","done":true}
]

This is one valid JSON document, and it’s convenient to send/store as a “full export”. You don’t need a loop over Decode here. You do one Decode(&tasks) and you’re done.

package main

import (
	"encoding/json"
	"io"
)

func readTaskArray(r io.Reader) ([]Task, error) {
	dec := json.NewDecoder(r)

	var tasks []Task
	if err := dec.Decode(&tasks); err != nil {
		return nil, err
	}
	return tasks, nil
}

4. Two data protocols: stream vs array

What “one JSON value” means

The phrase “one JSON value” may sound like something esoteric, but the meaning is very down-to-earth. In JSON, a value is not only an object {...}. A JSON value can be:

• an object {...}
• an array [...]
• a string "hi"
• a number 123
• a boolean true or false
• null

Why is this important? Because the difference between an “array” and a “stream of objects” often looks the same to a human (“there are many tasks”), but for Decoder these are two different protocols:

• if the input is an array, that is one value, and you read it with a single Decode(&tasks)
• if the input is multiple objects in a row, that is multiple values, and you read them in a Decode loop until io.EOF

To lock it in, keep this thought in mind: Decode does not read “everything that looks like JSON”; it reads exactly one value and stops.

Format comparison

Choosing between an array and a stream of values is not “right/wrong”; it’s a question of protocol and convenience. An array is convenient when you work “all at once”. A stream is convenient when there’s a lot of data or you want to process it as it arrives.

The most important thing to fix in your mind: you must agree in advance what format your code expects. If your import expects an array but you feed it a stream of objects, Decode will honestly read the first object… and then be surprised that the “array” suddenly ends in a strange way.

5. Integration into the app: export/import via io.Writer/io.Reader

When we add a feature to a learning app, we want it to be not a “disconnected example” but a small building block. That’s why we define the taskio layer, which can write/read tasks in both formats.

Reading in the array format is straightforward:

package taskio

import (
	"encoding/json"
	"io"
)

func ReadTasksFromArray(r io.Reader) ([]Task, error) {
	dec := json.NewDecoder(r)

	var tasks []Task
	if err := dec.Decode(&tasks); err != nil {
		return nil, err
	}
	return tasks, nil
}

And reading from a stream of values (our loop until io.EOF):

package taskio

import (
	"encoding/json"
	"io"
)

func ReadTasksFromStream(r io.Reader) ([]Task, error) {
	dec := json.NewDecoder(r)

	var out []Task
	for {
		var t Task
		err := dec.Decode(&t)
		if err == io.EOF {
			return out, nil
		}
		if err != nil {
			return nil, err
		}
		out = append(out, t)
	}
}

Now you can quickly “demo” this from main without tying yourself to files: use bytes.Buffer as an io.Writer (memory) and strings.NewReader as an io.Reader (string).

package main

import (
	"bytes"
	"fmt"

	"example.com/app/taskio"
)

func main() {
	tasks := []taskio.Task{{ID: 1, Title: "Learn Encoder", Done: false}}

	var buf bytes.Buffer
	_ = taskio.WriteTasksAsArray(&buf, tasks)

	fmt.Print(buf.String()) // print the JSON we accumulated in memory
}

Notice the “beauty” of this approach: all JSON logic lives in taskio, while main simply chooses where to write and where to read from. This will be very convenient when CLI export/import commands or network requests appear later: the JSON layer won’t need to be rewritten.

6. Common mistakes when working with json.Encoder/json.Decoder

Error #1: trying to Decode without a pointer (dec.Decode(t)).
Decode must write data into a variable, and you can only write via an address. That’s why the form is almost always: dec.Decode(&x). If you forget &, you either won’t compile or you’ll get behavior that makes even a cat sad after it accidentally walked across the keyboard.

Error #2: mixing up the input format — expecting an array but getting a stream of objects (or vice versa).
If the code is written for an array, it calls one Decode(&tasks) and expects [...]. But a stream is {...}\n{...}, i.e., “multiple values” that are read in a loop. There is no universal auto-detection here: you should define the format as part of the contract (for example, “export is always an array”, “logs are always a stream”).

Error #3: ending stream reading “based on data” rather than on io.EOF.
Sometimes you see a naive idea: “if we got Task{} (zero values), then it’s the end”. No: zero values can be a perfectly valid task (for example, an empty title from bad input). End of stream in Go I/O is io.EOF, and you must check it explicitly.

Error #4: not realizing that Encode adds a newline.
Encoder.Encode(v) usually finishes the write with \n. This is not an error and not an “extra character”; it’s a convenience for “one value per line”. If you compare output as strings in tests or by hand, account for this newline; otherwise it will feel like the JSON “somehow has extra emptiness”.

Error #5: reusing one global Encoder/Decoder “for all occasions”.
An encoder and decoder are tied to a specific stream (io.Writer/io.Reader). Making them global means tying all code to a single data source and getting problems when other streams appear in parallel (even if it’s just different buffers in tests). It’s much healthier to create enc := json.NewEncoder(w) right next to where you actually write.