io.Reader and io.Writer: the foundation of I/O and testability

1. The streaming model and the Reader/Writer contracts

If you previously pictured input/output as “read the whole file” or “print a line to the screen,” today we’ll expand that mental model a bit. Most real I/O in programs is structured as a stream (stream): data arrives in chunks, the total size is unknown in advance, and the speed depends on the outside world. Even if you work “with memory,” it’s often convenient to think the same way.

Imagine drinking tea through a straw. You don’t demand from the straw: “Give me all the tea at once in a single 2-liter gulp.” You do many small “reads.” A stream in Go is exactly such a “straw” for data: we read and write in portions.

Schematically (in the style of “everything brilliant is rectangles and arrows”):

flowchart LR
    A[Data source<br/>Reader] -->|"Read(p)"| B[Our logic<br/>processing]
    B -->|"Write(p)"| C[Data sink<br/>Writer]

And the key idea of the day: our logic should depend on Reader/Writer, not on what’s inside (file, network, memory, stdin/stdout).

Interface io.Reader: “put bytes into my buffer”

Now there will be a bit of “official” language, but without it we’ll start getting confused in the details. io.Reader is a standard-library interface that describes a source of bytes. Not strings, not numbers, not JSON, but bytes: “raw material” from which you can build anything.

io.Reader has exactly one method:

Read(p []byte) (n int, err error)

The meaning is: you (as the reading side) provide a buffer p, and the reader tries to fill it with data. It returns n — how many bytes were actually written into p. And it also returns err, which indicates whether “everything was OK.”

The most important rule that saves you from a ton of bugs: after Read you may use only p[:n]. Not the whole p. Not “well, there used to be something useful there.” Only the first n bytes.

Mini example: reading from a string in small chunks. Yes, strings.NewReader is a “reader from memory,” but it demonstrates the streaming model perfectly.

package main

import (
	"fmt"
	"io"
	"strings"
)

func main() {
	r := strings.NewReader("hello-go")
	buf := make([]byte, 3)

	for {
		n, err := r.Read(buf)
		if n > 0 {
			fmt.Printf("chunk=%q\n", buf[:n]) // chunk="hel" ...
		}
		if err == io.EOF {
			break
		}
		if err != nil {
			fmt.Println("read error:", err)
			return
		}
	}
}

Interface io.Writer: “take bytes from my buffer”

If Reader is the straw “into the program,” then Writer is the straw “outward.” It describes a sink for bytes: somewhere you can write data (console, file, network, an in-memory buffer, a logger, and so on).

The contract for io.Writer is also extremely short:

Write(p []byte) (n int, err error)

The meaning is similar but mirrored: you give the writer data in p, and it reports how many bytes it actually accepted (n) and whether there was an error (err). And there’s an unpleasant thing for beginners: Write is allowed to write not all of it, that is, return n < len(p).

Why? Because the outside world is not obliged to be convenient. A communication channel may be congested, a buffer may overflow, a sink may be “slow.” So in serious code, writing is often done in a loop: “write the remaining tail.”

One more important thought: Go’s standard library very actively builds I/O around small interfaces. For example, the buffer bufio.Writer implements io.Writer and supports the idea “accumulate writes first, then check the error on Flush,” because it’s more convenient to live with output errors that way.

How to correctly read n and err (and what io.EOF is)

Almost all “weird bugs” in I/O for beginners happen because of an incorrect interpretation of the pair (n, err). It seems logical: “if err != nil, then everything is bad and we must exit immediately.” But in streaming I/O this sometimes corrupts data.

io.EOF is not a “horror-error,” but the normal “end of stream” signal

io.EOF means: “data is over.” It’s not a crash; it’s the usual way to finish reading correctly. In most read loops, EOF is a break condition or return nil.

n/err combinations for Read

Here’s a table that’s useful to keep in your head (this isn’t “cramming,” it’s a map of the terrain):

>0

nil

0

io.EOF

>0

io.EOF

>0

0

Where does the case n > 0 and err != nil come from at all? It’s not a “bad implementation”; it’s an allowed part of the contract. Inside the standard library you can also find implementations where a useful portion of data is returned first, and only then an error or end-of-stream is reported.

A mini demonstration of a “normally weird” reader that returns data and an error together:

package main

import (
	"fmt"
	"io"
)

type glitchyReader struct{ done bool }

func (r *glitchyReader) Read(p []byte) (int, error) {
	if r.done {
		return 0, io.EOF
	}
	r.done = true

	copy(p, "OK")
	return 2, fmt.Errorf("network glitch")
}

func main() {
	var r glitchyReader
	buf := make([]byte, 8)

	n, err := r.Read(buf)
	fmt.Printf("n=%d data=%q err=%v\n", n, buf[:n], err)
	// n=2 data="OK" err=network glitch
}

The moral here is simple: first process the data when n > 0, then decide what to do with err.

Partial write in Write: “please write the rest”

For Write the situation is similar: n can be less than len(p), and that means “only part was written.” If you need to guarantee that everything is written, do a “write-all” loop.

package main

import "io"

func writeAll(w io.Writer, p []byte) error {
	for len(p) > 0 {
		n, err := w.Write(p)
		if err != nil {
			return err
		}
		p = p[n:]
	}
	return nil
}

The code is short, but it closes a huge class of bugs like “why did my string get cut off in the middle.”

2. Why Reader/Writer help testability

When we say “testability,” you don’t necessarily have to immediately imagine go test, mock generators, and other grown-up things. Testability at a basic level is when you can verify your logic without shamanic rituals: without touching files, the network, real stdin/stdout, and without forcing a human to “manually type 100 lines.”

And this is where io.Reader and io.Writer are real cape-less superheroes. If your function accepts io.Reader, you can feed it anything: a file or a string from memory. If your function writes to io.Writer, you can direct the output to the console or to an in-memory buffer and compare the result.

Compare the two approaches by feel (and by future pain).

Bad (hard-wired to the console: “how to test it — no idea, but I believe”):

package main

import "fmt"

func PrintHelloBad() {
	fmt.Println("hello") // always to stdout
}

Good (can write anywhere):

package main

import (
	"fmt"
	"io"
)

func PrintHello(w io.Writer) {
	fmt.Fprintln(w, "hello")
}

The difference is not “beauty.” The difference is that the second version can be tested by substituting a buffer, while the first can only be tested by eye (or with complicated stdout tricks).

3. Example: TaskBook

Now we’ll assemble a small educational piece of an application that will grow along with the course. Let it be a simple “task book” (TaskBook): the list of tasks can be written to any io.Writer, and loaded from any io.Reader. Today: no files and no JSON — only streaming and the contract.

Data model

package main

type Task struct {
	ID    int
	Title string
	Done  bool
}

Writing tasks to io.Writer

We’ll keep the format simple and “scannable”: one line = one task.

Example line:

1 buy_milk false

Write code:

package main

import (
	"fmt"
	"io"
)

func WriteTasks(w io.Writer, tasks []Task) error {
	for _, t := range tasks {
		_, err := fmt.Fprintf(w, "%d %s %t\n", t.ID, t.Title, t.Done)
		if err != nil {
			return err
		}
	}
	return nil
}

Notice the style: the function doesn’t print to stdout by itself; it doesn’t “know” where it writes. It can do only one thing: “take a writer and write the tasks into it.”

Reading tasks from io.Reader

For reading, we’ll use fmt.Fscan, which you’ve already seen earlier in the input topic. It’s convenient because Fscan can read from any io.Reader, not only from stdin.

package main

import (
	"fmt"
	"io"
)

func ReadTasks(r io.Reader) ([]Task, error) {
	var tasks []Task

	for {
		var t Task
		_, err := fmt.Fscan(r, &t.ID, &t.Title, &t.Done)
		if err == io.EOF {
			return tasks, nil
		}
		if err != nil {
			return nil, err
		}
		tasks = append(tasks, t)
	}
}

What matters here is that io.EOF means “that’s it, we’re done,” not “it broke.”

Putting it all together in main and checking via a buffer

Here we use strings.NewReader as input (it is a Reader), and bytes.Buffer as output (it is a Writer). Yes, we’ll cover bytes.Buffer in detail as the next step of the day, but as a “container for checking” it’s too convenient to ignore.

package main

import (
	"bytes"
	"fmt"
	"strings"
)

func main() {
	input := "1 buy_milk false\n2 learn_go true\n"
	r := strings.NewReader(input)

	tasks, err := ReadTasks(r)
	if err != nil {
		fmt.Println("read error:", err)
		return
	}

	var out bytes.Buffer
	if err := WriteTasks(&out, tasks); err != nil {
		fmt.Println("write error:", err)
		return
	}

	fmt.Print(out.String())
	// 1 buy_milk false
	// 2 learn_go true
}

The point of this example isn’t that “we read from a string” (in reality later it can be a file or a network). The point is that the task read/write logic does not depend on the source and sink at all. And that means: testing, moving, reusing, and combining such code becomes much easier.

4. Typical mistakes when working with io.Reader/io.Writer

Mistake #1: using the whole buffer after Read instead of buf[:n].
This is very insidious: “it seems to work,” but then it starts printing garbage, duplicating tails, or mixing data from different reads. After Read, only the first n bytes are valid (buf[:n]); the rest of the buffer contains old data that could have remained there.

Mistake #2: treating io.EOF as a “real error.”
If you write if err != nil { return err } and don’t single out err == io.EOF, your function will “fail” in the most normal scenario: when the stream ends. EOF is the normal signal to finish reading. In a read loop it usually means “exit.”

Mistake #3: doing return err before processing data when n > 0.
The “data + error” combination looks strange, but it is allowed by the contract. If you immediately exit on err != nil, you lose bytes you already received. The rule is simple: first process p[:n], then deal with the error.

Mistake #4: assuming one Write writes everything.
With real writers (network, some wrappers, writer chains), the write may be partial. If you need to write everything in full, use a loop that keeps writing based on n, or use ready-made tools when you already understand the contract precisely.

Mistake #5: binding business logic to fmt.Println and fmt.Scan.
Such code is pleasant to write for the first half hour, but later it becomes “untestable” and hard to reuse: you can’t direct output to a string, you can’t read data from memory, you can’t neatly embed it into another system. Accept io.Reader/io.Writer as parameters — and you’ll suddenly start writing code that is easy to move around and test.

1

Task

Go SELF, level 26, lesson 0

Locked

Chat greeting

Chat greeting

Open

1

Task

Go SELF, level 26, lesson 0

Locked

Traffic counter

Traffic counter

Open

1

Task

Go SELF, level 26, lesson 0

Locked

Space Filter

Space Filter

Open

1

Task

Go SELF, level 26, lesson 0

Locked

Reliable writing

Reliable writing

Open