1. Why separate I/O into its own layer
When you’re just starting to program, it’s natural to write “everything in one place”: open a file, read it, parse it, compute something, write it back — all in one function, ideally 60 lines long (because “it’s clear that way”). The problem is that after a couple of weeks “clear” turns into “scary,” and testing something like this is nearly impossible without real files and a bunch of environment juggling.
Imagine we have a small console app todo: it stores the task list in the file tasks.txt. An innocent idea: make a function LoadTasksFromFile(path string). It opens the file, reads the lines, parses them — and returns a slice of tasks.
Looks logical. But honestly: how do you test it?
In a test you’d have to create a temporary file, write data into it, close it, then read it… and now your tests have become “integration” tests — slow and brittle. And if the function also prints something to stdout, the tests turn into a quest.
This is where the principle shows up: the logic that works with data should not know where that data came from. Give it a “source of bytes,” and it will handle the rest. In Go, such a byte source is typically io.Reader. And a byte “sink” is io.Writer.
On top of that, in Go, error checking often takes a lot of space, and the more you keep “in one place” I/O + logic, the higher the chance you’ll miss an important check. The Go community has explicitly noted that code where half the lines are error handling is hard to read and easy to break by accident.
2. Whoever opened it closes it: the resource ownership contract
Here is an important idea that saves nerves about as effectively as a seat belt saves lives: whoever opens a resource is the one who closes it. This rule is especially critical for files, but it also applies to other things that “must be closed” (Close()): network connections, archives, and some specialized readers.
When you design a function like:
func LoadTasks(r io.Reader) ([]Task, error)
it receives a ready-to-use Reader. It doesn’t know whether it’s a file, an in-memory string, or a buffer. Therefore, it should not close anything. Closing is the responsibility of the code that created this Reader.
The same applies to writing:
func SaveTasks(w io.Writer, tasks []Task) error
This function writes to “some writer” and does not decide the fate of the resource. If the caller writes to a file, the caller closes the file.
To avoid confusion, it helps to keep a “responsibility boundary” in mind:
flowchart LR
A[main: opened file] --> B["LoadTasks(r): parses data"]
B --> A
A --> C["SaveTasks(w): formats and writes"]
C --> A[main: Flush/Close where needed]
With bufio.Writer there is one more detail: you get a “semi-transparent layer” between your logic and the real file, and you must explicitly call Flush() to push data further down. In standard explanations about error handling and bufio.Writer, it is emphasized that an error may show up specifically on Flush(), not on every Write.
3. Example: todo and a simple data format
To discuss the I/O layer with something real, we need a file format. We are not using JSON/CSV yet (those are separate topics), so we’ll make a simple “almost human-friendly” text format: one task per line.
Task line format
Let the line look like:
<id>|<done>|<title>
where:
- id is an integer,
- done is 0 or 1,
- title is the task text (no \n; everything else is allowed).
Examples:
1|0|buy milk
2|1|read about io.Reader
3|0|do not forget Flush()
Yes, the | delimiter is not the most romantic symbol, but it rarely appears in normal text. And if it does appear — we’ll later realize the format needs improvement. (Programs often evolve exactly like this: first a simple format, then “oops.”)
Task model
Let’s describe the task model with a struct — we already know how to work with struct:
package main
type Task struct {
ID int
Done bool
Title string
}
4. Reading: LoadTasks(r io.Reader)
Now the best part: our read function should not “poke the file system.” It should simply read a byte stream, split it into lines, and turn each line into a Task.
For reading “by lines,” bufio.Scanner is convenient. It’s simple, but disciplined: a Scan() loop and then a mandatory Err() check.
Let’s start with a small parser for a single line. It will be a pure function: a string in — a Task or an error out.
package main
import (
"fmt"
"strconv"
"strings"
)
func parseTaskLine(line string) (Task, error) {
parts := strings.SplitN(line, "|", 3)
if len(parts) != 3 {
return Task{}, fmt.Errorf("bad format: %q", line)
}
id, err := strconv.Atoi(parts[0])
if err != nil {
return Task{}, fmt.Errorf("bad id %q: %w", parts[0], err)
}
return Task{ID: id, Done: parts[1] == "1", Title: parts[2]}, nil
}
Notice %w: we add context and do not “throw away” the original error. This makes diagnostics nicer: at the top you can understand where it broke, while the root cause is still preserved inside.
Now let’s write LoadTasks(r io.Reader). Inside we’ll have a Scanner, and we’ll count line numbers so errors are human-friendly.
package main
import (
"bufio"
"fmt"
"io"
)
func LoadTasks(r io.Reader) ([]Task, error) {
sc := bufio.NewScanner(r)
var tasks []Task
lineNo := 0
for sc.Scan() {
lineNo++
t, err := parseTaskLine(sc.Text())
if err != nil {
return nil, fmt.Errorf("line %d: %w", lineNo, err)
}
tasks = append(tasks, t)
}
if err := sc.Err(); err != nil {
return nil, fmt.Errorf("scan tasks: %w", err)
}
return tasks, nil
}
What’s important here is that LoadTasks:
- does not call os.Open,
- does not call Close,
- does not print to the console,
- returns data and an error.
That is, it’s a “normal library function” that’s easy to reuse and test.
5. Writing: SaveTasks(w io.Writer) and Flush()
Now for the symmetric task: write tasks back into a stream. And here comes the typical beginner trap: “I wrote it with WriteString, so it must be saved.” But if you have a bufio.Writer, part of the text may remain in the buffer until Flush().
In Go there’s even a nice observation: bufio.Writer essentially behaves like a “writer that accumulates an error and reports it on finalization,” i.e., it’s very similar to the errWriter pattern. This does not mean Write is always safe — but it does mean the last word belongs to Flush().
Let’s format a single task as a line:
package main
import "fmt"
func formatTaskLine(t Task) string {
done := 0
if t.Done {
done = 1
}
return fmt.Sprintf("%d|%d|%s\n", t.ID, done, t.Title)
}
Now the writing itself. We’ll wrap w in bufio.NewWriter to avoid doing thousands of tiny writes to a file.
package main
import (
"bufio"
"fmt"
"io"
)
func SaveTasks(w io.Writer, tasks []Task) error {
bw := bufio.NewWriter(w)
for _, t := range tasks {
if _, err := bw.WriteString(formatTaskLine(t)); err != nil {
return fmt.Errorf("write task %d: %w", t.ID, err)
}
}
if err := bw.Flush(); err != nil {
return fmt.Errorf("flush tasks: %w", err)
}
return nil
}
Note: Flush() is part of the function’s contract. We do not return “success” until we have verified that the buffer has actually been flushed downstream.
6. Files on the outside: main as boundary code
Now let’s put everything together into a minimal app where main is responsible for the “real world” (paths, permissions, creating files), while LoadTasks/SaveTasks are responsible for the format logic.
This is exactly what “I/O as a layer” means: file-related code stays outside; data-related code stays inside.
Let’s make a simple program: it reads tasks.txt, adds one default task, and writes it back. Yes, the functionality is a bit silly, but architecturally it’s correct — and we’ll bring in more fun later with CLI commands (when we get to flag).
package main
import (
"fmt"
"os"
)
func main() {
f, err := os.Open("tasks.txt")
if err != nil {
fmt.Println("open:", err)
return
}
defer f.Close()
tasks, err := LoadTasks(f)
if err != nil {
fmt.Println("load:", err)
return
}
tasks = append(tasks, Task{ID: len(tasks) + 1, Done: false, Title: "new task"})
out, err := os.Create("tasks.txt")
if err != nil {
fmt.Println("create:", err)
return
}
defer out.Close()
if err := SaveTasks(out, tasks); err != nil {
fmt.Println("save:", err)
return
}
fmt.Println("ok") // ok
}
Here you can notice one subtlety not directly related to I/O as a layer, but important for “real life”: we open tasks.txt for reading, and then we create tasks.txt again for writing. This overwrites the file “straight up.” In practice you can do it, but it’s risky in case of failures — however, we are deliberately not discussing reliable file update protocols right now.
What matters to us is different: LoadTasks and SaveTasks would work just as well if instead of a file there were, for example, strings.NewReader(...) or a bytes.Buffer. And that brings us to tests.
7. Testing without a disk: strings.NewReader and bytes.Buffer
When you first see a test that doesn’t create a file at all, a little gremlin wakes up inside and says: “Suspiciously convenient. What’s the catch?” There is no catch: we simply designed the functions up front so they don’t care whether it’s a file or memory.
Let’s write a test for reading: we’ll provide input lines via strings.NewReader, get tasks, and verify the values.
package main
import (
"strings"
"testing"
)
func TestLoadTasks(t *testing.T) {
in := "1|0|a\n2|1|b\n"
tasks, err := LoadTasks(strings.NewReader(in))
if err != nil {
t.Fatalf("LoadTasks: %v", err)
}
if len(tasks) != 2 || tasks[1].Done != true {
t.Fatalf("unexpected tasks: %#v", tasks)
}
}
Now a test for writing: we’ll use bytes.Buffer as an “in-memory file.” This is a classic technique: Buffer implements io.Writer, so it fits.
package main
import (
"bytes"
"strings"
"testing"
)
func TestSaveTasks(t *testing.T) {
var out bytes.Buffer
tasks := []Task{{ID: 1, Done: false, Title: "milk"}}
if err := SaveTasks(&out, tasks); err != nil {
t.Fatalf("SaveTasks: %v", err)
}
if !strings.Contains(out.String(), "1|0|milk") {
t.Fatalf("bad output: %q", out.String())
}
}
And this is where “I/O as a layer” fully reveals itself: tests are fast, deterministic, and do not depend on the file system, access rights, the current directory, or the operating system’s mood.
Bonus: such tests are easier to read. And readability in Go is not about aesthetics — it’s a way to survive in a codebase where a lot of errors are checked.
8. Common mistakes when designing an I/O layer
Mistake #1: a “logic” function calls os.Open/os.Create by itself.
When data processing starts opening files by path inside, you nail your code to the disk. Later you’ll want to read data from memory (for tests) or from another source, and half of your logic will have “grown into” os.*. It’s better to keep opening/closing in main (or other boundary code) and pass io.Reader/io.Writer inside.
Mistake #2: calling Close() inside LoadTasks/SaveTasks even though the function did not open the resource.
This is especially annoying because it sometimes “almost works,” and then suddenly breaks the calling code: for example, you passed a *os.File, planned to read more from it, and it’s already closed. Stick to the rule: whoever opened it closes it.
Mistake #3: forgetting Flush() when using bufio.Writer.
From the outside it looks like the data “was written,” but in reality it’s sitting in the buffer. Worse: a real I/O error (for example, “disk full”) may show up specifically on Flush(), and that is a normal buffering model. In Go this is even discussed as a typical pattern: Write accumulates state, and the final step reports the error.
Mistake #4: printing errors instead of returning error.
Printing is UI. In a library function, printing turns an error into “noise,” and the caller loses control: you can’t handle the situation properly, you can’t reuse the function in another interface (CLI/HTTP), and you can’t test it. The application boundary (for example, main) prints; internal functions return error.
Mistake #5: ignoring context in errors.
If you return just err, then at the top you’ll see something like “invalid syntax” and you’ll be guessing: was it id? was it done? which line was it? Add context via fmt.Errorf("line %d: %w", lineNo, err) — it’s almost free diagnostics that saves hours.
GO TO FULL VERSION