1. Why buffering is needed
When you are just starting to write programs, it is easy to think: “I am just writing text to a file—how can that be expensive?”. But I/O has a personality. Any real-world read/write often comes down to system calls, drivers, the file system, OS caches, and other things that definitely do not happen in “one nanosecond”.
Buffering is the idea of “not poking the lower layer too often”. Instead of doing 10,000 tiny writes of 1–10 bytes, we accumulate data in memory and pass it down in larger chunks. Same for reading: we read from a file/socket “in bulk” into memory, and then serve your code the pieces it asks for.
Imagine a courier: if the courier drives out for each single paper clip, you will go broke. If they pick up a whole box of paper clips in one trip—you are a logistics wizard. bufio in Go is exactly that “box of paper clips” between your code and the real byte source/sink.
2. bufio.Reader: reading on top of io.Reader
bufio.Reader is a wrapper over any io.Reader. From the outside it looks like a normal reader, but inside it keeps a buffer and pre-fetches data from the underlying source. This gives two practical benefits: first, you “bother” the lower I/O less, and second, you get convenient methods like reading “until a delimiter”.
How to create a bufio.Reader and what it “wraps”
Let us start with the simplest thing: bufio.NewReader(r) accepts any io.Reader. It can be a file (*os.File), a string reader (strings.NewReader), a network connection—anything. And this is important: bufio is not “about files”, it is “about a stream of bytes”.
package main
import (
"bufio"
"fmt"
"strings"
)
func main() {
r := bufio.NewReader(strings.NewReader("hello\nworld\n"))
fmt.Println(r.Size()) // e.g.: 4096 (default buffer size)
}
Here we are not doing anything useful, but we capture the idea: bufio.Reader has an internal buffer. It does not have to match what you are reading.
Reading up to "\n": ReadString and the last line
In learning tasks (and in real CLI utilities) you very often need to read text “line by line”. Yes, you can read bytes and manually search for "\n", but this is one of those cases where you either write your own bufio.Reader, or you simply use bufio.Reader. We are not that rich in time.
ReadString('\n') reads data up to the delimiter and returns a string. Important detail: if "\n" is found, it is usually included in the result. That is convenient, but sometimes surprising.
package main
import (
"bufio"
"fmt"
"io"
"strings"
)
func main() {
r := bufio.NewReader(strings.NewReader("a\nb\nlast"))
for {
line, err := r.ReadString('\n')
if len(line) > 0 {
fmt.Printf("line=%q\n", line) // line="a\n", then "b\n", then "last"
}
if err == io.EOF {
break
}
if err != nil {
fmt.Println("read error:", err)
return
}
}
}
The key point here is not ReadString, but how you handle end-of-stream: the last line can arrive without "\n". And then ReadString('\n') will return a chunk of data (for example, "last") and the io.EOF error at the same time. If you check err first and then decide “EOF means we are done, exit”, you will lose the last line and be sad.
This “double” situation (len(line) > 0 and err == io.EOF) is a classic of streaming protocols and a direct continuation of what you did with (n, err) on the “raw” Read.
ReadBytes: the same, but as []byte
Sometimes it is more convenient for you to work with bytes, not strings. For example, you want to call bytes.TrimSpace later, or you process data as “raw” bytes.
ReadBytes('\n') is similar to ReadString('\n'), but returns []byte.
package main
import (
"bufio"
"fmt"
"io"
"strings"
)
func main() {
r := bufio.NewReader(strings.NewReader("x\ny\n"))
for {
b, err := r.ReadBytes('\n')
if len(b) > 0 {
fmt.Printf("chunk=%q\n", b) // chunk="x\n", chunk="y\n"
}
if err == io.EOF {
break
}
if err != nil {
fmt.Println("read error:", err)
return
}
}
}
The idea is the same: first handle what you received, and only then decide what to do about the error.
3. bufio.Writer: writing and Flush()
If bufio.Reader makes reading more convenient, then bufio.Writer makes writing both faster and… more dangerous for beginners. Because a new step appears: data can “get stuck” in the buffer and not reach the file/output until you call Flush().
And yes: this is exactly the moment when a programmer writes a program for the first time, and it “sometimes does not write the last lines”. Sounds like mysticism, but in reality—it is just forgetting Flush().
How bufio.Writer works and where data sits before Flush()
bufio.NewWriter(w) accepts any io.Writer (a file, os.Stdout, bytes.Buffer) and starts accumulating data in an internal buffer. At some point the buffer fills up—and bufio.Writer pushes part of the data down by itself. But if you wrote only a little, it may stay inside until you say: “That is it, we are finishing, push the remainder out”.
That command is called Flush().
To feel this in your hands, it is convenient to use bytes.Buffer as the “underlying” writer.
package main
import (
"bufio"
"bytes"
"fmt"
)
func main() {
var out bytes.Buffer
bw := bufio.NewWriter(&out)
_, _ = bw.WriteString("hello")
fmt.Println("now:", out.String()) // now: "" (still empty!)
_ = bw.Flush()
fmt.Println("after flush:", out.String()) // after flush: "hello"
}
Yes, we ignored errors from WriteString here because the example is small. In real code—do not ignore them.
Flush() is part of the write protocol
It is very important to internalize: Flush() is a full-fledged I/O operation. It can return an error. Sometimes the error shows up specifically on Flush(), because before that the data was sitting in memory and nothing was actually written anywhere.
An interesting detail: internally, bufio.Writer behaves like an “error piggy bank”: on write it may remember the first error and then make subsequent operations no-ops, and you will only see the final problem on Flush().
Hence a practical rule: if you used bufio.Writer, then Flush() should become as mandatory a step as “close the file”.
Close() on a file and Flush() on a buffer are different things
A very common thought: “But I did defer f.Close(), so everything will close and be written”. Here is the thing: Close() closes the file, but it is not required to automatically “push out” what you accumulated in bufio.Writer, because the buffer is a different object.
The correct sequence is usually: first Flush(), then Close() (or defer Close(), but you still call Flush() explicitly).
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
f, err := os.Create("tasks.txt")
if err != nil {
fmt.Println("create:", err)
return
}
defer f.Close()
bw := bufio.NewWriter(f)
if _, err := bw.WriteString("first line\n"); err != nil {
fmt.Println("write:", err)
return
}
if err := bw.Flush(); err != nil {
fmt.Println("flush:", err)
return
}
}
Yes, you can do defer bw.Flush(), but it is a subtle point: first, you then must handle the Flush() error correctly (and errors in defer often “get lost” if you do not think about it). Second, sometimes you want to be explicit: “right here the write is definitely finished”.
Mental map: bufio.Reader vs bufio.Writer
To avoid confusion, it helps to keep a simple diagram in mind: where the data is located and who “pokes” whom.
flowchart TD
subgraph R["READ (Reader)"]
A["your code"] --> B["bufio.Reader\n(buffer in memory)"] --> C["underlying io.Reader\n(file / network / string)"]
end
subgraph W["WRITE (Writer)"]
D["your code"] --> E["bufio.Writer\n(buffer in memory)"] --> F["underlying io.Writer\n(file / stdout / buffer)"]
E -.-> G["Flush()"]
G -.-> F
end
The point is that on reads, bufio.Reader pre-fetches data ahead of time, while on writes, bufio.Writer only pushes data down when you ask or when the buffer overflows. That is why Flush() is the central ritual you cannot skip.
And yes, this is exactly the case where rituals are useful in programming. Like brushing your teeth. Except Flush() sometimes returns an error.
4. Mini app: “a task list in a file”
Now let us put together a small piece of code that looks like part of a real app: we have a list of tasks, and we want to save it to a file and then read it back. We are not doing “perfect architecture” (that is a separate topic), but we will write carefully enough so the code is not disposable.
Let us assume the storage format is as simple as possible: one line = one task. No JSON, CSV, or other fun—just plain text.
Writing tasks line by line via bufio.Writer
Let us start with a function that takes a file and writes tasks into it. For simplicity, assume each task is a string.
package main
import (
"bufio"
"fmt"
"io"
)
func WriteTasks(w io.Writer, tasks []string) error {
bw := bufio.NewWriter(w)
for _, t := range tasks {
if _, err := bw.WriteString(t + "\n"); err != nil {
return fmt.Errorf("write task: %w", err)
}
}
if err := bw.Flush(); err != nil {
return fmt.Errorf("flush tasks: %w", err)
}
return nil
}
Two things matter here right away. We do not print errors inside the function; we return them upward (the Go way). And we treat Flush() as part of the contract: if we did not “push” the buffer, the write is not finished.
Reading tasks line by line via bufio.Reader
Now let us read the file back. We use ReadString('\n'). At the same time, we carefully handle end-of-file and the last line without "\n".
package main
import (
"bufio"
"fmt"
"io"
"strings"
)
func ReadTasks(r io.Reader) ([]string, error) {
br := bufio.NewReader(r)
var tasks []string
for {
line, err := br.ReadString('\n')
if len(line) > 0 {
tasks = append(tasks, strings.TrimRight(line, "\n"))
}
if err == io.EOF {
break
}
if err != nil {
return nil, fmt.Errorf("read task line: %w", err)
}
}
return tasks, nil
}
Yes, strings.TrimRight(line, "\n") looks a bit odd (why not TrimSpace?), but that is intentional: we remove only the newline, and let spaces in the task remain (maybe the user really wants the task "buy milk "—people are different).
Glue it together: save and read back
Now we will show a small main that writes tasks to a file and immediately reads them back. This is not a “production scenario”, just a sanity check.
package main
import (
"fmt"
"os"
)
func main() {
tasks := []string{"buy milk", "learn Go", "sleep"}
f, err := os.Create("tasks.txt")
if err != nil {
fmt.Println("create:", err)
return
}
if err := WriteTasks(f, tasks); err != nil {
fmt.Println("write tasks:", err)
_ = f.Close()
return
}
_ = f.Close()
in, err := os.Open("tasks.txt")
if err != nil {
fmt.Println("open:", err)
return
}
defer in.Close()
got, err := ReadTasks(in)
if err != nil {
fmt.Println("read tasks:", err)
return
}
fmt.Println("loaded:", got) // loaded: [buy milk learn Go sleep]
}
Notice a subtle point: we closed the file after writing, and then opened it again for reading. This is not strictly required, but in such mini checks it helps you think clearly: finish the write, close the file, then read again.
5. Typical mistakes when working with bufio.Reader/Writer
Mistake #1: “I wrote, but the file is empty / the last line is missing”.
This is almost always a forgotten Flush(). bufio.Writer can keep data in memory and not send it to the underlying writer immediately. If the program ends and the buffer was not flushed, you lose the tail of the data. The fix is simple: if you use bufio.Writer, then at the end of writing you must call Flush() and you must check its error.
Mistake #2: mixing up Close() and Flush() and hoping “it will do it by itself”.
Close() closes a resource (for example, a file), but bufio.Writer is a separate wrapper with its own internal state. You need to finish the buffer first (usually Flush()), and only then close the file. If you try to close the file without flushing the buffer, you can get a partially written result or weird effects.
Mistake #3: losing the last line when reading via ReadString('\n').
If the file does not end with a newline (that is completely normal), then the last chunk of data will be returned together with io.EOF. If your code is structured as “if error, exit”, you lose data. A reliable pattern: first process line (if it is not empty), then check err, and treat io.EOF as a normal completion.
Mistake #4: creating a bufio.Writer/bufio.Reader for each operation.
Sometimes beginners wrap inside a loop: on every iteration they create a new bufio.NewWriter(f) and write one line. This almost completely destroys the point of buffering, because you keep creating a new buffer and do not get accumulation. A buffer is usually created for the whole stream: one bufio.Writer for the entire writing process, one bufio.Reader for the entire reading process.
Mistake #5: ignoring Flush() errors and assuming that if WriteString did not complain, everything is fine.
First, Flush() is I/O and it can fail. Second, the bufio.Writer model assumes buffering and delayed error reporting: it can behave like a “first-error piggy bank”, and you will learn about the problem precisely on Flush().
GO TO FULL VERSION