1. The streaming model: why read and write in chunks
If you’re a beginner, your brain naturally wants to “simplify everything”: read the whole file, get []byte, and then work calmly. And with small files, that really is great. But the world, as usual, ruins everything: files can be huge, input can be infinite (for example, a stream from the network), and memory isn’t unlimited — even if you have 32 GB, they can suddenly run out too.
The streaming model in Go is the idea that data comes and goes in chunks, and we must be able to work with it in parts. This isn’t an “optimization”; it’s a fundamental technique. Even when you read a regular file, the operating system doesn’t magically return “everything at once”, but via system calls that also work in batches.
Imagine the file is a water tap, and your buffer []byte is a glass. You can fill the glass many times. If you try to “pour the whole ocean at once”, you run into philosophical questions and memory allocation errors.
io.Reader and io.Writer: two contracts that Go I/O is built on
Before writing code, it’s important to accept one thought: io.Reader and io.Writer are not “about files”. A file is just one special case. These interfaces describe a source of bytes and a sink of bytes. And that makes code flexible: the same algorithm can work with a file, a string, a buffer, and a network connection.
Here’s what the contracts look like:
type Reader interface {
Read(p []byte) (n int, err error)
}
type Writer interface {
Write(p []byte) (n int, err error)
}
The most important thing here isn’t the signatures (you can memorize them), but the meaning of the pair (n, err): n tells you how many bytes were actually processed, and err tells you what went wrong (or that the data has ended). In Go, this is a real “mini-protocol”, and once you understand it, you stop being afraid of I/O.
2. The Read contract: how to correctly read (n, err)
Here comes the key point: Read(p) tries to fill your buffer p with data and returns how many bytes were actually put there. That means after Read the valid data is only p[:n]. The rest of p is just “old garbage” (more precisely, old bytes that were there before the read).
This is where beginners most often make a mistake: they read n, but then print the entire p, getting a tail from the previous read in the output and wondering: “Go, what’s wrong with you?”. Nothing is wrong with Go — you’re printing extra data.
Another important fact: partial reads are normal. n can be less than len(p) even when err == nil. For example, the data arrived in a smaller chunk, or the OS decided to return whatever it has right now.
And, of course, io.EOF. In Go, the end of data isn’t a “special mode”, but an error value. At the same time, when implementing Read you can return fewer bytes than requested, and EOF is a normal signal that the stream is finished.
Table: how to interpret (n, err) in real code
| n | err | What it means in practice |
|---|---|---|
|
|
Read n bytes; keep going |
|
|
Read n bytes and this is the last chunk: process p[:n] and finish the loop |
|
|
No more data; finish |
|
|
Rare, but possible: usually you should just continue (and avoid an infinite loop) |
|
|
Read error; stop with an error |
Main rule: process n > 0 first, then look at the error. Otherwise you risk losing the last chunk of data when you get n > 0 together with io.EOF.
Basic file read loop: read in chunks and work with buf[:n]
Now we’ll assemble the canonical template. It looks boring, but it’s useful boredom: this code will outlive you, your grandkids, and a couple OS upgrades (okay, that’s dramatization, but you get the idea).
package main
import (
"fmt"
"io"
"os"
)
func main() {
f, err := os.Open("tasks.txt")
if err != nil {
fmt.Println("open:", err)
return
}
defer f.Close()
buf := make([]byte, 4096)
for {
n, err := f.Read(buf)
if n > 0 {
fmt.Println("chunk bytes:", n) // for example: chunk bytes: 4096
}
if err == io.EOF {
break
}
if err != nil {
fmt.Println("read:", err)
return
}
}
}
Notice: for now we’re not doing anything “smart” with the data; we’re just capturing the mechanics. In a real app, instead of fmt.Println("chunk bytes:", n) you would process buf[:n]: counting, searching, parsing, writing somewhere else — whatever you need.
Diagram: the correct order to check n and err
flowchart TD
A["Read(buf)"] --> B{n > 0?}
B -- yes --> C["process buf[:n]"]
B -- no --> D[nothing to process]
C --> E{err == EOF?}
D --> E
E -- yes --> F[break]
E -- no --> G{err != nil?}
G -- yes --> H[return error]
G -- no --> A
3. The Write contract: what short write is and why you can’t ignore n
After reading, you want to exhale: “Okay, got it, n matters.” And then Go says: “Yep. And in Write too.” Because Write(p) also returns (n, err), and you can’t ignore n either.
The situation n < len(p) when err == nil is called a short write. It’s less common than a partial read, but it’s allowed by the contract. And if you’re writing reliable code, you must either correctly “write the remainder” or return an error if you don’t know how to continue.
Sometimes people discuss the approach “accumulate a write error and then silently stop writing”, so you don’t have to sprinkle if err != nil after every Write. But what matters more here is the mechanics: you must respect n.
The writeAll function: keep writing until everything is written
package main
import "io"
func writeAll(w io.Writer, p []byte) error {
for len(p) > 0 {
n, err := w.Write(p)
if err != nil {
return err
}
p = p[n:]
}
return nil
}
The whole point is in the line p = p[n:]: we “cut off” what we already wrote and try to write the remainder. This is very similar to slice logic you’ve already seen: p[:n] is the processed part, and p[n:] is the tail.
4. Practice: read tasks.txt as a stream and count stats
So the lecture isn’t pure theory, let’s wire this into our educational “task file”. Imagine we have a file tasks.txt, where each task is a separate line. We’re not parsing tasks into structures yet (that’s a different story), but we can already do something useful: count size and number of lines without reading the whole file at once.
Counting bytes and lines via streaming reads
package main
import "io"
func countBytesAndLines(r io.Reader) (bytes int, lines int, err error) {
buf := make([]byte, 4096)
for {
n, readErr := r.Read(buf)
if n > 0 {
bytes += n
for _, b := range buf[:n] {
if b == '\n' {
lines++
}
}
}
if readErr == io.EOF {
return bytes, lines, nil
}
if readErr != nil {
return 0, 0, readErr
}
}
}
Notice how nicely the “streaming” style fits: we don’t build a giant string, we don’t do ReadFile, we don’t ask memory to “hold on a little longer”. We read chunks, count '\n' in each chunk, and that’s it.
Hook the counter up to a file
package main
import (
"fmt"
"os"
)
func main() {
f, err := os.Open("tasks.txt")
if err != nil {
fmt.Println("open:", err)
return
}
defer f.Close()
bytes, lines, err := countBytesAndLines(f)
if err != nil {
fmt.Println("count:", err)
return
}
fmt.Println("bytes =", bytes, "lines =", lines) // bytes = 123 lines = 7
}
Pay attention to an important architectural habit we’re starting to form: the function countBytesAndLines takes an io.Reader, not an *os.File. This isn’t “big architecture” yet, but it’s already the right direction of thinking: logic shouldn’t depend on whether it’s a file or something else.
5. Practice: show the file contents and don’t break on short write
Now we’ll build a second utility: print the file to the screen (to stdout). This is what you might do with os.ReadFile + fmt.Print(string(data)), but the streaming version is more honest and it trains the contract well.
Copy manually: Read → writeAll
package main
import "io"
func dump(dst io.Writer, src io.Reader) error {
buf := make([]byte, 4096)
for {
n, readErr := src.Read(buf)
if n > 0 {
if err := writeAll(dst, buf[:n]); err != nil {
return err
}
}
if readErr == io.EOF {
return nil
}
if readErr != nil {
return readErr
}
}
}
Here you can see the same “ritual” again: process n, then interpret err. This is a rhythm worth turning into muscle memory.
Hook it up to a file and os.Stdout
package main
import (
"fmt"
"os"
)
func main() {
f, err := os.Open("tasks.txt")
if err != nil {
fmt.Println("open:", err)
return
}
defer f.Close()
if err := dump(os.Stdout, f); err != nil {
fmt.Println("dump:", err)
return
}
}
Here’s where a small “enlightenment” happens: the file is an io.Reader, Stdout is an io.Writer, and our function doesn’t know what’s going on. It just “pumps bytes”. And this is one reason Go I/O code often ends up very reusable.
6. Common mistakes when working with io.Reader/io.Writer
Mistake #1: using the whole buffer instead of buf[:n].
The most common problem looks like this: you read n, _ := r.Read(buf), and then print string(buf) or compute something over the whole buf. As a result, you process the “tail” from the previous read. The fix is simple: discipline—“work only with buf[:n]”.
Mistake #2: checking err first, and then processing n.
This is the classic io.EOF trap: Read is allowed to return n > 0 and at the same time err == io.EOF. In that case, the data in buf[:n] is real and must not be thrown away. It’s useful to remember this like a mantra: “data first, status second.”
Mistake #3: treating io.EOF as a “fatal error” and printing it as a problem.
io.EOF doesn’t mean “something broke”; it means “the data ended”. In a normal file-reading scenario, EOF will always happen. If you log EOF as an error, your logs will look like the program is constantly crashing, even though it simply reached the end of the file. In the example stream-reading implementations, EOF is used as the normal completion signal.
Mistake #4: ignoring n in Write.
Yes, usually Write writes everything. But the contract says n matters. If you’re writing code that must be reliable, it’s better to either check n == len(p) or use a write-until-done loop (like writeAll). This is especially useful when writing to “unusual” writers (wrappers, network connections, custom implementations).
Mistake #5: creating an infinite loop on n == 0 and err == nil.
This is rare, but theoretically possible: a source may temporarily provide no data and also report no error. If your code doesn’t change anything in that case and just spins, you get an infinite loop and a hot laptop (pleasant in winter, but still a bug). A practical way to avoid this is to always have clear exit logic (EOF/error) and understand that a “zero read without error” is a reason to take a closer look at the data source.
GO TO FULL VERSION