temp file + rename pattern: atomic file update

1. Why we need a protocol, not “just write the file”

When we first start saving data to a file, our hand reaches for os.WriteFile(...). That’s normal: concise, readable, almost like magic. The problem is that write reliability is not about “which function to call”, but about “which properties of the result we want to guarantee under failures”. And this is where the grown-up conversation begins.

Imagine a typical scenario: our program stores data (for example, a task list) in a single file. We overwrite the file with a new version. And then something unpleasant happens: the program crashes, the computer is powered off, the disk “sneezes”, space runs out, permissions are revoked, an antivirus decides to “help”. If we wrote directly into the target file, it is very easy to end up with a “half-version”: the file is already truncated, but the new version was only partially written.

It is important to lock in the fact that a failure is possible not only at Write. In the world of buffers and file systems, errors can show up later—during buffer finalization (Flush), on Sync, and even on Close, which is far from always “just closing the handle”. And defer (which we love) works such that it will execute a deferred close when exiting the function—but it does not guarantee that you will handle that error correctly if it matters to you to check Close.

2. The idea of an atomic update

Instead of “overwriting the old file in place”, we act like a careful document editor: first we produce a clean new copy on a separate sheet, verify that it is intact, and only then, at the very last moment, put it on the desk in place of the old one. In the file world, this “last moment” is a rename (rename).

The point of the temp file + rename pattern is that we split the operation into two phases. The long phase is writing all the data into a temporary file. The short phase is “publishing” the result: rename the temporary file to the target. If a failure happens during the long phase, then either the target file is not updated at all, or the old file remains. If a failure happens after a successful publish, then we already have the new version.

The key word here is atomicity. In an applied sense, for us it means: “at some single point in time the target file becomes either the old one or the new one; we do not give the user a ‘halfway’ state”. This is not magic and not an “absolute guarantee on all OSes and disks in all modes”, but it is a very powerful improvement over direct overwrite.

Why the temporary file must be next to the target

Now for a nuance that beginners love to skip (and then sadly meet confusing errors). For atomic publish, the temporary file must be created in the same directory where the target file lives. Not “in the system temp”, not “somewhere in /tmp”, but right next to it.

The reason is simple: rename within one directory is usually a predictable name replacement operation (and often atomic). But renaming between different directories or different file systems can turn into “copy + delete”, and that is a completely different reliability level and a different failure surface.

So our baseline recipe always starts like this: take dir := filepath.Dir(targetPath) and create the temp file right there. For creation, it is convenient to use os.CreateTemp(dir, pattern): it will pick a unique name and open the file for writing.

Mini example — only “created a temp next to it and remembered the name”:

package main

import (
	"fmt"
	"os"
	"path/filepath"
)

func main() {
	target := "data/tasks.txt"
	dir := filepath.Dir(target)

	tmp, err := os.CreateTemp(dir, ".tmp-*")
	if err != nil {
		fmt.Println("create temp:", err)
		return
	}
	fmt.Println("temp file:", tmp.Name()) // temp file: data/.tmp-123456...
	_ = tmp.Close()
}

3. A transparent step protocol: order and failure points

Reliability does not come from a single “correct call”, but from discipline: we define a protocol and execute it the same way every time. To avoid turning this into a 200-line sheet of “if err != nil”, it’s better to keep a step table in front of your eyes.

Below is the basic temp file + rename protocol in the form we will implement. This is a sequence of steps; if you mix up the order, it is easy to lose the meaning of “atomic publish”.

A key reminder about Flush(): bufio.Writer can accumulate data in memory and “stay quiet” until finalization; a typical pattern in Go is to write many times and only check the error on Flush.

4. Implementation in Go: small functions instead of one giant one

Now we will assemble the code so it is readable: small functions of 5–10 lines, clear names, and one “conductor”—a function that plays the protocol step by step. This is a case where decomposition is not academic prettiness, but a way not to mix up steps and not to do a “double Close”.

Create a temp file next to the target file

Let’s extract “create a temp file next to it” into a separate function:

package storage

import (
	"fmt"
	"os"
	"path/filepath"
)

func createTempNear(targetPath string) (*os.File, string, error) {
	dir := filepath.Dir(targetPath)

	f, err := os.CreateTemp(dir, ".tmp-*")
	if err != nil {
		return nil, "", fmt.Errorf("create temp in %s: %w", dir, err)
	}
	return f, f.Name(), nil
}

Note: we return both *os.File and tmpName. We will need the name to call Chmod, Rename, and Remove.

Carefully write all bytes

We write to the file via Write(data) and verify that exactly as many bytes were written as we expected. Yes, in most everyday tasks you can ignore n, but for a reliable write protocol it is useful to be strict.

package storage

import "fmt"

func writeAll(f interface {
	Write([]byte) (int, error)
}, data []byte) error {
	n, err := f.Write(data)
	if err != nil {
		return fmt.Errorf("write: %w", err)
	}
	if n != len(data) {
		return fmt.Errorf("write: partial write %d/%d", n, len(data))
	}
	return nil
}

Finalization: Sync and Close

We are friends with defer, but with critical writes there is a subtlety: it is important for us to catch the Close() error as part of the protocol, not “sometime later in defer, when we are already returning a different error”. So we call Close explicitly and check the error. At the same time, we can keep defer as a safety net for cleanup in case of early exit. The idea of defer as a mechanism that runs deferred calls after leaving the function, and the LIFO order, is exactly what makes it convenient for cleanup.

package storage

import (
	"fmt"
	"os"
)

func finalizeTempFile(f *os.File) error {
	if err := f.Sync(); err != nil {
		return fmt.Errorf("sync temp: %w", err)
	}
	if err := f.Close(); err != nil {
		return fmt.Errorf("close temp: %w", err)
	}
	return nil
}

Publish: chmod and rename

It is better to set permissions before publishing, so that the target file appears with the expected perm right away.

package storage

import (
	"fmt"
	"os"
)

func publishTemp(tmpName, targetPath string, perm os.FileMode) error {
	if err := os.Chmod(tmpName, perm); err != nil {
		return fmt.Errorf("chmod temp: %w", err)
	}
	if err := os.Rename(tmpName, targetPath); err != nil {
		return fmt.Errorf("rename temp -> target: %w", err)
	}
	return nil
}

5. Putting AtomicWriteFile together: no double Close and no junk

Now—the main function. It is easy to step on the rake of “defer Close + Close manually”. We will do it like this: defer is only a conditional safety net, and the real Close() is called explicitly exactly once. Technically, it looks like a pair of flags: “do we still need to close/remove”.

Yes, it is a bit more verbose. But the behavior is predictable, and most importantly—you won’t accidentally call Close() twice, you won’t lose the Close() error, and you won’t forget to delete the temp file on early exit.

package storage

import (
	"fmt"
	"os"
)

func AtomicWriteFile(path string, data []byte, perm os.FileMode) error {
	tmp, tmpName, err := createTempNear(path)
	if err != nil {
		return err
	}

	needClose := true
	needRemove := true
	defer func() {
		if needClose {
			_ = tmp.Close() // cleanup: we cannot "propagate" this error upward anymore
		}
		if needRemove {
			_ = os.Remove(tmpName)
		}
	}()

	if err := writeAll(tmp, data); err != nil {
		return fmt.Errorf("atomic write: %w", err)
	}

	if err := finalizeTempFile(tmp); err != nil {
		return fmt.Errorf("atomic write: %w", err)
	}
	needClose = false // Close was called exactly once

	if err := publishTemp(tmpName, path, perm); err != nil {
		return fmt.Errorf("atomic write: %w", err)
	}
	needRemove = false // temp became the target file

	return nil
}

6. Integrate into a learning app: save tasks reliably

To keep this from being theory for theory’s sake, let’s tie it to a simple learning app: a task list (mini todo). We are not using JSON yet (that is a separate topic later), so we will make the simplest possible text format: one line—one task, starting with 0 or 1 (not done / done), then a space and the text.

First, the data model (minimal, without methods—to avoid getting ahead of ourselves):

package model

type Task struct {
	Title string
	Done  bool
}

Next—serialization to text. We use strings.Builder, because it is convenient for building a string, and then we convert it to []byte for writing.

package storage

import (
	"strconv"
	"strings"

	"example.com/todo/model"
)

func encodeTasks(tasks []model.Task) []byte {
	var b strings.Builder

	for _, t := range tasks {
		done := "0"
		if t.Done {
			done = "1"
		}
		b.WriteString(done + " " + t.Title + "\n")
	}

	return []byte(b.String())
}

And finally—the “save tasks” function that uses our atomic protocol:

package storage

import (
	"os"

	"example.com/todo/model"
)

func SaveTasks(path string, tasks []model.Task) error {
	data := encodeTasks(tasks)
	return AtomicWriteFile(path, data, 0644)
}

If you ever want to write via bufio.Writer (for example, to avoid building everything in memory), remember the idea: bufio.Writer can accumulate errors and “report” them on Flush(). So in the streaming version of the protocol, the Flush() step becomes a mandatory part of finalization.

Diagram: what atomic publish looks like over time

To lock the picture into your head, it helps to imagine that the user (or another part of the program) “observes” the target file over time.

flowchart TD
    A[Old target.txt exists] --> B[Create temp next to it: .tmp-123]
    B --> C[Write data into temp]
    C --> D[Flush/Sync/Close temp]
    D --> E[Rename .tmp-123 -> target.txt]
    E --> F[Now target.txt = new version]

The key point: before the Rename step, the target file must not be “damaged” by our attempts to write. After Rename, it is considered updated.

7. Common mistakes when implementing temp+rename

Mistake #1: create the temp file “somewhere in the system temp”, not next to the target file.
People often do this out of habit: “it’s a temporary file, so /tmp”. But then Rename may turn into a move across file systems and lose the meaning of “atomic publish”. The correct rule is simple: os.CreateTemp(filepath.Dir(target), pattern).

Mistake #2: do defer tmp.Close() and then call tmp.Close() manually again.
This is one of the most popular mini-disasters: it seems like “no big deal”, but you either get unpredictable behavior or (worse) you lose an important close/flush error. It is better to pick one strategy: in a reliable write protocol, call Close() explicitly with a check, and keep defer only as a conditional safety net that you disable with a flag after a successful close. The semantics of defer and the LIFO execution order are exactly what makes this technique work—but it requires discipline.

Mistake #3: call Rename before Close.
Intuitively it feels like: “I’ll rename now, and close later”. In practice, it makes the protocol muddy: you publish a file that has not yet been finalized. The correct order: first finalize the write (Flush/Sync/Close), and only then publish.

Mistake #4: assume that “if Write did not fail—then everything is saved”.
With bufio.Writer this is especially nasty: the write may have only reached the buffer, and the error will surface on Flush. The idea that “the error can be delayed” is standard for buffered writers in Go.

Mistake #5: do not delete the temp file on errors.
If you do not have guaranteed cleanup (usually via defer), your directory will eventually turn into a “temporary file graveyard”, and the worst part is you will start getting confused about which one is current. Even if the data is important, silently leaving junk is bad policy: either delete it, or explicitly log/signal it.

1

Task

Go SELF, level 42, lesson 1

Locked

Temp Next to Target

Temp Next to Target

Open

1

Task

Go SELF, level 42, lesson 1

Locked

Lossless write

Lossless write

Open

1

Task

Go SELF, level 42, lesson 1

Locked

Atomic replacement

Atomic replacement

Open

1

Task

Go SELF, level 42, lesson 1

Locked

Batch publishing

Batch publishing

Open