What exactly are we protecting — state in app, storage, and cache

1. Why ask “what are we protecting”

When you first learn about Mutex, it’s very tempting to use a universal spell: “If you’re scared, wrap it in mu.Lock()”. Sometimes it even works… while the project is small. But once you get an app layer, a storage layer, and “oh, let’s add a cache for speed too”, the universal lock turns into a universal cause of hangs, lag, and mysterious deadlocks.

The point of this lecture is not to memorize one more API (we’ll barely touch APIs today), but to learn to answer three questions: what state do we actually have, who owns it, and which synchronization primitive should protect it. Because Mutex is not “a paper clip for everything”, it’s more like “a seat belt”: you should buckle up, but not in someone else’s car and not around your neck.

Where state lives: app, storage, and cache

If we simplify, state is anything that can change and that multiple goroutines can simultaneously observe/influence. But in real applications, state usually spreads across layers: some lives in the app (business logic), some in storage (persistence/DB/files), and some in the cache (an accelerator and a source of “slightly stale truth”).

It helps to keep a table like this in mind—it’s not “the ultimate truth”, but it’s a good compass:

And here’s an important idea: not everything called state has to be shared. Sometimes the best way to “protect state” is to make sure it isn’t shared at all. In the spirit of Go’s well-known idea “share memory by communicating”, where one goroutine owns the data and everyone else communicates via channels.

But today we’ll stay in the world of “we have shared state, and we want to tame it”.

2. State in the app layer: what can break

Beginners often see the app layer as “just functions that call storage”. Because of that, it’s frequently left without synchronization: “well there’s no map there…”. Then suddenly a feature shows up: “show statistics”, “don’t run two identical imports in parallel”, “deduplicate identical requests”, “count metrics”—and now the app layer has its own mutable state.

To make this concrete, let’s mentally evolve our training app—a task manager (let’s call it tasker). We have an entity Task and operations “create”, “list”, “mark as done”. HTTP/CLI can call these operations in parallel (in HTTP that’s the norm).

Example app-state: “don’t run two identical operations in parallel”

Imagine we have task import from a file. We want two parallel import requests for the same file not to run at the same time (otherwise you get duplicates, races, weird effects). This is pure app-state: it’s not storage and not cache, it’s a business constraint: “one import per file”.

package app

import "sync"

type ImportGuard struct {
	mu       sync.Mutex
	inFlight map[string]bool
}

func (g *ImportGuard) TryStart(name string) bool {
	g.mu.Lock()
	defer g.mu.Unlock()

	if g.inFlight == nil {
		g.inFlight = make(map[string]bool)
	}
	if g.inFlight[name] {
		return false
	}
	g.inFlight[name] = true
	return true
}

Here we protect an invariant: “for each name, either an import is running or it isn’t”. And that invariant spans both the map and the check/set logic—so it’s Mutex, not atomic.

And note: this state must not leak outside. Nobody should get a reference to inFlight, otherwise the whole point of protection is lost.

Example app-state: fast counters via atomic

Sometimes the app layer stores very simple numbers: “how many requests we had”, “how many errors we had”. It’s convenient for diagnostics. Here atomic is often a good fit, because we don’t need a large invariant—we just increment a counter.

package app

import "sync/atomic"

type Metrics struct {
	Requests atomic.Int64
	Errors   atomic.Int64
}

func (m *Metrics) IncRequest() { m.Requests.Add(1) }
func (m *Metrics) IncError()   { m.Errors.Add(1) }

Remember the rule: if you chose atomic for a variable, then all access to it must go through Load/Store/Add. “Just once m.Requests++” is like “just once in prod without migrations”: it might work, but later you’ll be embarrassed.

4. State in the storage layer: where races appear

Storage is where the “truth” lives (or what the application considers truth). Even if instead of a DB you currently have an in-memory map or a file on disk, the requirements are roughly the same: writes must be sequential and predictable, reads must not see a half-broken state, and data must not “accidentally” change because someone kept a reference to internals.

In-memory storage: map + Mutex and copies to the outside

Let’s start simple: an in-memory task store. A map[int]Task cannot be read/written concurrently without coordination. Even “read-only” is dangerous if there is a write somewhere nearby.

package storage

import "sync"

type Task struct {
	ID    int
	Title string
	Done  bool
}

type MemStore struct {
	mu    sync.Mutex
	tasks map[int]Task
	next  int
}

The task creation method has to update two fields at once (next and tasks), so this is one critical section.

package storage

func (s *MemStore) Create(title string) Task {
	s.mu.Lock()
	defer s.mu.Unlock()

	if s.tasks == nil {
		s.tasks = make(map[int]Task)
		s.next = 1
	}
	t := Task{ID: s.next, Title: title}
	s.tasks[t.ID] = t
	s.next++
	return t
}

Think of it like a store checkout: while the cashier is ringing up items, a second cashier must not simultaneously change the same numbers on the display.

Now the key point: when we implement List(), we must not return something that lets external code bypass synchronization. With a map it’s obvious (don’t return the map itself). With slices it’s often “not obvious”—but the principle is the same: return a copy.

package storage

func (s *MemStore) List() []Task {
	s.mu.Lock()
	defer s.mu.Unlock()

	out := make([]Task, 0, len(s.tasks))
	for _, t := range s.tasks {
		out = append(out, t)
	}
	return out
}

Yes, it’s slightly more expensive. But that’s the price of storage remaining the owner of its data.

File-based storage: state is also the write protocol

When storage writes to a file, state is not only variables in memory. State is also the process: open the file, write, close, maybe replace the old version with a new one. If two parallel requests start writing to the same file, you can get interleaved bytes and a file that “seems to exist, but can’t be read”.

That’s why file-based storage often uses a simple rule: all write operations are serialized by one Mutex. And it’s highly desirable not to hold that lock longer than necessary: under the lock we do I/O, yes, but we try not to do extra work under it, like JSON validation of input data (it’s better to do that earlier in the app layer).

5. Cache: speed up reads, but don’t replace the truth

A cache is like a cheat sheet during an exam. It speeds you up, but if the cheat sheet is outdated, you can confidently write nonsense. That’s why a cache must be designed with understanding: what is the source of truth, and what is a derived accelerator.

In our tasker, the cache could store, for example, “the latest task list” and be reset (invalidated) on any write (create/done/delete). Then reads will be fast, and writes will invalidate the cache.

Cache wrapper around Store: RWMutex + invalidation

Let’s define a storage interface (it already logically exists in the project), and on top of it a caching wrapper:

package app

import "tasker/storage"

type Store interface {
	Create(title string) (storage.Task, error)
	List() ([]storage.Task, error)
	MarkDone(id int) error
}

Cache:

package app

import "sync"

type CachedStore struct {
	mu          sync.RWMutex
	underlying  Store
	cachedList  []storage.Task
	hasList     bool
}

Reading the list: under RLock we check the cache; if present, we return a copy so nobody can “tweak the cache by hand”.

package app

import "tasker/storage"

func (c *CachedStore) List() ([]storage.Task, error) {
	c.mu.RLock()
	if c.hasList {
		cp := make([]storage.Task, len(c.cachedList))
		copy(cp, c.cachedList)
		c.mu.RUnlock()
		return cp, nil
	}
	c.mu.RUnlock()

	list, err := c.underlying.List()
	if err != nil {
		return nil, err
	}

	c.mu.Lock()
	c.cachedList = list
	c.hasList = true
	c.mu.Unlock()

	return list, nil
}

Yes, there’s a subtlety here: we don’t hold the lock while calling into the underlying storage. This matters: storage may do I/O, may be slow, and holding the cache lock for that is a bad idea. We fetch the data “from the outside”, and then carefully put it into the cache.

Write: after a successful write we invalidate the cache. Invalidate under Lock, otherwise you can leave a half-state.

package app

func (c *CachedStore) MarkDone(id int) error {
	if err := c.underlying.MarkDone(id); err != nil {
		return err
	}

	c.mu.Lock()
	c.hasList = false
	c.cachedList = nil
	c.mu.Unlock()

	return nil
}

And here’s the main concept: the cache protects its own structure (hasList, cachedList), but it must not “intrude” into protecting the underlying storage. Storage has its own lock; the cache has its own.

6. How to choose and place locks

Define the “objects of protection”

When you think “I need a Mutex here”, try to formulate a sentence like:

“I protect _____ from _____ to guarantee _____.”

For example:

“I protect MemStore.tasks and MemStore.next from concurrent writes to guarantee that IDs are unique and the map does not break.”

Or:

“I protect CachedStore.cachedList from concurrent reads/writes to guarantee that the cache is either fully valid or fully invalidated.”

This wording helps you understand something important: we protect not code, but meaning (an invariant). And the same Mutex should guard one set of related meanings. If a lock “guards everything” sooner or later you’ll start holding it while doing something long, and you’ll end up with a line at the door.

Where the lock should live: app vs storage vs cache

It’s very tempting to do this:

var mu sync.Mutex // one for everything!

And then call mu.Lock() at the beginning of every handler. This really does eliminate data races… at the cost of turning your application into a strictly single-threaded one. It’s like you bought Go for concurrency, and then taped all goroutines with “do not run”.

A much healthier way is to think in terms of state owners:

Storage owns persisted data and protects it with its lock. Cache owns its cache and protects it with its lock. App owns its business invariants and protects them with its locks/atomics.

And it’s highly desirable that locks do not stick out as part of the public API. If someone outside can do store.mu.Lock(), you lose control: external code can forget Unlock(), can hold the lock too long, can create a deadlock, and you won’t even be able to debug it properly (because “it’s not me, it’s them” is a weak argument).

Request flow and lock order

When an HTTP request POST /tasks/{id}/done arrives, roughly this happens:

sequenceDiagram
    participant H as HTTP handler
    participant A as App service
    participant C as Cache wrapper
    participant S as Storage

    H->>A: MarkDone(id)
    A->>A: atomic Requests.Add(1)
    A->>C: MarkDone(id)
    C->>S: MarkDone(id) (storage lock inside)
    S-->>C: ok
    C->>C: cache lock -> invalidate list
    C-->>A: ok
    A-->>H: 204 No Content

In this diagram, what matters is that locks do not “cross layers”: storage does its job, the cache does its job, the app does its job. That makes it easier to maintain order and avoid situations like “I’m holding the app lock and waiting for storage, while storage is waiting for something that’s waiting for the app lock”.

Nuance: defer Unlock and loops

defer mu.Unlock() is a great habit, but it has a personality: defer runs at the end of the function, not “at the end of the current loop iteration”. Because of that, you can accidentally hold a lock much longer than you think.

This mistake is so real that people love catching it in production after everything “seemed to work”. In one performance postmortem, a case is shown where a defer inside a loop causes the lock to be held until the end of the function and creates a huge waiting queue.

The moral is simple: use defer, but consciously. If the lock must be released on each iteration—call Unlock() explicitly or move the iteration body into a small helper function so the defer fires “earlier”.

7. Common mistakes

Mistake #1: confusing the “source of truth” with the cache.
People often start using the cache as storage: “let’s just mutate cachedList and not go to storage”. This quickly leads to desynchronization: different operations start seeing different versions of reality, and after a restart everything suddenly “disappears”. The right discipline is: storage is the truth; the cache is an accelerator that you can throw away at any moment without losing meaning.

Mistake #2: one Mutex for the whole world.
A global lock feels simple: no races, you can sleep well. But very soon you notice the app “for some reason is slow”, even though you have goroutines. The reason is usually that you turned a concurrent system into a queue for a single lock. It’s much better to split state by owners and guard each meaning with its own lock.

Mistake #3: holding a lock while calling “someone else’s” code or doing long I/O.
The most toxic combination is holding a Mutex while also waiting for something: reading a file, a network response, a message from a channel. That’s a direct path to deadlocks and “hangs without a panic”. It’s more reliable to take the lock only for what you need to snapshot/update, then release it and only then do long operations.

Mistake #4: returning internal structures and thereby bypassing synchronization.
If storage returns a reference to its slice or map, external code can mutate data without a lock—and you’re back in the world of races, but now without explicit Lock() calls in code, which makes the bug “magical”. In these places it’s better to return a copy (a snapshot). Yes, that’s allocations. But they’re cheaper than a week of debugging “why sometimes Done=true all by itself”.

Mistake #5: mixing atomic and Mutex for the same value without a clear reason.
Sometimes people do this: “read under a lock, but increment via atomic—it’s faster”. This creates a very muddy memory model for the team: nobody understands where the truth is. If a variable is atomic, it’s atomic always. If a variable is part of an invariant, it’s under a Mutex. Mixing is possible, but only when you can clearly explain what exactly you guarantee (and beginners should usually stay away from that).