1. Why you need a worker pool
When you first learn about goroutine, your brain makes a natural conclusion: “Great! Then it’s one goroutine per task.” It’s like the moment when someone first hears about endless pasta: you want to check whether it’s really endless. The problem is that “endless” usually ends at the word memory.
If there are few tasks (say, 10–100), the “one goroutine per task” approach often really works and looks elegant. But if there are many tasks (thousands, hundreds of thousands), you suddenly create a huge army of goroutine that:
- first, has overhead (maybe small, but not zero);
- second, is harder to shut down correctly (especially if there are result channels);
- third, makes you lose control: “how many tasks are running right now” becomes a matter of faith.
Worker pool solves this: we create a fixed number of workers (for example, 4, 8, 16) that take tasks from a shared queue and process them. It’s roughly like checkout lanes in a supermarket: there may be many customers, but the number of lanes is finite, and the queue regulates the load.
2. Roles and the worker pool protocol
Before writing code, it’s important not to jump into make(chan ...) like into deep water, but to agree on roles. In a worker pool it’s often convenient to think in terms of several “characters”: someone puts tasks into the queue, someone takes them, someone closes the output, someone collects the final result.
Below is a mini-diagram to lock in the mental picture (we’ll come back to it when we connect everything into one program):
flowchart LR
P[Producer<br/>creates tasks] -->|jobs channel| W1[Worker 1]
P -->|jobs channel| W2[Worker 2]
P -->|jobs channel| WN[Worker N]
W1 -->|results channel| C[Collector<br/>collects results]
W2 -->|results channel| C
WN -->|results channel| C
The most important rule that saves you from half of panics and hangs:
The channel is closed by whoever writes to it (the sender owner), and it is closed exactly once.
And if there are many writers (many workers write into results), then the closing is done by a separate entity (a coordinator) that does not write results, but only waits for workers to finish and closes the channel. This rule helps you avoid the classic panic: close of closed channel.
About close, it’s useful to remember the basic meaning: closing a channel means “no more values will be sent.” There is no “magic data deletion” there: previously sent values can still be received.
3. Prepare the example: Task, Job, and Result
So the examples aren’t abstract like “job #123,” we’ll continue in the style of a hypothetical “task list” app. Imagine we have a Task, and we want to compute stats on titles: for example, the number of words in Title. The task is intentionally simple: we don’t need a file system, HTTP, databases—only concurrent processing.
Task model
package main
type Task struct {
ID int
Title string
Done bool
}
TaskJob and TaskResult types
Let’s define what “work” is and what a “result” is. Important: we introduce separate types not because Go forces us to, but because of readability. A chan int without a name is a mystery. A chan TaskJob is almost documentation.
package main
type TaskJob struct {
Task Task
}
type TaskResult struct {
TaskID int
WordCount int
Err error
}
Note: TaskResult contains Err error. Today we won’t build a complex error strategy, but even now it’s useful to get used to the idea that the result of work can be “success or error,” and it’s convenient to pass that in a single object.
Error ErrEmptyTitle
package main
import "errors"
var ErrEmptyTitle = errors.New("empty title")
4. Baseline versions without a worker pool
Any concurrency becomes clearer if you first see the usual sequential version. And then—what channels and goroutine turn it into.
Sequential version
package main
import (
"fmt"
"strings"
)
func wordCount(title string) int {
return len(strings.Fields(title))
}
func main() {
tasks := []Task{
{ID: 1, Title: "buy milk"},
{ID: 2, Title: "write go code"},
}
for _, t := range tasks {
fmt.Println(t.ID, wordCount(t.Title)) // 1 2; 2 3 (order is stable)
}
}
The sequential version gives stable ordering, simplicity, and minimal moving parts. But if processing is heavy (for example, parsing, hashing, validating a large import), sequential execution can be slow. And that’s when you want to parallelize.
Naive concurrency: one goroutine per task
Now let’s write the naive version: we’ll start a goroutine for each Task. It looks elegant… right up until the moment there are “a lot” of tasks.
package main
import (
"fmt"
"strings"
"sync"
)
func main() {
tasks := []Task{{ID: 1, Title: "buy milk"}, {ID: 2, Title: "write go code"}}
var wg sync.WaitGroup
for _, t := range tasks {
wg.Add(1)
go func(task Task) {
defer wg.Done()
n := len(strings.Fields(task.Title))
fmt.Println(task.ID, n)
}(t)
}
wg.Wait()
}
Here we hit an important newbie “bump”: we pass t as an argument task Task, so we don’t fall into the loop variable trap. You’ve seen this before: even in Go 1.22+ there are scenarios where you can accidentally share one variable between iterations if it’s declared in the wrong place.
But the main problem isn’t that. The problem is that if tasks becomes, say, 200_000, you’ll create 200_000 goroutine. They may be lightweight, but not lightweight enough to celebrate it like a holiday. Worker pool makes this construction manageable.
5. Worker pool implementation: step-by-step code
Now we’ll assemble the key elements: the worker, the job queue, the closing coordinator for results, the producer and the collector, and then the main that ties it all together.
Worker: reads from jobs and writes to results
The key element is the worker. It should take tasks from jobs until the channel is closed. This fits perfectly with for range, because range over a channel ends when the channel is closed and all values are drained.
Pay attention to the signature: jobs <-chan TaskJob (read-only), results chan<- TaskResult (write-only). Directional channels are a small but very useful “brain safety”: the function can’t accidentally close someone else’s channel or read where it shouldn’t.
package main
import (
"strings"
"sync"
)
func worker(jobs <-chan TaskJob, results chan<- TaskResult, wg *sync.WaitGroup) {
defer wg.Done()
for j := range jobs {
title := j.Task.Title
if title == "" {
results <- TaskResult{TaskID: j.Task.ID, Err: ErrEmptyTitle}
continue
}
n := len(strings.Fields(title))
results <- TaskResult{TaskID: j.Task.ID, WordCount: n}
}
}
Important: the worker does not close results. It is not the “owner of the results channel,” because there are many workers. If every worker tries to close results, you’ll get a “who closes first” race, and the rest will panic.
Starting the pool: create N workers and a WaitGroup
A worker pool starts with a queue, but technically it’s more convenient to first write the “launcher” for workers.
package main
import "sync"
func startPool(n int, jobs <-chan TaskJob, results chan<- TaskResult) *sync.WaitGroup {
var wg sync.WaitGroup
for i := 0; i < n; i++ {
wg.Add(1)
go worker(jobs, results, &wg)
}
return &wg
}
Notice the WaitGroup discipline: Add(1) is done before starting the goroutine. This isn’t bureaucracy—it’s how you avoid a rare but very nasty race where the goroutine finishes and calls Done() before you’ve incremented the counter.
Coordinator: close results after workers finish
The collector reads for r := range results { ... }. For that loop to finish, the results channel must be closed. But the only one who can close it “at the right time” is the one who knows that all workers have finished.
That’s the coordinator’s role: wait for wg.Wait() and then do close(results).
package main
func closeResultsWhenDone(wg *sync.WaitGroup, results chan<- TaskResult) {
go func() {
wg.Wait()
close(results)
}()
}
Why close results in a separate goroutine? Because if you call wg.Wait() directly in main, you might block too early and not have time to, for example, start reading results. In simple examples this isn’t always critical, but as a habit it’s very useful: “the coordinator lives separately and does the closing when work is finished.”
Producer: send tasks and close jobs
The producer puts tasks into the queue. The ownership rule applies here: since the producer writes to jobs, the producer also closes jobs.
package main
func produceTasks(tasks []Task, jobs chan<- TaskJob) {
defer close(jobs)
for _, t := range tasks {
jobs <- TaskJob{Task: t}
}
}
defer close(jobs) is a good technique: even if you later add early returns or error handling, the chance of forgetting to close the channel drops noticeably.
Collector: read results until closed and aggregate
The collector reads results and does something useful with them: prints, appends to a slice, builds a report.
Let’s make a simple collector: print to the console and count how many errors there were.
package main
import "fmt"
func collect(results <-chan TaskResult) int {
errCount := 0
for r := range results {
if r.Err != nil {
fmt.Println("error:", r.TaskID, r.Err) // error: 3 empty title
errCount++
continue
}
fmt.Println("ok:", r.TaskID, r.WordCount)
}
return errCount
}
Here it’s important to accept this fact: the order of results is not guaranteed. Worker #2 can process task #10 faster than worker #1 processes task #1. If you need stable ordering, that’s solved separately (usually by sorting or placing the result by index), but that’s already the next layer of logic.
Put it all together in main
main does five things:
- creates channels,
- starts workers,
- starts the coordinator that closes results,
- starts the producer,
- reads results.
package main
import "fmt"
func main() {
tasks := []Task{
{ID: 1, Title: "buy milk"},
{ID: 2, Title: "write go code"},
{ID: 3, Title: ""},
}
jobs := make(chan TaskJob)
results := make(chan TaskResult)
wg := startPool(2, jobs, results)
closeResultsWhenDone(wg, results)
go produceTasks(tasks, jobs)
errCount := collect(results)
fmt.Println("errors:", errCount) // errors: 1
}
Notice a few points that together produce a “no-hangs” scheme:
- produceTasks is started in a goroutine so main doesn’t stop on sending tasks and can start collecting results in parallel.
- jobs is closed by the producer, and workers naturally finish: for range jobs ends.
- results is closed by the coordinator, which waited for all workers.
- the collector reads results via range, and the loop ends automatically after the channel is closed.
That’s the worker pool “protocol”: no one waits for a “magic number of results,” no one closes channels “by feel”; completion follows from the agreement.
Why the scheme doesn’t hang
When a beginner first sees a worker pool, they usually think: “There are so many blocking points: channels, waits, goroutines… why doesn’t it hang at all?” The calm answer is: every blocking point has a guaranteed “unblocker.”
- The producer can block on jobs <- ..., but workers read jobs and drain the queue.
- A worker can block on results <- ..., but the collector reads results.
- The collector can “wait forever,” but the coordinator closes results when workers finish.
- Workers can “wait forever” when reading jobs, but the producer closes jobs.
So completion is not “on a timer” and not “by luck”—it logically follows from channel closures and WaitGroup.
6. Common mistakes when working with a worker pool
Error #1: forgot to close jobs.
Then workers will stay forever in for range jobs, because range over a channel ends only after close(jobs). And since the workers didn’t finish, the coordinator won’t close results, and the collector will also wait forever. Typically this looks like “the program does nothing and never exits,” and you start suspecting your computer is offended.
Error #2: closing results inside the worker.
This is an almost guaranteed panic: close of closed channel, because there are multiple workers, and each one thinks it’s the boss. If closing a channel depends on several goroutines finishing, it is closed not by “any participant,” but by the coordinator after wg.Wait().
Error #3: calling wg.Add(1) after go worker(...).
Sometimes this “works,” and sometimes you catch a panic like sync: negative WaitGroup counter or weird hangs. The reason is that the goroutine might manage to call Done() before you call Add(1). The discipline is simple: Add always before starting the goroutine.
Error #4: the collector doesn’t read via range results, but “exactly N times”.
This code is fragile: you change the number of tasks, add an early exit in the producer, add filtering—and suddenly you either stop reading too early or wait for extra. Reading via range over results is tied to the real completion protocol, not assumptions.
Error #5: expecting results to arrive in the same order as tasks.
In a worker pool, execution order is not deterministic. If you need ordering, you must explicitly restore it: for example, place results by TaskID into a slice/map and then output in the desired order. The important thing is not to “fight” concurrency by expecting stable ordering, but to treat ordering as a separate formatting task.
GO TO FULL VERSION