Filtering and sorting traversal results

1. Why filtering is needed

When you first run WalkDir, you want to do something like: “found a path — fmt.Println(path)”. That is a normal stage: your brain needs proof that the magic works, not that you accidentally printed the developer’s favorite mantra: “it works on my machine”.

But as soon as the traversal starts producing tens, hundreds, or thousands of paths, it becomes clear: the output turns into a mess. In real code you almost never need “everything”. Usually you need “only files”, “only .go”, “everything except hidden directories”, “everything that looks like a log”, “everything that starts with task_”. This is where filtering appears: we turn a stream of “everything” into a list of “what we actually need”.

To make it easier to keep in your head, it helps to think like this:

flowchart LR
    A[WalkDir/ReadDir] --> B["Collect results into []Item"]
    B --> C[Filtering]
    C --> D[Sorting]
    D --> E[Output/further logic]

We already know how to do stages A and B. Today we will tighten up C and D.

2. Data model and collecting results

You can filter and sort strings “on the fly” inside the callback, but for beginners this often ends with three concerns mixed in one place: traversal, selection conditions, output formatting. The code quickly starts looking like earphones pulled out of a pocket: useful, but painful to untangle.

So a good practical step is to introduce a result structure that we will collect first, and only then process “cleanly” and linearly. Let our learning app of the day be a small utility fspeek: it traverses a directory, collects found items, filters them, and prints them in a predictable order.

Mini data model

Let’s start with a simple type:

package main

import "time"

type Item struct {
	Path    string
	Name    string
	IsDir   bool
	Size    int64
	ModTime time.Time
}

It has everything you typically need for filters and sorting: path (convenient for output), name (convenient for HasPrefix/HasSuffix rules), a directory flag, and a bit of metadata for sorting “by size” and “by modification time”.

Notice a small but important idea: the Size and ModTime fields are metadata. We do not read the file contents. We do not open the file for text. We just look at its “passport”.

Collecting results

Right now our goal is not to re-learn WalkDir, but to get a []Item we can work with. So I will show a minimal “collector” (it will be the basis for the examples later).

package main

import (
	"os"
	"path/filepath"
)

func collect(root string) ([]Item, error) {
	var items []Item

	err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
		if err != nil {
			return err
		}

		info, err := d.Info()
		if err != nil {
			return err
		}

		items = append(items, Item{
			Path:    path,
			Name:    d.Name(),
			IsDir:   d.IsDir(),
			Size:    info.Size(),
			ModTime: info.ModTime(),
		})
		return nil
	})

	return items, err
}

Yes, we call d.Info(), and it can be “more expensive” than just d.Name()/d.IsDir(). But for a lecture about sorting by size/time, we need these fields.

In a real app you often do this: first collect a “lightweight list”, and fetch metadata only if the user requested sorting “by size”. For now we will leave it as-is to avoid branching too much.

3. String filters: HasPrefix, HasSuffix, Contains

Filtering by file name is one of the most common tasks. Almost always you have rules like “do not show hidden”, “keep only .go”, “find everything that looks like a config”.

For this, Go has the strings package, which is pleasantly simple: it does not force you to earn a degree in regular expressions and go through an initiation ritual into “regex wizard”.

Let’s import strings and make a few small predicate functions. A predicate is a function that answers a “yes/no” question (bool). In our case: “do we keep this item or discard it?”

Hidden files and directories: HasPrefix(name, ".")

A very typical filter: hidden items in Unix-like systems often start with a dot: .git, .idea, .DS_Store. If you are making a project report, this is almost always noise.

package main

import "strings"

func isHidden(name string) bool {
	return strings.HasPrefix(name, ".")
}

Usage looks like this: if isHidden(it.Name) then we either skip it or, conversely, select “only hidden” (yes, that also happens when you are looking for “where did .env go”).

Files by extension: HasSuffix(name, ".go")

Extension is the second most popular filter. For a learning project it makes sense to show “only Go source files”.

package main

import "strings"

func hasExt(name, ext string) bool {
	return strings.HasSuffix(name, ext)
}

For example, hasExt(it.Name, ".go") will keep Go files. hasExt(it.Name, ".md") will keep Markdown.

Substring search: Contains

Sometimes the rule sounds like: “show everything where the name contains test” or “find everything that looks like a log: log”.

package main

import "strings"

func nameContains(name, part string) bool {
	return strings.Contains(name, part)
}

It is important to remember that Contains is case-sensitive. That is, "Readme.md" does not contain "readme". For “human” search, normalization via strings.ToLower is often used.

package main

import "strings"

func nameContainsFold(name, part string) bool {
	return strings.Contains(strings.ToLower(name), strings.ToLower(part))
}

4. Filtering as a separate step: the Filter function and condition composition

Now we can check conditions. But we need a “machine” that takes []Item and keeps only the needed elements. This is where a good Go style often appears: “simple function, single responsibility, clear name”.

Basic Filter function

package main

func Filter(items []Item, keep func(Item) bool) []Item {
	var out []Item
	for _, it := range items {
		if keep(it) {
			out = append(out, it)
		}
	}
	return out
}

This function does one thing: builds a new slice. It does not print, does not sort, does not poke around in the file system. And that is great: such a function is easy to read, test, and reuse.

Example: keep only files

package main

func onlyFiles(items []Item) []Item {
	return Filter(items, func(it Item) bool {
		return !it.IsDir
	})
}

If deep down you love “one-liners”, then yes, you can write it shorter. But right now we are learning to write code so that it can be read by a person after three cups of coffee and one random meeting.

Example: remove hidden items

package main

func withoutHidden(items []Item) []Item {
	return Filter(items, func(it Item) bool {
		return !isHidden(it.Name)
	})
}

Multiple conditions at once: “pipeline” or one pass

Very quickly you get a request: “keep only files” and “only .go” and “not hidden”. Beginners are tempted to make one giant if sheet, then add printing, then add sorting... and that is it, you are in a swamp.

It is much nicer to think of filtering as a pipeline:

package main

func filterGoFiles(items []Item) []Item {
	items = onlyFiles(items)
	items = withoutHidden(items)
	items = Filter(items, func(it Item) bool {
		return hasExt(it.Name, ".go")
	})
	return items
}

Yes, that is multiple passes over the slice. But at the learning level and in most real projects (where we talk about thousands of files, not millions) this is usually fine and much more readable. And if it ever becomes “heavy”, you will at least know exactly which part to optimize, rather than rewriting the entire mess.

Sometimes you still want one pass. That is also possible; the condition is just more complex:

package main

func filterGoFilesOnePass(items []Item) []Item {
	return Filter(items, func(it Item) bool {
		return !it.IsDir && !isHidden(it.Name) && hasExt(it.Name, ".go")
	})
}

Both approaches are correct. The first usually wins in readability, the second in minimal work. In the learning project, we will prefer readability.

5. Sorting: make output deterministic

After filtering we have a list that we want to show to the user. And here an unexpected truth appears: “natural order” is a slippery concept. Even if it seems that WalkDir always walks the same way, it is better not to turn that into a contract.

Deterministic output matters for comparing results, reproducibility, future tests, and so that two program runs do not look like a “file randomizer”. So a typical step before printing is sorting.

Sort by path

The clearest sorting criterion is by path:

package main

import "sort"

func sortByPath(items []Item) {
	sort.Slice(items, func(i, j int) bool {
		return items[i].Path < items[j].Path
	})
}

Note: sorting changes the order “in place”, right inside items. That is why the function returns nothing.

Sort by size and modification time

Sorting by path is useful almost always, but sometimes you want to see “top largest files” or “most recent changes”. Here we use the Size and ModTime fields.

By size (descending), biggest first:

package main

import "sort"

func sortBySizeDesc(items []Item) {
	sort.Slice(items, func(i, j int) bool {
		return items[i].Size > items[j].Size
	})
}

By modification time (newest first):

package main

import "sort"

func sortByModTimeDesc(items []Item) {
	sort.Slice(items, func(i, j int) bool {
		return items[i].ModTime.After(items[j].ModTime)
	})
}

After reads almost like English: “i after j” means “i is later than j”.

Tie-breaker: when keys are equal

An important nuance: what if two elements have the same size? Or the same time? Then your comparator (less) returns false both for “i less than j” and for “j less than i” (because neither is “less”). In that situation the order of “equal” elements can be arbitrary.

If you need a predictable order, add a second criterion, for example path. This is called a tie-breaker.

package main

import "sort"

func sortBySizeDescThenPath(items []Item) {
	sort.Slice(items, func(i, j int) bool {
		if items[i].Size == items[j].Size {
			return items[i].Path < items[j].Path
		}
		return items[i].Size > items[j].Size
	})
}

Now even if sizes are equal, the order will be stable: by path.

Stable sorting: sort.SliceStable

Sometimes you sort in two passes. For example, you want: first by path, and within the same directories by size. Or first by size, and for equal ones keep “as it was”. Then you might need a stable sort.

Stable sorting guarantees that elements that are “equal” by the key keep their relative order from the original array.

package main

import "sort"

func sortStableByName(items []Item) {
	sort.SliceStable(items, func(i, j int) bool {
		return items[i].Name < items[j].Name
	})
}

In everyday practice it is often easier to make an explicit tie-breaker (as in the example above) than to rely on “preserving the original order”. But it is useful to know that SliceStable exists.

6. Mini program fspeek: filter + sort + print

Now let’s make a short main that ties everything together. It will traverse the current directory, keep only .go files (not hidden), sort by path, and print a few fields.

package main

import (
	"fmt"
)

func main() {
	items, err := collect(".")
	if err != nil {
		fmt.Println("collect:", err)
		return
	}

	items = filterGoFiles(items)
	sortByPath(items)

	for _, it := range items {
		fmt.Printf("%s size=%d\n", it.Path, it.Size)
		// ./main.go size=1234
	}
}

The key point here is that the code reads like a story:

we collected → filtered → sorted → printed.

If you come back to this code in a week, it will not look like a “magic ritual of three nested loops and two continue in a row”.

7. Cheat sheet: which string filter to use when

Sometimes it is helpful to have a small cheat sheet. In Go, the strings package is your first Swiss Army knife for file names.

strings.HasPrefix

strings.HasSuffix

strings.Contains

strings.ToLower + strings.Contains

8. Common mistakes

Error #1: filtering by path when you actually need to filter by name.
path can contain "./internal/.cache/file.tmp", and if you check HasPrefix(path, "."), you will find almost nothing: the path typically starts with "." only in relative forms like "./...", and that is not the same as a “hidden file”. For rules like “dot at the beginning of the name”, use it.Name (or d.Name() during traversal).

Error #2: making one “super function” that traverses, filters, sorts, and prints at the same time.
Such code first feels fast, and then becomes subtly scary: any change breaks everything. It is much easier to live by the scheme: collect []Item, then filter in a separate stage, then sort, then print. Even if that is multiple passes over the data, readability pays off.

Error #3: forgetting about a tie-breaker and being surprised by “floating” order.
If you sort only by size, two files of the same size can swap places between runs. This is especially noticeable when you generate a report and try to compare results visually. Add a second criterion (usually path) or use sort.SliceStable if stability is important.

Error #4: expecting sorting to guarantee the order of “equal” elements.
Regular sort.Slice does not have to preserve the order of elements considered equal by the key. Moreover, even with the same input, different Go versions or different sorting implementations can produce a different order among equal elements. If you care about reproducible output, make explicit comparison rules.

Error #5: accidentally including directories in the file list and then interpreting Size in a weird way.
Directory size (Size on FileInfo) is not the “total size of all files inside”. If you sort by size and did not filter out directories, your top results may contain folders with “unexpected” numbers that do not make sense to a human. Usually before sorting “by file size” you do onlyFiles(items).