sed & awk

The Text-Processing Power Twins

15 min read

#commandlinetools, #sed, #awk, #textprocessing, #friday2

Before Python had pandas, before anyone said "data pipeline," Unix already had two tools that could slice, filter, transform, and summarize text at ludicrous speed — straight from the command line, on files bigger than your RAM. They're on every Linux box, every Mac, and inside WSL: sed, the stream editor, and awk, the pattern-processing language. Most developers know just enough to copy-paste a magic incantation from Stack Overflow. This crash course will make you fluent enough to write your own. Learn these two and half your "I'll write a quick script" moments turn into one-liners.

The Division of Labor

They overlap, but they have distinct sweet spots, and knowing which one to reach for is half the battle:

sed is for editing a stream of text — find/replace, delete lines, insert text. Think "search-and-replace on steroids, without opening an editor."
awk is for columnar and structured data — it splits every line into fields and gives you a real programming language (variables, math, conditionals, loops) to process them.

Rule of thumb: reach for sed when you're changing text, and awk when you're extracting or computing from columns. Both read line by line, so they handle gigabyte files in constant memory — no loading the whole thing before you can start working, the way a naive script in a "real" language often does.

Part 1: sed — The Stream Editor

The Substitute Command (99% of Real Usage)

Almost everything you'll ever do with sed comes down to one command: s, for substitute. The syntax is s/find/replace/flags, and once it clicks, you'll see find-and-replace jobs everywhere that used to mean opening an editor.

# Replace the FIRST occurrence on each line
sed 's/cat/dog/' pets.txt

# Replace ALL occurrences on each line (g = global)
sed 's/cat/dog/g' pets.txt

# Case-insensitive replace
sed 's/cat/dog/gi' pets.txt

# Use a different delimiter when your text has slashes (paths!)
sed 's|/usr/local|/opt|g' config.txt

That last example matters more than it looks — the moment your find/replace text contains a /, the standard s/.../.../ delimiter turns into a syntax error. sed lets you pick almost any character as the delimiter instead, so swapping to | (or #, or ,) sidesteps the whole problem.

By default sed prints to stdout and leaves your file untouched — it's a filter, not an editor, unless you tell it otherwise. To edit the file in place, use -i:

sed -i 's/foo/bar/g' file.txt          # GNU/Linux
sed -i '' 's/foo/bar/g' file.txt       # macOS/BSD needs an explicit backup arg

⚠️ macOS ships BSD sed, which differs subtly from GNU sed (the -i flag being the classic gotcha). On a Mac, brew install gnu-sed gives you gsed that behaves like Linux.

Backreferences and Capture Groups

Substitution gets genuinely powerful once you can reuse part of what you matched. Wrap parts of your pattern in  and refer back to them with \1, \2, and so on:

# Swap two comma-separated columns: "Lastname,Firstname" -> "Firstname Lastname"
sed 's/\(.*\),\(.*\)/\2 \1/' names.txt

# Wrap every number in brackets
sed 's/\([0-9]\+\)/[\1]/g' data.txt

# & is a shortcut for "the whole match"
sed 's/error/<<&>>/g' log.txt          # error -> <<error>>

All those backslashes in front of (, ), and + are sed's default "basic" regex dialect showing its age. Add -E (or -r on GNU) for extended regex, and the same patterns read the way you'd expect from any other tool:

sed -E 's/(.*),(.*)/\2 \1/' names.txt

Beyond Substitution: Deleting, Printing, Line Ranges

Substitution isn't the whole story — sed can also act on entire lines, addressed by a number, a range, or a /pattern/. This is what turns it into a general-purpose line filter, not just a find-and-replace tool:

sed '3d' file.txt                 # delete line 3
sed '2,5d' file.txt              # delete lines 2 through 5
sed '/^#/d' config.txt          # delete all comment lines (start with #)
sed '/^$/d' file.txt            # delete blank lines

sed -n '10,20p' file.txt        # print ONLY lines 10-20 (-n silences default print)
sed -n '/START/,/END/p' log.txt # print everything between two markers

sed '$d' file.txt               # delete the last line ($ = last)
sed '1i\Header line' file.txt   # insert a line before line 1
sed '/pattern/a\New text' f.txt # append a line after matches

The -n flag trips people up the first time: sed normally echoes every line back out after processing it, so without -n, a p command prints each matched line twice. Once you internalize "-n means only show what I explicitly print," the rest falls into place.

Handy sed One-Liners to Steal

These are the ones worth keeping in your back pocket — the kind you'll reach for weekly once you know they exist:

# Strip trailing whitespace from every line
sed -i 's/[ \t]*$//' file.txt

# Remove Windows carriage returns (CRLF -> LF)
sed -i 's/\r$//' file.txt

# Comment out every line matching a pattern
sed '/DEBUG/s/^/# /' config.txt

# Print a single line (e.g. line 42) fast
sed -n '42p' file.txt

# Double-space a file
sed 'G' file.txt

Part 2: awk — A Language Disguised as a Command

Where sed thinks in lines, awk thinks in columns. Its model is simple and brilliant: for every line, it splits the text into fields ($1, $2, ... $NF), then runs your program against a pattern { action } structure.

awk 'pattern { action }' file

If you omit the pattern, the action runs on every line. If you omit the action, matching lines are printed — which means awk '/ERROR/' file is a perfectly legitimate (if quiet) way to write grep.

Fields Are the Whole Point

This is the feature that makes awk worth learning on its own: by default it splits on whitespace, and suddenly every line is a row of addressable columns instead of an opaque string. $0 is the whole line, $1 the first field, $NF the last field — and $NF matters more than it looks, because it means "the last column" regardless of how many columns there actually are.

# Print the first column of every line
awk '{ print $1 }' data.txt

# Print first and last fields
awk '{ print $1, $NF }' data.txt

# Rearrange columns (name is field 2, id is field 1)
awk '{ print $2, $1 }' users.txt

# Use a different field separator (-F) — perfect for CSV or /etc/passwd
awk -F',' '{ print $3 }' data.csv
awk -F':' '{ print $1 }' /etc/passwd     # list all usernames

Filtering With Patterns

This is where awk starts to outclass grep — a pattern doesn't have to be a regex; it can be a full boolean expression evaluated against the fields you just split out:

# Lines where column 3 is greater than 100
awk '$3 > 100' sales.txt

# Lines matching a regex in the whole line
awk '/ERROR/' app.log

# Regex on a specific field
awk '$2 ~ /^admin/' users.txt

# Combine conditions
awk -F',' '$3 > 100 && $1 == "US" { print $2 }' sales.csv

# Print lines longer than 80 characters
awk 'length($0) > 80' code.txt

Notice that $3 > 100 up there is a complete, working awk program — no { print } needed, because a bare pattern with no action defaults to "print the matching line."

Math, Sums, and Running Totals

Because awk has real variables and arithmetic built in, tasks that would normally mean piping to bc or reaching for a scripting language become one-liners:

# Sum column 3
awk '{ sum += $3 } END { print sum }' sales.txt

# Average of column 2
awk '{ sum += $2; n++ } END { print sum / n }' data.txt

# Count lines (a poor man's wc -l)
awk 'END { print NR }' file.txt

NR is the current line number (total records), NF is the number of fields on the current line — two of awk's most useful built-in variables, and worth memorizing before anything else.

BEGIN and END Blocks

Most awk programs have a shape: something that happens once before any input, something that happens for every line, and something that happens once at the end. BEGIN and END blocks give you exactly that — a header, a per-line body, and a summary:

awk 'BEGIN { print "Name\tScore" }
     { print $1 "\t" $2 }
     END { print "-- " NR " rows --" }' scores.txt

Associative Arrays: awk's Secret Weapon

This is the feature that quietly does the work of a "real" scripting language. awk arrays are indexed by strings, not just numbers, which makes group-by and counting trivial — the kind of thing people spin up Python for without realizing awk already has it built in:

# Count occurrences of each value in column 1 (a group-by count)
awk '{ count[$1]++ } END { for (k in count) print k, count[k] }' log.txt

# Sum sales per region (column 1 = region, column 3 = amount)
awk '{ total[$1] += $3 } END { for (r in total) print r, total[r] }' sales.txt

# Count unique IP addresses in a web log
awk '{ ips[$1]++ } END { print length(ips), "unique visitors" }' access.log

That last line replaces the classic awk '{print $1}' | sort | uniq -c pipeline with a single pass — one tool doing what used to take three.

Part 3: Putting Them to Work

The commands above are the vocabulary. Here's what fluency actually looks like — a couple of real situations where reaching for sed or awk beats opening an editor or writing a script.

Cleaning up a messy export

Say a client hands you a CSV export from an old system: Windows line endings, trailing spaces on every field, and a header row you don't want. Before it goes anywhere near a database import, one sed pass fixes all three:

sed -i -e 's/\r$//' -e 's/[ \t]*$//' -e '1d' export.csv

No editor, no script file, no round-trip through Excel — the file is clean in the time it takes to run one command.

Finding your top talkers in a web log

A server's access log is a goldmine of columnar data that most people only ever grep through. Say you want the five IP addresses hitting your site the hardest:

awk '{ hits[$1]++ }
     END {
         for (ip in hits) print hits[ip], ip
     }' access.log | sort -rn | head -5

Five lines that would've been a 30-line script in most languages — and it scales to a log file of any size, because awk never holds more than the current line and the running counts in memory.

sed + awk Together

They compose beautifully with pipes: clean the text with sed, then compute with awk.

# Strip commas from numbers, then sum column 2
sed 's/,//g' sales.txt | awk '{ s += $2 } END { print s }'

# Extract, transform, summarize in one pipeline
grep "POST" access.log | awk '{ print $7 }' | sed 's/?.*//' | sort | uniq -c | sort -rn

That second pipeline is a small case study in the Unix philosophy: grep filters, awk extracts a column, sed trims a query string, and sort/uniq tally it up — five tiny, single-purpose tools chained into an ad-hoc analytics query, with no code written and no file saved.

Quick Reference

Task	Tool	Command
Find & replace in a file	sed	`sed -i 's/old/new/g' f`
Delete matching lines	sed	`sed '/pattern/d' f`
Print a line range	sed	`sed -n '5,10p' f`
Strip CRLF / trailing space	sed	`sed -i 's/\r$//' f`
Print specific columns	awk	`awk '{print $1,$3}' f`
Filter by a column value	awk	`awk '$3 > 100' f`
Sum a column	awk	`awk '{s+=$2} END{print s}' f`
Group-by count	awk	`awk '{c[$1]++} END{for(k in c)print k,c[k]}' f`
Use a custom separator	awk	`awk -F',' '{print $2}' f`

Gotchas Worth Knowing

BSD vs GNU. macOS's sed and awk differ from Linux's. Install gnu-sed and gawk via Homebrew for portable behavior.
Quote your scripts in single quotes so the shell doesn't expand $1, $NF, etc. before awk sees them.
sed regex is basic by default — use -E for the extended syntax you know from other tools.
-i edits in place with no undo. Test without -i first, or keep backups (sed -i.bak).
awk fields are re-split on assignment — the moment you set $2 = ..., awk rebuilds $0.

Your Path Forward

You've gone from copy-pasting one-liners to understanding why they work — which means you can now write your own instead of hunting for the right Stack Overflow answer. Start small: the next time you catch yourself opening a file in an editor just to delete a few lines or pull out a column, stop and reach for sed or awk instead.

Essential commands to remember:

# Find and replace (sed)
sed -i 's/old/new/g' file.txt

# Delete matching lines (sed)
sed '/pattern/d' file.txt

# Print specific columns (awk)
awk '{ print $1, $3 }' file.txt

# Filter by a column condition (awk)
awk '$3 > 100' file.txt

# Group-by count (awk)
awk '{ count[$1]++ } END { for (k in count) print k, count[k] }' file.txt

Learn s/old/new/g, line deletion, field printing, and associative-array counting, and you'll handle 90% of daily text wrangling without ever opening an editor or writing a throwaway script. The rest — the exotic flags, the obscure GNU extensions — you can look up the day you actually need them.

P.S. — sed and awk both predate the GUI. They were built in 1974 and 1977 respectively, for machines with a tiny fraction of the memory in your phone, and they still outrun modern tools on the exact job they were designed for: turning raw text into an answer, one line at a time. That's not nostalgia — that's fifty years of a good design holding up.