Shell Scripting — The Stuff Nobody Teaches You

2025/01/08

BUILDcidevexinfra🍞

Every shell scripting tutorial starts with echo "Hello, World!" and ends around if [ -f file.txt ]. Then you’re on your own. You go write your 50th CI build script and discover — through suffering — that the language has a whole second layer of behavior that nobody warned you about.

This is that second layer. The stuff you learn from reading Kubernetes entrypoint scripts at 2am, debugging a CI pipeline that works on your Mac but explodes on Ubuntu, and slowly realizing that quoting rules are responsible for 80% of your bugs.

The set trinity

If your script doesn’t start with this, it’s a draft:

#!/usr/bin/env bash
set -euo pipefail

Three flags. Each one prevents a different class of silent failure.

set -e — exit on error. If any command returns non-zero, the script dies immediately instead of cheerfully continuing. Without this, your script will rm -rf a directory that failed to cd into, and you’ll have a very bad afternoon.

set -u — treat unset variables as errors. Without this, $UNSET_VAR silently expands to empty string. That means rm -rf /$UNSET_VAR becomes rm -rf /. Yes, really. The -u flag turns that into a hard error.

set -o pipefail — propagate pipe failures. Normally, a pipeline’s exit code is the exit code of the last command. So bad_command | grep foo succeeds if grep succeeds, even though bad_command blew up. pipefail makes the pipeline return the first non-zero exit code.

Together, they turn bash from “optimistic and silent” to “fail fast and loud.” This is what you want. Especially in CI.

One gotcha: set -e interacts weirdly with commands you expect to fail. If you’re doing grep -q pattern file and the pattern might not exist, the script dies. The fix:

# This kills your script if grep finds nothing
grep -q "pattern" file

# This doesn't
if grep -q "pattern" file; then
  echo "found"
fi

# Or use || true for "I know this might fail"
grep -q "pattern" file || true

The rule: set -e respects conditionals. If a failing command is the condition of an if, while, or the left side of &&/||, it doesn’t trigger the exit. Anywhere else, it does.

Quoting — the #1 source of bugs

I’m going to say this once: always double-quote your variables. Always. "$var", not $var. Every time you skip the quotes, you’re rolling dice.

Here’s why. Bash does word splitting on unquoted variables. If $file contains my document.txt, then:

# What you wrote
rm $file
# What bash executes
rm my document.txt
# Two arguments: "my" and "document.txt"

This isn’t a corner case. This is default behavior. Unquoted variables get split on whitespace and then glob-expanded. A variable containing * will expand to every file in the directory.

files="*.txt"
echo $files     # expands the glob — lists actual .txt files
echo "$files"   # prints literal "*.txt"

The quoting rules, concisely:

Syntax Behavior
"$var" Expands variable, preserves spaces. Use this.
'$var' Literal string. No expansion at all.
$var Expands variable, then word-splits and globs. Don’t.
"$(cmd)" Runs command, captures output, preserves spaces.
$(cmd) Same but word-splits the result.

The only time you want unquoted expansion is when you’re intentionally splitting a string into arguments. Which is almost never, and even then there’s usually a better way (arrays).

# Bad — breaks on spaces in filenames
for f in $(find . -name "*.log"); do
  rm "$f"
done

# Good — handles any filename
find . -name "*.log" -print0 | while IFS= read -r -d '' f; do
  rm "$f"
done

# Also good — if you have bash 4+
readarray -t files < <(find . -name "*.log")
for f in "${files[@]}"; do
  rm "$f"
done

Related: "${array[@]}" expands each element as a separate quoted word. "${array[*]}" joins them into a single string. You almost always want [@].

Function patterns

Bash functions are not like functions in real languages. They don’t have return types, named parameters, or local scope by default. But you can fake most of it.

Return values via echo

Functions can only return an integer (0-255). For actual data, you echo and capture:

get_branch() {
  git rev-parse --abbrev-ref HEAD
}

branch=$(get_branch)

This works because $(...) captures stdout. The function “returns” by printing to stdout. Anything else you need to print (like log messages) goes to stderr:

get_branch() {
  echo "Resolving branch..." >&2   # logging → stderr
  git rev-parse --abbrev-ref HEAD   # return value → stdout
}

Local variables

Without local, function variables leak into global scope:

bad_function() {
  result="oops"  # this is global now
}

good_function() {
  local result="contained"  # this dies when the function returns
}

Always local your variables inside functions. Always.

Error propagation

With set -e, if any command in your function fails, the whole script exits. Sometimes you want to catch that instead:

# Option 1: check return code
if deploy_service; then
  echo "deployed"
else
  echo "deploy failed with $?"
  rollback
fi

# Option 2: capture output and status separately
output=$(deploy_service 2>&1) || status=$?
if [[ ${status:-0} -ne 0 ]]; then
  echo "Failed: $output"
fi

A useful pattern for functions that need to signal failure with context:

try_connect() {
  local host=$1
  local result
  result=$(curl -sf "http://${host}/health" 2>&1) || {
    echo "Connection to ${host} failed: ${result}" >&2
    return 1
  }
  echo "$result"
}

The trap pattern

trap is bash’s equivalent of try/finally. You register a command that runs when the script exits — no matter how it exits.

cleanup() {
  rm -rf "$TMPDIR"
  echo "Cleaned up temp files" >&2
}

TMPDIR=$(mktemp -d)
trap cleanup EXIT

The EXIT signal fires on normal exit, set -e errors, and Ctrl-C (sort of — see below). This is how you prevent temp files from littering your CI runners.

You can trap specific signals too:

trap 'echo "Caught SIGINT" >&2; exit 1' INT
trap 'echo "Caught SIGTERM" >&2; exit 1' TERM
trap cleanup EXIT

The EXIT trap fires after signal traps, so you can stack them — signal trap does signal-specific stuff, then exits, which triggers the EXIT trap for general cleanup.

A CI pattern I use constantly — logging context on failure:

on_error() {
  local exit_code=$?
  echo "::error::Script failed with exit code ${exit_code}" >&2
  echo "::error::Failed at line ${BASH_LINENO[0]}" >&2
  # dump recent logs, environment, whatever helps debug
  env | sort >&2
}

trap on_error ERR
trap cleanup EXIT

ERR fires on any command that returns non-zero (with set -e). EXIT fires after that. So ERR captures the diagnostic context, then EXIT does cleanup.

Portable vs bash-specific

This matters more than you think, especially in CI. Alpine Linux containers use ash (BusyBox). Some Docker base images have dash as /bin/sh. Your Mac has zsh as default. If your shebang says #!/bin/sh, you’re writing POSIX shell, not bash.

The key differences:

Feature POSIX sh Bash
Test syntax [ "$x" = "y" ] [[ "$x" == "y" ]]
Arithmetic $(( x + 1 )) or expr $(( x + 1 )) (same, but also (( x++ )))
Arrays ❌ Not available arr=("a" "b" "c")
[[ ]] double bracket ❌ Syntax error Pattern matching, regex, no word-split
local keyword ❌ Not guaranteed Supported
Process substitution ❌ <(command), >(command)
set -o pipefail ❌ Not POSIX Supported

[[ ]] vs [ ] deserves a callout. [[ ]] is a bash keyword — it doesn’t word-split, doesn’t glob-expand, and supports =~ for regex. [ ] is a command (literally /usr/bin/[), and all the usual expansion rules apply inside it.

# With [ ] — this breaks if $var is empty or contains spaces
[ $var = "hello" ]        # syntax error if var is unset
[ "$var" = "hello" ]      # works but fragile

# With [[ ]] — handles empty, spaces, everything
[[ $var == "hello" ]]     # fine even if var is empty
[[ $var =~ ^hel.*$ ]]     # regex matching!

My rule: if the shebang says bash, use [[ ]]. If it says sh, use [ ] with aggressive quoting. Know which one you’re writing.

When to stop using bash

Bash is a glue language. It’s excellent at connecting programs, orchestrating processes, and doing file system operations. It is terrible at string parsing, data structures, error handling, and anything involving JSON.

Here’s my threshold:

Stay in bash when:

Switch to Python when:

A hybrid pattern that works well in CI:

#!/usr/bin/env bash
set -euo pipefail

# Bash does the orchestration
RESULT=$(python3 -c "
import json, sys
data = json.load(sys.stdin)
failed = [t['name'] for t in data['tests'] if t['status'] == 'failed']
print('\n'.join(failed))
" < test-results.json)

# Bash does the looping
while IFS= read -r test; do
  echo "Re-running failed test: $test"
  pytest "$test" --retry 2
done <<< "$RESULT"

Bash for the plumbing. Python for the parsing. Each language does what it’s good at.

CI patterns

These are patterns I’ve pulled from real CI scripts — the kind of things you end up writing for the fifth time before extracting them into a function.

Retry loop

Network calls in CI fail. Registries time out. DNS hiccups. You need retries:

retry() {
  local max_attempts=${1}
  local delay=${2}
  shift 2
  local cmd=("$@")

  local attempt=1
  while (( attempt <= max_attempts )); do
    echo "Attempt ${attempt}/${max_attempts}: ${cmd[*]}" >&2
    if "${cmd[@]}"; then
      return 0
    fi
    echo "Failed. Waiting ${delay}s..." >&2
    sleep "$delay"
    (( attempt++ ))
  done

  echo "All ${max_attempts} attempts failed." >&2
  return 1
}

# Usage
retry 3 5 docker push "registry.example.com/image:tag"
retry 5 10 curl -sf "https://api.example.com/health"

Note: "${cmd[@]}" — quoted array expansion. Each element stays as a separate argument. This is why we use an array instead of a plain string.

Timeout wrapper

Some commands hang. timeout exists on Linux but not on macOS (unless you install coreutils). A portable version:

with_timeout() {
  local seconds=$1
  shift

  "$@" &
  local pid=$!

  (
    sleep "$seconds"
    kill "$pid" 2>/dev/null
  ) &
  local watchdog=$!

  wait "$pid" 2>/dev/null
  local exit_code=$?

  kill "$watchdog" 2>/dev/null
  wait "$watchdog" 2>/dev/null

  return $exit_code
}

with_timeout 300 make build

Parallel execution with wait

When you need to run things in parallel and collect results:

pids=()
for service in api worker scheduler; do
  deploy_service "$service" &
  pids+=($!)
done

failed=0
for pid in "${pids[@]}"; do
  if ! wait "$pid"; then
    (( failed++ ))
  fi
done

if (( failed > 0 )); then
  echo "${failed} deployments failed" >&2
  exit 1
fi

The & backgrounds the command. $! captures its PID. wait blocks until it finishes and gives you the exit code. Simple, no dependencies, works everywhere.

Short-circuit debug logging

This is a tiny pattern but I use it everywhere:

DEBUG=${DEBUG:-false}

$DEBUG && echo "[DEBUG] Processing file: $file" >&2
$DEBUG && echo "[DEBUG] Cache dir: $CACHE_DIR" >&2

$DEBUG expands to false (a real command that returns 1) or true (returns 0). The && short-circuits. Zero overhead when debugging is off, no if blocks cluttering your code. Set DEBUG=true in your environment and every debug line lights up.

Template scripts

After writing enough scripts, you end up with a template. Here’s mine:

#!/usr/bin/env bash
set -euo pipefail

# --- Config ---
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DEBUG=${DEBUG:-false}
DRY_RUN=${DRY_RUN:-false}

# --- Functions ---
log()   { echo "[$(date +'%H:%M:%S')] $*" >&2; }
debug() { $DEBUG && log "[DEBUG] $*"; }
die()   { log "[ERROR] $*"; exit 1; }

usage() {
  cat <<EOF
Usage: $(basename "$0") [OPTIONS] <arg>

Options:
  -h, --help      Show this help
  -n, --dry-run   Print what would happen without doing it
  -v, --verbose    Enable debug output

EOF
}

cleanup() {
  # Remove temp files, restore state, etc.
  debug "Cleanup complete"
}
trap cleanup EXIT

# --- Argument parsing ---
while [[ $# -gt 0 ]]; do
  case $1 in
    -h|--help)    usage; exit 0 ;;
    -n|--dry-run) DRY_RUN=true; shift ;;
    -v|--verbose) DEBUG=true; shift ;;
    --)           shift; break ;;
    -*)           die "Unknown option: $1" ;;
    *)            break ;;
  esac
done

[[ $# -lt 1 ]] && { usage; die "Missing required argument"; }

# --- Main ---
main() {
  local target=$1
  log "Starting operation on ${target}"
  debug "Script dir: ${SCRIPT_DIR}"

  if $DRY_RUN; then
    log "[DRY RUN] Would process: ${target}"
    return 0
  fi

  # actual work here
}

main "$@"

A few things worth noting:

Learning from the best

The best shell scripting education isn’t a tutorial. It’s reading scripts written by people who’ve been burned by every edge case.

Some repos worth studying:

The exec pattern from Docker entrypoints is worth highlighting:

#!/usr/bin/env bash
set -euo pipefail

# Do setup work
configure_app
wait_for_database

# Replace shell process with the actual application
exec "$@"

exec replaces the current process. The shell is gone — the app becomes PID 1 and receives signals directly. Without exec, your app runs as a child of the shell, and signals (like Docker’s SIGTERM) go to the shell, which may or may not forward them.

Shell scripting is a weird skill. It’s not hard to learn the basics, but the gap between “works on my machine” and “works reliably in CI across different OS, different shells, at 3am when nobody’s watching” — that gap is enormous. And it’s filled entirely with quoting rules, set flags, and trap handlers.

The good news: once you internalize these patterns, they become muscle memory. You stop writing bugs that only show up when a filename has a space in it. You stop wondering why your pipeline reported success when the build clearly failed. Your scripts get a set -euo pipefail at the top, a trap cleanup EXIT three lines later, and everything in between is just the actual logic.

That’s the goal. Get the ceremony right so you can focus on the work.