You SSH into a GPU server. You start a training run. You close your laptop to go get coffee. You come back, open the lid, and the training is dead. Three hours of compute, gone.
The problem is SIGHUP. When your terminal disconnects, the kernel sends a hangup signal to every process attached to that session. Your training script doesn’t know the difference between “the user wants me to stop” and “the WiFi blipped.” It just dies.
There are three classic tools that solve this: tmux, screen, and nohup. They solve the same underlying problem — keeping processes alive after disconnection — but they operate at completely different levels. Picking the right one depends on what you actually need.
nohup — the one-liner
nohup does exactly one thing: it tells a process to ignore the hangup signal. That’s it. That’s the whole tool.
nohup python train.py --epochs 100 &
The nohup prefix intercepts SIGHUP. The & sends it to the background. Output goes to nohup.out by default (or wherever you redirect it). You can now close your terminal, fly to another continent, and that process keeps running.
When nohup is the right call:
- You have one command to run
- You don’t need to interact with it after launch
- You just want fire-and-forget
When nohup falls short:
- You can’t reattach to see live output (it’s gone, you’re reading a log file)
- You can’t scroll back through what happened
- Multiple processes? You’re managing a pile of background jobs by hand
- Want to send input to the process later? Too bad
nohup is a screwdriver. Sometimes you need a screwdriver. But if you need to build a cabinet, you want a workshop.
nohup vs disown vs & — clearing up the confusion
People use these interchangeably. They’re not the same thing.
# Just background — still attached to terminal, dies on disconnect
python train.py &
# Background + ignore SIGHUP — survives disconnect
nohup python train.py &
# Already running? Background it, THEN detach it
python train.py
# Ctrl+Z (suspend)
bg # resume in background
disown # detach from shell's job table
| Tool | Ignores SIGHUP | Backgrounds | Detaches from shell |
|---|---|---|---|
& |
No | Yes | No |
nohup |
Yes | No (add &) |
No |
disown |
Yes (sort of) | No | Yes |
nohup ... & |
Yes | Yes | No |
disown -h |
Yes | No | No (keeps in job table) |
The distinction: nohup is proactive — you decide before launch. disown is reactive — the process is already running and you realize “oh wait, I need to close this terminal.” & by itself is neither — it just frees up your prompt.
The gotcha with disown: it removes the process from your shell’s job table, which means jobs, fg, and bg forget about it. The process is still running, but your shell pretends it doesn’t exist. This is fine if you’re just trying to survive a disconnect. It’s annoying if you wanted to bring it back to the foreground later.
tmux — the workshop
tmux is a terminal multiplexer. That’s a fancy way of saying: it creates virtual terminals that live on the server, independent of your SSH connection.
tmux new -s training
python train.py --epochs 100
# Ctrl+b, d (detach)
You’re now back at your regular shell. The tmux session is still running on the server. Your training is still going. You can close the SSH connection, close your laptop, wait a week, SSH back in, and:
tmux attach -t training
Everything is exactly where you left it. Scrollback, output, cursor position — all of it.
Why tmux is the default answer in 2024
tmux gives you:
- Detach/reattach — the killer feature. Survive disconnects, pick up where you left off
- Split panes — run training in one pane,
nvidia-smiin another,htopin a third - Multiple windows — like browser tabs for terminals
- Session sharing — pair programming over SSH (both people see the same terminal)
- Scrollback — Ctrl+b, [ to enter copy mode and scroll through history
- Scriptable — automate window layouts, startup commands, entire workflows
The basics:
# Session management
tmux new -s name # create named session
tmux attach -t name # reattach to session
tmux ls # list sessions
tmux kill-session -t name
# Inside tmux (Ctrl+b is the prefix key)
Ctrl+b, d # detach
Ctrl+b, c # new window
Ctrl+b, n / p # next / previous window
Ctrl+b, % # split pane vertically
Ctrl+b, " # split pane horizontally
Ctrl+b, arrow keys # move between panes
Ctrl+b, [ # scroll mode (q to exit)
The tmux + SSH workflow every ML engineer should know
This is the pattern that saves hours of compute and sanity:
# 1. SSH into GPU box
ssh gpu-server
# 2. Create (or reattach to) a tmux session
tmux new -s experiment || tmux attach -t experiment
# 3. Set up your workspace
# Pane 0: training script
# Pane 1: nvidia-smi -l 1 (GPU monitoring)
# Pane 2: tail -f logs/train.log
# 4. Start training in pane 0
python train.py --config configs/run42.yaml
# 5. Detach: Ctrl+b, d
# 6. Close laptop. Go live your life.
# 7. Later — from anywhere:
ssh gpu-server
tmux attach -t experiment
# Everything is right where you left it
Pro tip: add this to your .tmux.conf so you don’t lose sessions when SSH drops unexpectedly:
# Reattach or create
set-option -g detach-on-destroy off
Scripting tmux layouts
If you have a standard multi-pane setup you use every time:
#!/bin/bash
SESSION="ml-dev"
tmux new-session -d -s $SESSION
# Window 0: Training
tmux rename-window -t $SESSION:0 'train'
tmux send-keys -t $SESSION:0 'cd ~/experiments && python train.py' Enter
# Window 1: Monitoring
tmux new-window -t $SESSION:1 -n 'monitor'
tmux split-window -h -t $SESSION:1
tmux send-keys -t $SESSION:1.0 'nvidia-smi -l 2' Enter
tmux send-keys -t $SESSION:1.1 'htop' Enter
tmux attach -t $SESSION
screen — the elder
GNU Screen came first (1987). It does the same core thing — persistent terminal sessions that survive disconnects. For a long time it was the only game in town.
screen -S training
python train.py
# Ctrl+a, d (detach)
screen -r training # reattach
screen -ls # list sessions
tmux vs screen in 2024
Here’s the honest comparison:
| Feature | tmux | screen |
|---|---|---|
| Split panes | Yes (horizontal + vertical) | Yes, but clunkier |
| Session sharing | Built-in, easy | Possible, harder to set up |
| Scripting | Excellent (send-keys, etc.) | Basic |
| Configuration | .tmux.conf — clean |
.screenrc — arcane |
| Copy mode | vi or emacs bindings | Works but less intuitive |
| Active development | Yes | Essentially maintenance mode |
| Default on servers | Sometimes | More often (older systems) |
| Mouse support | Good | Painful |
| Plugin ecosystem | tpm, themes, statusline | Not really |
| Documentation | Abundant | Adequate, dated |
The verdict: tmux won. Not because screen can’t do the job — it can, and it’s been doing it since before tmux existed. But tmux’s interface is cleaner, its configuration is saner, and it’s where the community’s energy goes. If you’re starting fresh, learn tmux. If you’re on a legacy server that only has screen installed, screen works fine — the core workflow (create, detach, reattach) is nearly identical.
The one exception: if you’re on an embedded system or minimal server where only screen is available and you can’t install anything, screen is your friend. It’s been around for nearly 40 years. It’ll be around for 40 more.
The comparison table
| nohup | screen | tmux | |
|---|---|---|---|
| What it is | Signal interceptor | Terminal multiplexer | Terminal multiplexer |
| Detach/reattach | No | Yes | Yes |
| Multiple sessions | No | Yes | Yes |
| Split panes | No | Yes (basic) | Yes (excellent) |
| Live interaction | No (fire-and-forget) | Yes | Yes |
| Learning curve | None | Low | Low-medium |
| Active development | N/A (POSIX standard) | Maintenance mode | Active |
| Best for | Single background job | Legacy systems | Everything else |
Modern alternatives
The tmux/screen duopoly isn’t the only option anymore.
Zellij
Zellij is a terminal multiplexer written in Rust. It has a discoverable UI — keybindings are displayed at the bottom of the screen, which is huge for people who can’t memorize tmux’s Ctrl+b, [obscure key] combos.
# Install
cargo install --locked zellij
# or
brew install zellij
# Use it
zellij # start
zellij attach # reattach
Why people like it: Better defaults. Built-in layout system. Plugin architecture (WebAssembly). Floating panes. The onboarding experience is genuinely friendlier than tmux.
Why you might stick with tmux: Zellij is newer and less ubiquitous. It won’t be on that random GPU server you SSH into. tmux is everywhere. Zellij is where you choose to install it.
abduco + dvtm
The Unix philosophy take: separate session management from window management.
- abduco — detach/reattach sessions (the
nohupreplacement part of tmux) - dvtm — tiling window management for the terminal (the panes/windows part of tmux)
abduco -c session-name dvtm # create session with dvtm inside
abduco -a session-name # reattach
This is for the purists who think tmux does too much. I respect it, but I wouldn’t recommend it for beginners — the documentation is sparse and the community is small.
systemd — when you actually mean “always running”
Here’s the thing about tmux and nohup: they keep a process alive as long as someone starts it. If the server reboots, your tmux session is gone. Your nohup process is gone.
If you need something to survive reboots and restart automatically on failure, you want a service.
# /etc/systemd/system/my-training.service
[Unit]
Description=Training Run
After=network.target
[Service]
Type=simple
User=shiyuan
WorkingDirectory=/home/shiyuan/experiments
ExecStart=/usr/bin/python3 train.py --config run42.yaml
Restart=on-failure
RestartSec=10
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl start my-training
sudo systemctl status my-training
journalctl -u my-training -f # live logs
When to use systemd over tmux:
- The process should survive server reboots
- It should auto-restart on crash
- It’s a long-running service, not a one-off experiment
- You want structured logging (journald)
- You need resource limits (cgroups, memory caps)
When tmux is still better:
- You need to interact with the process (send input, read terminal output)
- It’s a temporary experiment, not a permanent service
- You don’t have root access (systemd user services exist, but they’re fiddly)
The mental model: tmux is for interactive sessions that outlive your connection. systemd is for services that outlive your attention.
The decision tree
Start here when you’re about to run something long on a remote server:
Do you need to interact with it after starting?
├── No — just fire and forget
│ ├── One-off command? → nohup command &
│ └── Needs to survive reboots / auto-restart? → systemd service
│
└── Yes — need live terminal access
├── Is tmux available? → tmux (always the first choice)
├── Only screen available? → screen (same workflow, older tool)
└── Can install things? → tmux (or zellij if you want modern UX)
Running something and realized too late you need it to survive?
→ Ctrl+Z, bg, disown
Or even simpler:
| Situation | Tool |
|---|---|
| Quick background task, don’t care about output | nohup cmd & |
| Already running, need to detach NOW | Ctrl+Z → bg → disown |
| Remote development, ML training, anything interactive | tmux |
| Legacy server, nothing else available | screen |
| Production service, must auto-restart | systemd |
| Local dev, want a modern multiplexer | zellij |
Combining them
These tools aren’t mutually exclusive. Some combinations make sense:
# tmux session with a nohup process inside
# (belt and suspenders — process survives even if tmux dies)
tmux new -s training
nohup python train.py > train.log 2>&1 &
# Ctrl+b, d
# systemd service that you monitor from tmux
tmux new -s monitoring
journalctl -u my-training -f # live logs in a tmux pane
Though honestly, if you’re in tmux, you usually don’t need nohup — tmux already handles persistence. The nohup inside tmux pattern is for the truly paranoid. (Which, after losing a 12-hour training run, you might be.)
The mental model
Think of it as layers of persistence:
Layer 0: bare process — dies when terminal closes
Layer 1: nohup / disown — survives terminal close, but no reattach
Layer 2: tmux / screen — survives terminal close, can reattach
Layer 3: systemd — survives everything, auto-restarts
Each layer adds capability and complexity. Most of the time, Layer 2 (tmux) is the sweet spot — enough persistence to be useful, enough interactivity to be pleasant, and not so much ceremony that you need a config file to start a training run.
The server doesn’t care about your laptop. Your processes shouldn’t either.