Linux Fundamentals
Linux is a kernel, not a full OS. What you run is a distribution — the kernel plus a userland (GNU utilities, init system, package manager, libraries). Understanding the split between kernel and userspace is the single most useful mental model; almost every question about “how Linux works” resolves to “is that kernel or userspace?”
Kernel vs userspace
Two address spaces, two privilege levels:
┌──────────────────── userspace ───────────────────┐
│ bash sshd nginx python docker │ ring 3
│ ↕ system calls (open, read, socket…) │
├──────────────────── kernel ──────────────────────┤
│ scheduler VFS netfilter drivers VM │ ring 0
└──────────────────────────────────────────────────┘
- Kernel owns hardware, memory, scheduling, filesystems, networking.
- Userspace is everything else: shells, daemons, your applications.
- They talk via system calls — a narrow, stable API (
open,read,write,socket,mmap,clone, …).
If something is slow, broken, or mysterious: ask “is this kernel or userspace?” strace shows userspace↔kernel (syscalls). perf / bpftrace look inside the kernel. htop / ps stay in userspace.
Everything is (mostly) a file
The pervasive Unix abstraction. Device, pipe, socket, kernel state — all appear as file paths with read/write semantics.
| Path | What it is |
|---|---|
/dev/sda | Block device (whole disk) |
/dev/null | Discard sink |
/dev/random, /dev/urandom | Entropy sources |
/proc/<pid>/status | Live state of process <pid> |
/proc/cpuinfo, /proc/meminfo | Kernel telemetry |
/sys/class/net/eth0/statistics/rx_bytes | NIC counters |
/sys/class/block/sda/queue/scheduler | I/O scheduler knob |
/proc and /sys aren’t real files — they’re pseudo-filesystems the kernel exposes. That’s why cat /proc/loadavg always shows fresh data.
Exception: sockets and modern device interfaces (DRM, nftables, eBPF) break “everything is a file.” The slogan is 1975-accurate, 2025-approximate.
Filesystem hierarchy (FHS)
/ root
├── bin → /usr/bin user binaries
├── sbin → /usr/sbin system binaries (root)
├── boot kernel + initrd + bootloader
├── etc system-wide config (plain text!)
├── home/<user> users' homes
├── root root's home
├── var
│ ├── log logs (syslog, journald)
│ ├── lib persistent app state (DBs, Docker)
│ └── cache regeneratable caches
├── tmp ephemeral; often tmpfs in RAM
├── usr
│ ├── bin, sbin, lib OS-provided binaries
│ └── local locally-installed (not managed by pkg)
├── opt third-party packaged apps
├── dev device files
├── proc pseudo-fs: processes + kernel
├── sys pseudo-fs: devices + kernel knobs
└── run runtime state (pidfiles, sockets)
Rules of thumb:
- Config →
/etc— all text, all diffable, all backupable. - State →
/var— things that change during runtime (logs, DB data, caches). - Software —
/usr(distro),/usr/local(local compile),/opt(vendor bundle). /tmpis wiped on reboot on most distros./var/tmpsurvives reboots.
Processes
Every running program is a process — a (userspace address space) + (one or more threads) + (open files) + (credentials).
Key properties:
- PID — process ID, assigned at fork. PID 1 is the init system (systemd Fundamentals on modern distros).
- PPID — parent PID. If parent dies, the process is reparented to PID 1 (“orphan”).
- UID / GID — who’s running it. Affects what it can do.
- CWD — current working directory.
- Env — environment variables inherited from parent.
fork + exec
The Unix model for “run a new program”:
fork()— clone the current process (copy-on-write). Now you have two identical ones.exec()— in the child, replace its memory with the new program.
That’s why shells look like this:
bash ← parent
└── fork + exec ls ← new ls process, parent of which was bash
Consequences you’ll hit: cd can’t be a separate binary (would be a fork, couldn’t change your shell’s directory) — it’s a shell built-in. Environment variables set with export are inherited by forks but don’t propagate back.
Process states
ps axo pid,stat,cmd:
| State | Meaning |
|---|---|
R | Running or runnable |
S | Interruptible sleep (waiting for I/O or signal) |
D | Uninterruptible sleep (stuck in a syscall — usually disk) |
Z | Zombie (dead, waiting for parent to reap exit status) |
T | Stopped (e.g. by Ctrl-Z or SIGSTOP) |
Lots of D-state processes → something is blocking on storage. Lots of zombies → a buggy parent isn’t calling wait().
Signals
Asynchronous messages to a process.
| Signal | Default | Typical use |
|---|---|---|
SIGTERM (15) | Terminate (catchable) | Graceful shutdown — default for kill |
SIGINT (2) | Terminate (catchable) | Ctrl-C |
SIGHUP (1) | Terminate | Historically “terminal hung up”; many daemons treat it as “reload config” |
SIGKILL (9) | Terminate | Uncatchable — last resort |
SIGSTOP (19) | Stop | Uncatchable — pause process |
SIGUSR1/2 | Custom | App-defined |
Rule: always try kill <pid> (SIGTERM) first. Only kill -9 when graceful shutdown fails.
File permissions (the quick version)
See Linux Permissions and Users for the full treatment. In 30 seconds:
-rwxr-xr-- 1 alice devs 4096 Apr 24 file
│││││││││││
│└┬┘└┬┘└┬┘
│ u g o user / group / other
└── type: -file d-dir l-symlink c-char b-block s-socket p-pipe
Each of u/g/o has read, write, execute. Numeric shortcut: r=4, w=2, x=1.
chmod 755 file→ rwxr-xr-xchmod 644 file→ rw-r—r—chmod +x file→ make executablechown alice:devs file→ change owner
The shell
A shell is a command-line program (itself running in userspace) that reads your input, forks, and execs programs. bash is the default on most distros; zsh is popular for interactive use; sh is the POSIX lowest-common-denominator.
Streams and redirection
Every process has three standard streams:
| FD | Name | Default |
|---|---|---|
| 0 | stdin | keyboard |
| 1 | stdout | terminal |
| 2 | stderr | terminal |
Redirect them:
cmd > out.txt # stdout → file (overwrite)
cmd >> out.txt # stdout → file (append)
cmd 2> err.txt # stderr → file
cmd > out 2>&1 # both to file
cmd &> both.log # bash shortcut: both
cmd < input.txt # stdin from file
cmd1 | cmd2 # stdout of cmd1 → stdin of cmd2 (pipe)
cmd1 |& cmd2 # both streams of cmd1 → stdin of cmd2Pipes are the heart of Unix productivity — small tools, composed.
Exit codes
Every command returns 0 (success) or nonzero (failure). Check with $?:
cmd && echo "worked" # run only if success
cmd || echo "failed" # run only if failure
cmd1 ; cmd2 # run both regardlessPackage management (distro families)
| Family | Example distros | Package manager | Format |
|---|---|---|---|
| Debian | Debian, Ubuntu, Kali | apt, dpkg | .deb |
| Red Hat | RHEL, Rocky, Alma, Fedora | dnf / yum, rpm | .rpm |
| Arch | Arch, Manjaro | pacman | .pkg.tar.zst |
| Alpine | Alpine (containers) | apk | .apk |
| SUSE | openSUSE, SLES | zypper, rpm | .rpm |
Daily commands are roughly equivalent:
# Debian
apt update && apt upgrade
apt install nginx
apt remove nginx
apt search nginx
# RHEL-family
dnf check-update && dnf upgrade
dnf install nginx
dnf remove nginx
dnf search nginxinit systems
Whatever starts as PID 1 is the init system. On almost every modern distro, that’s systemd (see systemd Fundamentals). Legacy systems use SysVinit (/etc/init.d/* shell scripts) or Upstart. Containers often use a tiny init like tini or rely on the entrypoint as PID 1.
Networking on Linux (the one-screen version)
| Concern | Modern tool | Legacy |
|---|---|---|
| Interfaces, addresses | ip addr, ip link | ifconfig |
| Routing | ip route | route |
| Neighbors (ARP) | ip neigh | arp -a |
| Sockets / listening ports | ss -tulpn | netstat -tulpn |
| Packet capture | tcpdump, tshark | — |
| Firewall | nftables (nft) | iptables, firewalld (frontend) |
| DNS resolution | /etc/resolv.conf, systemd-resolved | — |
| Config | NetworkManager, systemd-networkd, /etc/network/interfaces (Debian legacy), /etc/netplan/* (Ubuntu) | — |
As a network engineer: the ip tool is your friend. ifconfig hides secondary IPs, multiple routing tables, and namespaces — ip shows everything.
Where to look when something is wrong
| Symptom | Look here |
|---|---|
| Service won’t start | systemctl status <svc>, journalctl -u <svc> |
| Mystery slowness | top / htop, vmstat 1, iostat -xz 1 |
| Disk full | df -h, du -sh /* |
| Out of memory / OOM kill | dmesg -T | grep -i oom, journalctl -k |
| Can’t reach host | ip route get <ip>, ping, traceroute, ss -tulpn |
| Something changed “recently” | /var/log/, journalctl --since="1 hour ago" |
Philosophy in one screen
The Unix philosophy is still the best guide:
- Do one thing well.
- Compose small tools with pipes.
- Text streams are the universal interface.
- Prefer flat files over binary formats.
- Silence is golden — a command that succeeded says nothing.
When you find yourself wishing Linux had a single tool to do X, the answer is usually: it has five tools that compose into X.