Understanding systemd: Units, Targets, and Service Management¶

systemd is the init system (PID 1) on virtually every mainstream Linux distribution today (Ubuntu, Debian, RHEL/Fedora/CentOS-family, SUSE, Arch). It starts services, manages dependencies between them, and is the first thing to blame — correctly or not — when "the server didn't come back up after a reboot."

How it works¶

systemd organizes everything into units. The most common unit types:

Units are plain text files, normally under /etc/systemd/system/ (local overrides/custom units) or /lib/systemd/system/ (packages installed by your distribution).

Targets replace runlevels¶

$ systemctl list-units --type=target --no-pager
UNIT                    LOAD   ACTIVE SUB    DESCRIPTION
basic.target            loaded active active Basic System
cloud-init.target       loaded active active Cloud-init target
getty.target            loaded active active Login Prompts
graphical.target        loaded active active Graphical Interface
local-fs.target         loaded active active Local File Systems
...

graphical.target and multi-user.target are the modern equivalents of old SysV runlevels 5 and 3. systemctl get-default shows which one boots by default; systemctl set-default multi-user.target switches a server to boot without a GUI.

Inspecting a service¶

$ systemctl status systemd-journald --no-pager -l
● systemd-journald.service - Journal Service
     Loaded: loaded (/lib/systemd/system/systemd-journald.service; static)
     Active: active (running) since Thu 2026-10-15 06:31:47 EEST; 5 days ago
TriggeredBy: ● systemd-journald.socket
       Docs: man:systemd-journald.service(8)
   Main PID: 3446425 (systemd-journal)
     Status: "Processing requests..."
      Tasks: 1 (limit: 19046)
     Memory: 40.8M
        CPU: 1min 19.843s
     CGroup: /system.slice/systemd-journald.service
             └─3446425 /lib/systemd/systemd-journald

Reading this: Loaded shows whether the unit file was parsed successfully and where it lives; Active shows the actual runtime state (active (running), active (exited) for one-shot units, failed, inactive); Main PID is what systemd is actually supervising — if this process dies and the unit has a Restart= directive, systemd restarts it automatically.

A minimal service unit¶

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/myapp
Restart=on-failure
RestartSec=5
User=myapp

[Install]
WantedBy=multi-user.target

• After=network.target — an ordering dependency only. It does not mean the network is necessarily usable yet (DHCP may still be negotiating) — for that, you'd want After=network-online.target and Wants=network-online.target, which is a stronger guarantee on distributions that properly implement it.
• Restart=on-failure — restart only on a non-zero exit or a signal, not on a clean exit 0 (Restart=always restarts unconditionally, even after a clean exit 0; an explicit systemctl stop is never auto-restarted).
• WantedBy=multi-user.target — this is what systemctl enable acts on: it creates a symlink so the unit starts automatically when multi-user.target is reached at boot.

After creating or editing a unit file, systemd does not see it automatically:

sudo systemctl daemon-reload
sudo systemctl enable --now myapp.service

daemon-reload re-reads unit files from disk; enable --now both enables it for future boots and starts it immediately.

Reading logs with journalctl¶

journalctl -u myapp.service --since "10 min ago"
journalctl -u myapp.service -f          # follow, like tail -f
journalctl -p err -b                    # errors only, current boot

Ubuntu/Debian vs. RHEL-family differences¶

The systemd mechanics are identical across distributions — it's the same upstream project. The differences are in surrounding tooling:

Practical exercise¶

1. Create the myapp.service unit above, but point ExecStart at /usr/bin/sleep infinity instead of a real application (safe to test with). Remove the User=myapp line first (or create that account with sudo useradd -r -s /usr/sbin/nologin myapp), otherwise the unit fails with status=217/USER.
2. Enable and start it, then confirm it's running and check its Main PID.
3. Kill the process directly with kill -9 <PID> and watch journalctl -u myapp -f in another terminal to see systemd detect the failure and restart it (because of Restart=on-failure).

How to verify you did it correctly¶

• systemctl status myapp should show Active: active (running) with a new PID after the kill -9, within RestartSec (5 seconds in the example).
• journalctl -u myapp should show a line indicating the process exited due to a signal, followed by a new "Started" entry.

Common mistakes¶

• Editing a unit file and forgetting daemon-reload. systemd keeps using the old, cached version of the unit until you reload.
• Using Restart=always without a sane RestartSec. The default is 100 ms, so a crash-looping service restarts ten times a second, burning CPU and flooding the journal, until systemd's start-rate limit (by default 5 starts in 10 seconds) trips and marks the unit failed. Always pair Restart= with an explicit RestartSec= and tune StartLimitBurst=/StartLimitIntervalSec= to stop retrying after N failures in a time window.
• Assuming enable starts the service. enable only arranges for the service to start at next boot — it does not start it now. Use enable --now or a separate start.

⚠ Dangerous commands¶

• systemctl set-default <target> on a remote server, if you set an invalid or GUI-only target on a headless machine, can leave it unable to reach a usable login prompt after reboot — verify the target exists and is appropriate (multi-user.target, not graphical.target, on a server) before applying.
• systemctl mask a unit that another unit depends on can produce confusing cascading failures — masking (unlike disable) makes a unit impossible to start even as a dependency, until explicitly unmasked.

Related articles¶

• Diagnosing High CPU and Memory Usage
• Linux Filesystem Hierarchy and Core Concepts
• Understanding conntrack and Connection Tracking

Review questions¶

1. What's the practical difference between Restart=on-failure and Restart=always?
2. Why doesn't editing a .service file take effect immediately, and what command fixes that?
3. What does After=network.target guarantee, and what does it not guarantee, compared to After=network-online.target?
4. How would you find out why a service failed to start, using only systemctl and journalctl?