Skip to content

Lab: systemd Failures

Pairs with: Understanding systemd

Environment: A single disposable VM.

Starting conditions

  1. Create a test service, exactly as in the parent article's exercise:
# /etc/systemd/system/labapp.service
[Unit]
Description=Lab Test Application
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/sleep infinity
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now labapp.service
  1. Confirm systemctl status labapp shows active (running).

Instructions

Part 1 — Break it, deliberately

Introduce one of the following faults:

  • Crash loop: change ExecStart to a command that fails immediately and consistently, e.g. /bin/false, keeping Restart=on-failure and a short RestartSec. Run daemon-reload and restart the unit.
  • Dependency failure: add Requires=nonexistent.service to the [Unit] section (referencing a unit that doesn't exist), daemon-reload, then try to start it.
  • Wrong user: add User=nonexistentuser to [Service] (a user account that doesn't exist on this system), daemon-reload, restart.

Part 2 — Diagnose it

Using systemctl status labapp and journalctl -u labapp, determine:

  1. Is the unit considered failed, or is it stuck in some other state?
  2. What specific error does journalctl show, and does it point at the actual root cause directly, or do you need to reason about it?

Expected results

  • Crash loop fault: systemctl status will show rapid restart attempts; if restarts exceed the default StartLimitBurst (5 within 10 seconds, by default) within the window, systemd gives up and the unit ends in failed state with Start request repeated too quickly in the journal — a very specific, greppable message.
  • Dependency failure fault: systemctl start labapp fails immediately with the dependency unit reported as not found; journalctl will reference the missing unit by name directly.
  • Wrong user fault: the service fails to start with a status=217/USER (or similar) exit code visible in systemctl status's own output, without needing to dig into journalctl at all for this particular one — the exit code itself is the diagnostic.

Diagnosis checklist

  • [ ] Did you check systemctl status for the specific status=<code>/<reason> on the last exit, not just whether it says failed?
  • [ ] For the crash-loop fault: did you recognize StartLimitBurst/StartLimitIntervalSec as the mechanism that eventually stops the thrashing, rather than it looping forever?
  • [ ] Did journalctl -u labapp --since "5 min ago" show enough history to see the pattern (rapid repeated starts), or did you need --since adjusted to capture it?

Success criteria

Given any of the three faults (without being told which), you can state the specific root cause — not just "it's failed" — using only systemctl status and journalctl -u labapp, within one minute.

Cleanup / rollback

sudo systemctl disable --now labapp.service
sudo rm /etc/systemd/system/labapp.service
sudo systemctl daemon-reload
sudo systemctl reset-failed labapp.service   # clear any lingering "failed" state record

Confirm systemctl status labapp now reports Unit labapp.service could not be found before considering the environment cleaned up.