Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

System Services and Configuration

Modern Linux distributions use an “init system” to control system operations, primarily systemd. Systemd manages services, boot processes, networking, and more.

If you are new to Linux, think of systemd as the traffic controller for background services. It decides what starts, when it starts, and what restarts after failures.

A Simple Mental Model

  • Service unit (.service): a background process such as SSH or a web server.
  • Target unit (.target): a grouped system state, similar to a boot stage.
  • Timer unit (.timer): a scheduled trigger, similar to cron jobs.

This model is enough to diagnose most beginner issues.


Controlling Services and Units

systemctl is the primary tool for managing services and units in systemd. Units can represent services, device drivers, network mounts, timers (similar to crontab), and more.

Common systemctl Commands:

  • systemctl status servicename - View the current status of a service.
  • systemctl start servicename - Start a service.
  • systemctl stop servicename - Stop a service.
  • systemctl restart servicename - Restart a service (stop and start).
  • systemctl reload servicename - Reload the configuration file without restarting the service.
  • systemctl enable servicename - Enable a service to start at system boot.
  • systemctl is-enabled servicename - Check if a service is enabled at startup.
  • systemctl is-active servicename - Check if a service is running and active.
  • systemctl list-units - List all running systemd units.
  • systemctl list-units --all - List all units, both active and inactive.
  • systemctl list-units --all --state=inactive - List all inactive units.
  • systemctl list-units --all --type=service - List all units of type “service.”

Unit File Locations

Systemd unit files are typically located in the following directories:

  • /usr/lib/systemd - The main location for system-created unit files.
  • /etc/systemd/system - Contains system-wide unit files, often symbolic links to /usr/lib/systemd. This directory has top priority when reading unit files.
  • ~/.config/systemd/user/ - Contains user-specific unit files. This directory is not created by default and requires the --user option for systemctl commands.

Example System Unit File

Below is an example of a system unit file:

[Unit]
Description=service_description
After=network.target

[Service]
ExecStart=path_to_executable
Type=forking

[Install]
WantedBy=default.target

For a full reference, Systemd Unit Documentation https://links.thelinuxbook.com/systemd.


User-Based Systemd Services and Unit Files

User-specific unit files are stored in the ~/.config/systemd/user/ directory. These files can be managed using the --user option with systemctl.

Example Command:

systemctl --user start usercreatedfile.service

This command starts a user-created service file located in the user’s home directory.

Example User Unit File:

[Unit]
Description=Run service as user
DefaultDependencies=no
After=network.target

[Service]
Type=simple
User=titus
Group=users
ExecStart=/home/titus/scripts/startup_script.sh
TimeoutStartSec=0
RemainAfterExit=yes

[Install]
WantedBy=default.target

Journalctl and Logging (Finding Error Messages)

journalctl is a powerful tool for viewing and analyzing system logs managed by systemd. It allows you to filter logs by various criteria, such as time, service, or priority.

Common journalctl Commands:

  • journalctl -u servicename - View logs for a specific service.
  • journalctl -b - View logs from the current boot.
  • journalctl -p err - View only error messages.
  • journalctl --since "2024-01-01" - View logs since a specific date.
  • journalctl -f - Follow the log output in real-time (similar to tail -f).

journalctl is usually your first stop, but good diagnostics often requires more than one tool.

Common status patterns to recognize quickly:

  • active (running): service is healthy.
  • inactive (dead): service is stopped.
  • failed: service started and then crashed or exited with an error.

A Practical Troubleshooting Flow

  1. Confirm what failed:
    • systemctl status servicename
    • This often shows the failing command, exit code, and the last few log lines.
  2. Pull focused logs:
    • journalctl -u servicename -b --no-pager
    • Add -n 100 to limit output while debugging.
  3. Check for broader system failures:
    • systemctl --failed
    • This quickly reveals related units that may have also failed.
  4. Check kernel-level messages:
    • dmesg -T | tail -n 50
    • Useful for hardware, driver, disk, memory, USB, and filesystem errors.
  5. Reproduce the issue and watch logs live:
    • Terminal 1: journalctl -f
    • Terminal 2: run the failing command or restart the service.

Safe Recovery When a Service Breaks Boot

If a service causes repeated boot problems, disable it from a recovery shell and reboot:

sudo systemctl disable servicename
sudo systemctl stop servicename
sudo reboot

After the system is stable, inspect logs and re-enable only when the root cause is fixed.

Beyond journalctl: Other Useful Diagnostic Commands

  • systemctl status servicename - Shows unit state, recent logs, and error summaries.
  • systemctl list-units --failed - Lists failed units only.
  • dmesg -T - Shows kernel ring buffer with human-readable timestamps.
  • dmesg -T | grep -i "error\|fail\|warn" - Quickly scan for critical kernel events.
  • lsblk -f - Verify disks, partitions, and mount points when storage is suspect.
  • df -h - Confirm you are not out of disk space.
  • free -h - Check memory pressure.
  • ip a and ip route - Validate interface and routing state for network issues.

Copy/Paste Errors the Right Way

When asking for help online, copy the exact error text instead of paraphrasing.

  • Include command + output together:
    • systemctl status nginx --no-pager
    • journalctl -u nginx -b -n 100 --no-pager
  • Keep the surrounding context (10-30 lines before and after the main error).
  • Preserve exact capitalization, punctuation, and file paths.
  • Remove secrets before sharing (tokens, passwords, internal hostnames, private IPs).

If you need to save output to a file for sharing:

systemctl status servicename --no-pager > status.txt
journalctl -u servicename -b -n 200 --no-pager > logs.txt
dmesg -T | tail -n 200 > dmesg.txt

Finding Solutions Efficiently

Search using the most specific part of the error message in quotes.

  • Good: "Failed to start OpenSSH server daemon"
  • Better: include context, distro, and version:
    • "Failed to start OpenSSH server daemon" ubuntu 24.04 systemd

Prioritize results from:

  • Distribution docs (Arch Wiki, Debian/Ubuntu docs, Fedora docs)
  • Man pages (man systemctl, man journalctl, man dmesg)
  • Project docs and issue trackers

As you test fixes, change one thing at a time and retest. This avoids “mystery fixes” and helps you learn exactly what solved the problem.

Last change: