System Services and Configuration
Modern Linux distributions use an “init system” to control system operations, primarily systemd. Systemd manages services, boot processes, networking, and more.
If you are new to Linux, think of systemd as the traffic controller for background services. It decides what starts, when it starts, and what restarts after failures.
A Simple Mental Model
- Service unit (
.service): a background process such as SSH or a web server. - Target unit (
.target): a grouped system state, similar to a boot stage. - Timer unit (
.timer): a scheduled trigger, similar to cron jobs.
This model is enough to diagnose most beginner issues.
Controlling Services and Units
systemctl is the primary tool for managing services and units in systemd. Units can represent services, device drivers, network mounts, timers (similar to crontab), and more.
Common systemctl Commands:
systemctl status servicename- View the current status of a service.systemctl start servicename- Start a service.systemctl stop servicename- Stop a service.systemctl restart servicename- Restart a service (stop and start).systemctl reload servicename- Reload the configuration file without restarting the service.systemctl enable servicename- Enable a service to start at system boot.systemctl is-enabled servicename- Check if a service is enabled at startup.systemctl is-active servicename- Check if a service is running and active.systemctl list-units- List all running systemd units.systemctl list-units --all- List all units, both active and inactive.systemctl list-units --all --state=inactive- List all inactive units.systemctl list-units --all --type=service- List all units of type “service.”
Unit File Locations
Systemd unit files are typically located in the following directories:
/usr/lib/systemd- The main location for system-created unit files./etc/systemd/system- Contains system-wide unit files, often symbolic links to/usr/lib/systemd. This directory has top priority when reading unit files.~/.config/systemd/user/- Contains user-specific unit files. This directory is not created by default and requires the--useroption for systemctl commands.
Example System Unit File
Below is an example of a system unit file:
[Unit]
Description=service_description
After=network.target
[Service]
ExecStart=path_to_executable
Type=forking
[Install]
WantedBy=default.target
For a full reference, Systemd Unit Documentation https://links.thelinuxbook.com/systemd.
User-Based Systemd Services and Unit Files
User-specific unit files are stored in the ~/.config/systemd/user/ directory. These files can be managed using the --user option with systemctl.
Example Command:
systemctl --user start usercreatedfile.service
This command starts a user-created service file located in the user’s home directory.
Example User Unit File:
[Unit]
Description=Run service as user
DefaultDependencies=no
After=network.target
[Service]
Type=simple
User=titus
Group=users
ExecStart=/home/titus/scripts/startup_script.sh
TimeoutStartSec=0
RemainAfterExit=yes
[Install]
WantedBy=default.target
Journalctl and Logging (Finding Error Messages)
journalctl is a powerful tool for viewing and analyzing system logs managed by systemd. It allows you to filter logs by various criteria, such as time, service, or priority.
Common journalctl Commands:
journalctl -u servicename- View logs for a specific service.journalctl -b- View logs from the current boot.journalctl -p err- View only error messages.journalctl --since "2024-01-01"- View logs since a specific date.journalctl -f- Follow the log output in real-time (similar totail -f).
journalctl is usually your first stop, but good diagnostics often requires more than one tool.
Common status patterns to recognize quickly:
active (running): service is healthy.inactive (dead): service is stopped.failed: service started and then crashed or exited with an error.
A Practical Troubleshooting Flow
- Confirm what failed:
systemctl status servicename- This often shows the failing command, exit code, and the last few log lines.
- Pull focused logs:
journalctl -u servicename -b --no-pager- Add
-n 100to limit output while debugging.
- Check for broader system failures:
systemctl --failed- This quickly reveals related units that may have also failed.
- Check kernel-level messages:
dmesg -T | tail -n 50- Useful for hardware, driver, disk, memory, USB, and filesystem errors.
- Reproduce the issue and watch logs live:
- Terminal 1:
journalctl -f - Terminal 2: run the failing command or restart the service.
- Terminal 1:
Safe Recovery When a Service Breaks Boot
If a service causes repeated boot problems, disable it from a recovery shell and reboot:
sudo systemctl disable servicename
sudo systemctl stop servicename
sudo reboot
After the system is stable, inspect logs and re-enable only when the root cause is fixed.
Beyond journalctl: Other Useful Diagnostic Commands
systemctl status servicename- Shows unit state, recent logs, and error summaries.systemctl list-units --failed- Lists failed units only.dmesg -T- Shows kernel ring buffer with human-readable timestamps.dmesg -T | grep -i "error\|fail\|warn"- Quickly scan for critical kernel events.lsblk -f- Verify disks, partitions, and mount points when storage is suspect.df -h- Confirm you are not out of disk space.free -h- Check memory pressure.ip aandip route- Validate interface and routing state for network issues.
Copy/Paste Errors the Right Way
When asking for help online, copy the exact error text instead of paraphrasing.
- Include command + output together:
systemctl status nginx --no-pagerjournalctl -u nginx -b -n 100 --no-pager
- Keep the surrounding context (10-30 lines before and after the main error).
- Preserve exact capitalization, punctuation, and file paths.
- Remove secrets before sharing (tokens, passwords, internal hostnames, private IPs).
If you need to save output to a file for sharing:
systemctl status servicename --no-pager > status.txt
journalctl -u servicename -b -n 200 --no-pager > logs.txt
dmesg -T | tail -n 200 > dmesg.txt
Finding Solutions Efficiently
Search using the most specific part of the error message in quotes.
- Good:
"Failed to start OpenSSH server daemon" - Better: include context, distro, and version:
"Failed to start OpenSSH server daemon" ubuntu 24.04 systemd
Prioritize results from:
- Distribution docs (Arch Wiki, Debian/Ubuntu docs, Fedora docs)
- Man pages (
man systemctl,man journalctl,man dmesg) - Project docs and issue trackers
As you test fixes, change one thing at a time and retest. This avoids “mystery fixes” and helps you learn exactly what solved the problem.
Companion resources