Service Management with systemd: A Complete Guide

Introduction

In the previous Blog Post, we learned about how processes work in Linux. Now it's time to look at how these processes are wrapped in an additional layer of abstraction: the systemd service.

The commands you've seen so far – ls, mv, rm, ps, and others – run in the foreground, attached to your shell session. You run them, the programs do their job, and then they exit. However, not all programs run like this.

Services (Daemons): Long-running processes that run in the background. These can be things like databases and web servers, but also regular system services like your network manager, your desktop environment, and so on.

These long-running background services are typically started and controlled via an init system such as systemd. The term init refers to the first process your operating system kernel starts, and it is the job of this process to take care of starting any other processes.

systemd services are controlled using a command line utility called systemctl. It will be used for starting and stopping services, for example, to restart a service that's misbehaving or to reload one whose configuration has changed.

Note: This chapter applies to Linux only – macOS and Windows (and even other Unixes) manage processes using different tools. While the concepts are similar, systemd is the most widely used init system in modern Linux distributions.

1. Understanding systemd Architecture

What is systemd?

systemd is a system and service manager for Linux that provides a standard way to manage services. It is now widely used as the default init system for most Linux distributions, replacing older systems like SysV init.

Figure 1: systemd architecture showing PID 1 managing multiple service units

The Role of init (PID 1)

In Linux, init – short for 'initialization' – is the first process that is started when the system boots up. You can find it at PID 1. init is responsible for:

Managing the boot process
Starting all other processes and services configured to run on the system
Re-parenting orphaned processes (processes whose original parent has died)
Ensuring system stability and process lifecycle management

                    $ ps -p 1
                    PID TTY TIME CMD
                    1 ? 00:00:02 systemd

                    # View systemd version
                    $ systemctl --version
                    systemd 249 (249.11-0ubuntu3.12)
                    +PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT
                

Processes vs Services

You can think of a service as packaging around a piece of software that makes it easier to manage as a running process. A service adds convenient features:

Dependencies: Define which services must start before others
Startup Order: Control the boot sequence
Environment Variables: Set specific environment for the process
Resource Limits: Control CPU, memory, and I/O usage
Permissions: Run services as specific users
Restart Policies: Automatically restart failed services
Simple Naming: Reference services by name instead of PID

2. systemctl: The Service Control Command

systemctl is the primary tool you'll use to interact with systemd. It provides a unified interface for managing services, checking their status, and controlling system state.

Basic Service Management

Let's explore the fundamental commands for managing services. We'll use nginx as an example service throughout this section.

Command	Description
`systemctl start <service>`	Start a service immediately
`systemctl stop <service>`	Stop a running service
`systemctl restart <service>`	Stop and then start a service
`systemctl reload <service>`	Reload configuration without stopping
`systemctl status <service>`	Display detailed service status

                    # Start the nginx web server
                    $ sudo systemctl start nginx

                    # Check if it's running
                    $ sudo systemctl status nginx
                    ● nginx.service - A high performance web server
                    Loaded: loaded (/lib/systemd/system/nginx.service; enabled)
                    Active: active (running) since Mon 2025-12-15 10:23:45 UTC; 2min ago
                    Docs: man:nginx(8)
                    Main PID: 12345 (nginx)
                    Tasks: 5 (limit: 4915)
                    Memory: 8.2M
                    CPU: 45ms
                    CGroup: /system.slice/nginx.service
                    ├─12345 nginx: master process /usr/sbin/nginx
                    └─12346 nginx: worker process

                    # Stop the service
                    $ sudo systemctl stop nginx
                

⚠ Important: Only users with root privileges (using sudo) can manage system services with systemd. This is a security measure to prevent unauthorized service manipulation.

Enable and Disable Services

Starting a service makes it run now, but it won't automatically start on the next boot. To control boot-time behavior, use enable and disable:

Command	Description
`systemctl enable <service>`	Configure service to start at boot
`systemctl disable <service>`	Prevent service from starting at boot
`systemctl is-enabled <service>`	Check if service is enabled
`systemctl is-active <service>`	Check if service is currently running

                    # Enable nginx to start on boot
                    $ sudo systemctl enable nginx
                    Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service →
                    /lib/systemd/system/nginx.service.

                    # Check if it's enabled
                    $ systemctl is-enabled nginx
                    enabled

                    # Disable it (won't start on next boot)
                    $ sudo systemctl disable nginx
                    Removed /etc/systemd/system/multi-user.target.wants/nginx.service.
                

Listing Services

systemd manages many different types of units (services, sockets, timers, etc.). Here's how to view them:

                    # List all active services
                    $ systemctl list-units --type=service

                    # List all services (including inactive)
                    $ systemctl list-units --type=service --all

                    # List only failed services
                    $ systemctl list-units --type=service --state=failed

                    # List enabled services
                    $ systemctl list-unit-files --type=service --state=enabled
                

3. Understanding Service States

A systemd service can be in various states throughout its lifecycle. Understanding these states is crucial for effective service management and troubleshooting.

Figure 2: Service state transitions and systemctl commands

Primary Service States

State	Description
`inactive (dead)`	Service is not running
`activating (start)`	Service is in the process of starting
`active (running)`	Service is running successfully
`active (exited)`	One-shot service completed successfully
`deactivating (stop)`	Service is in the process of stopping
`failed`	Service failed to start or crashed
`reloading`	Service is reloading its configuration

Checking Service Status

The systemctl status command provides comprehensive information about a service:

                    $ systemctl status sshd
                    ● ssh.service - OpenBSD Secure Shell server
                    Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
                    Active: active (running) since Mon 2025-12-15 08:15:22 UTC; 2h 30min ago
                    Docs: man:sshd(8)
                    man:sshd_config(5)
                    Process: 1234 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
                    Main PID: 1256 (sshd)
                    Tasks: 1 (limit: 4915)
                    Memory: 4.2M
                    CPU: 125ms
                    CGroup: /system.slice/ssh.service
                    └─1256 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
                

Let's break down what each field means:

Loaded: Shows the unit file path and whether it's enabled
Active: Current state and how long it's been in that state
Docs: Links to documentation
Process: Pre-start processes that ran
Main PID: The primary process ID
Tasks: Number of tasks (threads) the service is running
Memory/CPU: Resource usage
CGroup: Control group hierarchy

4. systemd Unit Files

Services are defined by unit files, which are configuration files that tell systemd exactly how to manage a service. Understanding unit files is essential for creating and customizing services.

Figure 3: Structure of a systemd unit file

Unit File Locations

systemd looks for unit files in several locations, in this order of priority:

/etc/systemd/system/ - System administrator's custom units (highest priority)
/run/systemd/system/ - Runtime units (created dynamically)
/lib/systemd/system/ - Distribution-provided units (lowest priority)

                    # View a service's unit file
                    $ systemctl cat nginx.service

                    # Edit a service (creates override in /etc/systemd/system/)
                    $ sudo systemctl edit nginx.service

                    # Edit the full file (not recommended, use override instead)
                    $ sudo systemctl edit --full nginx.service
                

Anatomy of a Unit File

A unit file is divided into sections, each containing key-value pairs. Here's a complete example:

                    # /etc/systemd/system/myapp.service

                    [Unit]
                    Description=My Custom Application
                    Documentation=https://example.com/docs
                    After=network.target postgresql.service
                    Requires=postgresql.service
                    Wants=redis.service

                    [Service]
                    Type=simple
                    User=myapp
                    Group=myapp
                    WorkingDirectory=/opt/myapp
                    Environment="NODE_ENV=production"
                    EnvironmentFile=/etc/myapp/environment
                    ExecStartPre=/opt/myapp/scripts/pre-start.sh
                    ExecStart=/usr/bin/node /opt/myapp/server.js
                    ExecReload=/bin/kill -HUP $MAINPID
                    Restart=on-failure
                    RestartSec=5s
                    StandardOutput=journal
                    StandardError=journal

                    [Install]
                    WantedBy=multi-user.target
                

[Unit] Section

The [Unit] section contains generic information about the unit:

Directive	Description
`Description`	Human-readable description of the service
`Documentation`	URLs to documentation
`After`	Start this service after the specified units
`Before`	Start this service before the specified units
`Requires`	Hard dependency - if these fail, this service fails
`Wants`	Soft dependency - will start these but won't fail if they fail

[Service] Section

The [Service] section contains service-specific configuration:

Directive	Description
`Type`	Service type: simple, forking, oneshot, dbus, notify, idle
`ExecStart`	Command to start the service
`ExecStop`	Command to stop the service
`ExecReload`	Command to reload configuration
`Restart`	When to restart: no, on-success, on-failure, always
`User/Group`	User and group to run the service as
`WorkingDirectory`	Working directory for the service
`Environment`	Environment variables

Service Types

Understanding service types is crucial for proper service configuration:

simple: Default. Process specified in ExecStart is the main process
forking: Process forks a child and parent exits (traditional daemons)
oneshot: Process exits after completing (used for setup tasks)
notify: Service sends notification when ready (requires sd_notify)
dbus: Service is ready when it acquires a D-Bus name

[Install] Section

The [Install] section defines how the service is enabled:

Directive	Description
`WantedBy`	Target that should pull this service in
`RequiredBy`	Target that requires this service
`Alias`	Alternative names for the service

Common Targets:

multi-user.target - Multi-user system (no GUI)
graphical.target - Multi-user system with GUI
network.target - Network is available

5. Service Dependencies and Ordering

One of systemd's most powerful features is its ability to manage complex service dependencies. This ensures services start in the correct order and handle failures gracefully.

Dependency Types

systemd provides several ways to express dependencies:

Requires vs Wants

Requires: Hard dependency. If the required unit fails, this unit fails too
Wants: Soft dependency. Will try to start the wanted unit, but won't fail if it can't

                    [Unit]
                    # This service MUST have postgresql running
                    Requires=postgresql.service

                    # This service would LIKE to have redis, but can work without it
                    Wants=redis.service
                

After vs Before

Dependencies alone don't control ordering. You must explicitly specify order with After and Before:

                    [Unit]
                    # Start this service AFTER network is ready
                    After=network.target

                    # Start this service BEFORE the web server
                    Before=nginx.service
                

⚠ Common Mistake: Using Requires without After. This will start both services simultaneously, which may cause race conditions. Always use After with Requires or Wants.

Viewing Dependencies

systemd provides tools to visualize and understand service dependencies:

                    # List all dependencies of a service
                    $ systemctl list-dependencies nginx.service

                    # Show reverse dependencies (what depends on this service)
                    $ systemctl list-dependencies --reverse nginx.service

                    # Show all dependencies recursively
                    $ systemctl list-dependencies --all nginx.service
                

Example output showing dependency tree:

nginx.service ● ├─system.slice ● └─basic.target ● ├─paths.target ● ├─slices.target ● ├─sockets.target ● ├─sysinit.target ● └─timers.target

Practical Example: Web Application Stack

Let's create a realistic example showing how to properly configure dependencies for a web application that requires a database and cache:

                    # /etc/systemd/system/webapp.service

                    [Unit]
                    Description=My Web Application
                    After=network.target postgresql.service redis.service
                    Requires=postgresql.service
                    Wants=redis.service

                    [Service]
                    Type=simple
                    User=webapp
                    ExecStart=/usr/bin/python3 /opt/webapp/app.py
                    Restart=on-failure
                    RestartSec=10s

                    # If PostgreSQL fails, this service will also fail and restart
                    # If Redis fails, this service will continue running

                    [Install]
                    WantedBy=multi-user.target
                

6. Troubleshooting Services

Viewing Logs with journalctl

systemd integrates with the journal logging system. Use journalctl to view service logs:

                    # View logs for a specific service
                    $ journalctl -u nginx.service

                    # Follow logs in real-time (like tail -f)
                    $ journalctl -u nginx.service -f

                    # Show only today's logs
                    $ journalctl -u nginx.service --since today

                    # Show logs from last boot
                    $ journalctl -u nginx.service -b

                    # Show last 50 lines
                    $ journalctl -u nginx.service -n 50

                    # Show logs with priority error or higher
                    $ journalctl -u nginx.service -p err
                

Common Issues and Solutions

Service Fails to Start

                    # Check detailed status
                    $ systemctl status myapp.service

                    # Check recent logs
                    $ journalctl -u myapp.service -n 100

                    # Verify unit file syntax
                    $ systemd-analyze verify /etc/systemd/system/myapp.service
                

Service Keeps Restarting

If a service is in a restart loop, check the restart policy and logs:

                    # Check if service is in restart loop
                    $ systemctl status myapp.service
                    Active: activating (auto-restart) (Result: exit-code)

                    # Temporarily stop restart attempts
                    $ sudo systemctl stop myapp.service

                    # Review configuration
                    $ systemctl cat myapp.service | grep Restart
                

Dependency Issues

                    # Check what's blocking a service
                    $ systemctl list-dependencies myapp.service --all

                    # Check reverse dependencies
                    $ systemctl list-dependencies --reverse myapp.service
                

Debugging with systemd-analyze

                    # Verify unit file
                    $ systemd-analyze verify myapp.service

                    # Check security settings
                    $ systemd-analyze security myapp.service

                    # Show critical chain (services that delayed boot)
                    $ systemd-analyze critical-chain
                

7. systemd in Container Environments

Container environments like Docker typically don't run a full init system. This creates some differences in how you manage processes:

Docker Containers

Most Docker containers run a single process directly (PID 1) without systemd. This means:

No service management layer - you start processes directly
No automatic restart of failed processes (unless configured in Docker)
Logs go to stdout/stderr instead of journald
Signal handling is the responsibility of your application

                    # Typical Docker container - no systemd
                    FROM ubuntu:22.04
                    CMD ["/usr/bin/myapp"]

                    # Docker handles restart policy
                    docker run --restart=on-failure myapp
                

Running systemd in Containers

If you need systemd in a container (e.g., for testing), you can configure it, but it requires special setup:

                    # Dockerfile with systemd
                    FROM ubuntu:22.04

                    RUN apt-get update && apt-get install -y systemd

                    # Run systemd as PID 1
                    CMD ["/lib/systemd/systemd"]

                    # Run with required privileges
                    docker run --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:ro myimage
                

Best Practice: For production containers, avoid running systemd. Instead, use Docker's built-in restart policies and orchestration tools like Kubernetes for service management.

8. Practical Examples

Example 1: Creating a Simple Python Web Service

Let's create a complete systemd service for a Python Flask application:

                    # /etc/systemd/system/flask-app.service

                    [Unit]
                    Description=Flask Web Application
                    After=network.target

                    [Service]
                    Type=simple
                    User=www-data
                    Group=www-data
                    WorkingDirectory=/opt/flask-app
                    Environment="FLASK_ENV=production"
                    ExecStart=/usr/bin/python3 /opt/flask-app/app.py
                    Restart=always
                    RestartSec=3

                    # Security hardening
                    PrivateTmp=yes
                    NoNewPrivileges=yes
                    ProtectSystem=strict
                    ReadWritePaths=/opt/flask-app/logs

                    [Install]
                    WantedBy=multi-user.target
                

                    # Enable and start the service
                    $ sudo systemctl daemon-reload
                    $ sudo systemctl enable flask-app.service
                    $ sudo systemctl start flask-app.service
                    $ sudo systemctl status flask-app.service
                

Example 2: Database Backup Service (Timer)

systemd timers can replace cron jobs. Here's a daily database backup:

                    # /etc/systemd/system/db-backup.service

                    [Unit]
                    Description=Database Backup Service
                    Wants=postgresql.service

                    [Service]
                    Type=oneshot
                    User=postgres
                    ExecStart=/usr/local/bin/backup-database.sh
                    StandardOutput=journal
                    StandardError=journal
                

                    # /etc/systemd/system/db-backup.timer

                    [Unit]
                    Description=Daily Database Backup Timer
                    Requires=db-backup.service

                    [Timer]
                    OnCalendar=daily
                    OnCalendar=02:00
                    Persistent=true

                    [Install]
                    WantedBy=timers.target
                

                    # Enable and start the timer
                    $ sudo systemctl daemon-reload
                    $ sudo systemctl enable db-backup.timer
                    $ sudo systemctl start db-backup.timer

                    # List all timers
                    $ systemctl list-timers
                

Example 3: Multi-Instance Services

You can create template units that can be instantiated multiple times:

                    # /etc/systemd/system/worker@.service

                    [Unit]
                    Description=Worker Instance %i
                    After=network.target

                    [Service]
                    Type=simple
                    User=worker
                    ExecStart=/usr/bin/worker --instance=%i
                    Restart=always

                    [Install]
                    WantedBy=multi-user.target
                

                    # Start multiple instances
                    $ sudo systemctl start worker@1.service
                    $ sudo systemctl start worker@2.service
                    $ sudo systemctl start worker@3.service

                    # Enable all instances
                    $ sudo systemctl enable worker@{1..3}.service
                

Summary

In this Post, we've learned how to manage Linux services with systemd. Let's recap the key points:

Key Takeaways:

systemd is the modern init system (PID 1) that manages services in most Linux distributions
systemctl is the primary command for controlling services
Services are defined by unit files in /etc/systemd/system/ or /lib/systemd/system/
Dependencies and ordering are controlled with Requires, Wants, After, and Before
Use journalctl to view service logs
Always reload systemd configuration with daemon-reload after editing unit files

Essential Commands Reference

                    # Service control
                    systemctl start|stop|restart|reload <service>
                    systemctl enable|disable <service>
                    systemctl status <service>

                    # Listing and inspection
                    systemctl list-units --type=service
                    systemctl list-dependencies <service>
                    systemctl cat <service>

                    # Logs
                    journalctl -u <service>
                    journalctl -u <service> -f

                    # Configuration
                    systemctl daemon-reload
                    systemctl edit <service>

                    # Analysis
                    systemd-analyze blame
                    systemd-analyze verify <service>
                    systemd-analyze security <service>
                