waldek
/
linux_course_doc
1
0
Fork 0
Code Issues 2 Pull Requests Projects Releases Wiki Activity
linux_course_doc/modules/qualifying/learning_systemd.md

28 KiB
Raw Blame History

Systemd

What is systemd?

Systemd is a collection of programs that aim to unify the service configuration and behavior on most modern Linux distributions. All of the distributions we've used up until now come with systemd and we've been manipulating most of our servers and services via systemctl which is the standard command line interface to systemd. It's worth pointing out that systemd is not just an additional piece of software that is added to your computer. You should see it as a sort of glue that ties the system together as it's responsible for launching and monitoring all services you run on your server.

Some history

As with most things Linux there are multiple alternatives to systemd and believe it or not, the introduction (around 2015) of systemd to Debian was a controversial moment. A lot of online debates were had to discuss the pro's and cons and Debian was even forked to remove systemd all together.

Devuan GNU+Linux is a fork of Debian without systemd that allows users to reclaim control over their system by avoiding unnecessary entanglements and ensuring Init Freedom.

You can be for or against systemd but the current reality is that it is the most widely used init system around today. This can, and probably will, change in the future but for now the world is run by systemd.

The basics

During the numerous hours you've spent using htop you have probably noticed the first process is often /lib/systemd/systemd --system on Debian machines. On Raspberry Pi's that first process is most likely /sbin/init but a closer look at this program shows the following.

pi@camone:~ $ pgrep -a systemd
1 /sbin/init
122 /lib/systemd/systemd-journald
150 /lib/systemd/systemd-udevd
295 /lib/systemd/systemd-timesyncd
378 /lib/systemd/systemd-logind
16414 /lib/systemd/systemd --user
pi@camone:~ $ which /sbin/init
/sbin/init
pi@camone:~ $ ls -l /sbin/init
lrwxrwxrwx 1 root root 20 Apr  1 14:57 /sbin/init -> /lib/systemd/systemd
pi@camone:~ $

Every running Linux computer must have a first process. But where does this first process come from? Below you can see a nice graph of the boot sequence of a standard Linux machine (taken from the Debian system administrator handbook).

startup sequence

By default the Linux kernel will run the init program but this can be overridden by passing an argument to the kernel upon boot. For those who have played around with the broken machines this is probably no real news. At the last stage of the boot sequence, systemd takes over and launches all services that are enabled for the requested runlevel. The runlevel might be new to you but we'll come back to that in a minute.

Interfacing with systemd

Your main tool to talk to systemd is systemctl. It's sort of a client to the systemd server. The most used commands, that you probably know by hearth, are:

sudo systemctl start sshd.service
sudo systemctl stop sshd.service
sudo systemctl restart sshd.service
sudo systemctl status sshd.service
sudo systemctl enable sshd.service
sudo systemctl disable sshd.service

Just knowing these will get you a long way but there are a few more handy commands to push it all a bit further.

Beyond the basics

A deeper look into what's available

If you invoke just sudo systemctl it lists all the units that are active. It's actually a shortcut to sudo systemctl list-units. You'll be confronted with an interface, less, that you know pretty well so have a look around and maybe search for some keywords.

At the bottom of the pager you'll see a few hints that point you to other commands that show even more output. When we disable a server such as sshd it's configuration files are not changed at all as the server never tries to start itself. Systemd is responsible for that so if we want to see all servers available on our system we type sudo systemctl list-unit-files which gives a clear table, also via less, that outlines the current state and vendor state.

We can add more command line arguments to systemctl to narrow down the output a bit. A handy one is --type service to only see services. I advise you to have a read of the man systemctl to grasp the full scope of it's capabilities.

Inspecting a running service

To inspect a running service we can run sudo systemctl status sshd.service. This gives us the following output:

● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2021-07-26 12:14:35 CEST; 2 weeks 6 days ago
       Docs: man:sshd(8)
             man:sshd_config(5)
   Main PID: 576 (sshd)
      Tasks: 1 (limit: 23851)
     Memory: 2.8M
        CPU: 112ms
     CGroup: /system.slice/ssh.service
             └─576 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups

Jul 26 12:14:35 deathstar systemd[1]: Starting OpenBSD Secure Shell server...
Jul 26 12:14:35 deathstar sshd[576]: Server listening on 0.0.0.0 port 22.
Jul 26 12:14:35 deathstar sshd[576]: Server listening on :: port 22.
Jul 26 12:14:35 deathstar systemd[1]: Started OpenBSD Secure Shell server.
Jul 28 20:13:38 deathstar sshd[175321]: Connection closed by authenticating user waldek 192.168.0.222 port 51542 [preauth]
Aug 14 09:05:36 deathstar sshd[1001518]: Connection closed by authenticating user waldek 192.168.0.33 port 35448 [preauth]
Aug 14 09:05:56 deathstar sshd[1001567]: Connection closed by authenticating user waldek 192.168.0.33 port 35636 [preauth]
Aug 14 09:06:20 deathstar sshd[1001648]: Connection closed by authenticating user waldek 192.168.0.236 port 53346 [preauth]

There is quite a bit of interesting information here. There are two blocks of information. At the top we see some details and links to the help about the service in question and at the bottom we see the last eight lines of the server logs. To see how systemd has the sshd service configured we need to have a look at the second line, the one that sais Loaded:. The path that follows is the service file that systemd uses to know how, when and where to run the service. As with most things Linux this is a simple text file we can open up with less, vim or even nano but there is a sweet shortcut supplied by systemd itself!

➜  ~ git:(master) ✗ sudo systemctl cat sshd.service
# /lib/systemd/system/ssh.service
[Unit]
Description=OpenBSD Secure Shell server
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify
RuntimeDirectory=sshd
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target
Alias=sshd.service
➜  ~ git:(master) ✗

Modifying a service

What can we do with these unit files you might ask? Well, we can have a look at the command line arguments for sshd with man sshd. This gives us an overview of all options available to us. One that peaks my interest is the -p argument which allows us to override the port and ignore all ports specified in the configuration file. Let's try it out!

To edit the unit file we need a text editor. There are two ways to do it but we'll go for the most straightforward one first. I'll be using vim to edit the file via sudo vim /lib/systemd/system/ssh.service. Notice the syntax highlighting, nice no? I modified the tenth line so that it reads:

ExecStart=/usr/sbin/sshd -D $SSHD_OPTS -p 2222

Now, how do we take this change into account? Let's restart the service.

  ~ git:(master) ✗ sudo systemctl restart sshd.service 
Warning: The unit file, source configuration file or drop-ins of sshd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
➜  ~ git:(master) ✗

We can see a warning but did the service restart? Let's have a look at the status.

● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2021-08-15 20:00:49 CEST; 38s ago
       Docs: man:sshd(8)
             man:sshd_config(5)
    Process: 1108166 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
   Main PID: 1108167 (sshd)
      Tasks: 1 (limit: 23851)
     Memory: 1.1M
        CPU: 15ms
     CGroup: /system.slice/ssh.service
             └─1108167 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups

Aug 15 20:00:49 deathstar systemd[1]: Starting OpenBSD Secure Shell server...
Aug 15 20:00:49 deathstar sshd[1108167]: Server listening on 0.0.0.0 port 22.
Aug 15 20:00:49 deathstar sshd[1108167]: Server listening on :: port 22.
Aug 15 20:00:49 deathstar systemd[1]: Started OpenBSD Secure Shell server.

Yes it did, but the service is still running on port 22. This is what systemd means by loaded. A configuration file is loaded into memory and used from there. To take changes to unit files into account we need to reload the files that have changed, sort of like we restart sshd when we make changes to it's configuration file but we can't restart systemd as that would freeze our computer. Luckily there is a command to do this and it's written in the warning notice.

➜  ~ git:(master) ✗ sudo systemctl daemon-reload      
➜  ~ git:(master) ✗ sudo systemctl restart sshd.service
➜  ~ git:(master) ✗ sudo systemctl status sshd.service 
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2021-08-15 20:05:53 CEST; 2s ago
       Docs: man:sshd(8)
             man:sshd_config(5)
    Process: 1108694 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
   Main PID: 1108695 (sshd)
      Tasks: 1 (limit: 23851)
     Memory: 1.1M
        CPU: 15ms
     CGroup: /system.slice/ssh.service
             └─1108695 sshd: /usr/sbin/sshd -D -p 2222 [listener] 0 of 10-100 startups

Aug 15 20:05:53 deathstar systemd[1]: Starting OpenBSD Secure Shell server...
Aug 15 20:05:53 deathstar sshd[1108695]: Server listening on 0.0.0.0 port 2222.
Aug 15 20:05:53 deathstar sshd[1108695]: Server listening on :: port 2222.
Aug 15 20:05:53 deathstar systemd[1]: Started OpenBSD Secure Shell server.
➜  ~ git:(master) ✗

Nice! Now, let's first undo our changes and explore the alternative way to modify unit files. Next we do the same changes but in the alternative way. Just as we have a handy shortcut to cat unit files we have one to edit them! I'll run the sudo -E systemctl edit --full sshd.service command, notice the -E, why would I do that? This opens up my editor of choice and I can go ahead an make my changes to line 10 which I add -p 2200 to this time.

➜  ~ git:(master) ✗ sudo -E systemctl edit sshd.service --full
➜  ~ git:(master) ✗ sudo systemctl daemon-reload          
➜  ~ git:(master) ✗ sudo systemctl restart sshd.service    
➜  ~ git:(master) ✗ sudo systemctl status sshd.service     
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/etc/systemd/system/ssh.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2021-08-15 20:24:47 CEST; 4s ago
       Docs: man:sshd(8)
             man:sshd_config(5)
    Process: 1111232 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
   Main PID: 1111233 (sshd)
      Tasks: 1 (limit: 23851)
     Memory: 1.1M
        CPU: 15ms
     CGroup: /system.slice/ssh.service
             └─1111233 sshd: /usr/sbin/sshd -D -p 2200 [listener] 0 of 10-100 startups

Aug 15 20:24:47 deathstar systemd[1]: Starting OpenBSD Secure Shell server...
Aug 15 20:24:47 deathstar sshd[1111233]: Server listening on 0.0.0.0 port 2200.
Aug 15 20:24:47 deathstar sshd[1111233]: Server listening on :: port 2200.
Aug 15 20:24:47 deathstar systemd[1]: Started OpenBSD Secure Shell server.
➜  ~ git:(master) ✗

Notice something different here? The location of the unit file is no longer /lib/systemd/system/ssh.service but /etc/systemd/system/ssh.service. This is the actual preferred way of modifying unit files supplied by your distribution because if at some point in the future your distro changes it's configuration file and you update, you'll overwrite your custom changes! (see this post on serverfault) Think of the similar situation we encountered with /etc/dnsmask.d/ when installing a pihole. What if you want to revert back to file supplied by Debian? A quick sudo systemctl revert sshd.service should do the trick! Don't forget to daemon-reload when you want to restart the service.

Writing your own service files

Imagine we want to run a custom server each time the machine boots. Here systemd comes to the rescue, plus we can run them as ourselves and don't need to interfere with the standard system services. Let's give this a go!

A simple example to a server would be a small python3 webserver. Let's create a directory in our home called website. We can do this with mkdir ~/website. In this folder we'll make an index.html file where we add the content of our website to. You can write anything you want, in html or plaintext. To spin up a quick webserver we can use the http.server class from the standard library. I must note that it's not a production proof server and should only be used for small testing purposes (and for our example).

➜  website git:(master) ✗ python3 -m http.server 8080
Serving HTTP on 0.0.0.0 port 8080 (http://0.0.0.0:8080/) ...
127.0.0.1 - - [15/Aug/2021 20:45:30] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [15/Aug/2021 20:45:30] code 404, message File not found
127.0.0.1 - - [15/Aug/2021 20:45:30] "GET /favicon.ico HTTP/1.1" 404 -

The website is now up and running and we can see all requests logged to the command line. Go to http://localhost:8080 to see your website, or to the IP address of one of the other students to see the logs grow.

Right, we're happy with our service and we would like to offer it permanently. In order to do so we need to create our own unit file and we can do this in two locations. The first one is /etc/systemd/system/ which houses most of our system services but users can have their own services! In order to create your own service, without root privileges, you can add unit files to ~/.local/share/systemd/user. You will probably have to create this directory. In this directory you can add as many .service files as you want. For now we'll just make one called website.service where we need to define some things in.

[Unit]
Description=Our own webserver

[Service]
WorkingDirectory=/home/waldek/website
ExecStart=/usr/bin/python3 -m http.server 8080

[Install]
WantedBy=default.target

Next we need to enable and start our service. Notice that I'm not using sudo and that I added the --user argument.

systemctl --user enable website.service
systemctl --user start website.service

And we can inspect the logs via the trusted status argument as such.

➜  ~ git:(master) ✗ systemctl --user status website.service
● website.service - Our own webserver
     Loaded: loaded (/home/waldek/.local/share/systemd/user/website.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2021-08-15 20:58:15 CEST; 5min ago
   Main PID: 1114451 (python3)
      Tasks: 1 (limit: 23851)
     Memory: 8.8M
        CPU: 92ms
     CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/website.service
             └─1114451 /usr/bin/python3 -m http.server 8080

Aug 15 20:58:15 deathstar systemd[585]: Started Our own webserver.
Aug 15 20:58:23 deathstar python3[1114451]: 127.0.0.1 - - [15/Aug/2021 20:58:23] "GET / HTTP/1.1" 304 -
Aug 15 20:58:24 deathstar python3[1114451]: 127.0.0.1 - - [15/Aug/2021 20:58:24] "GET / HTTP/1.1" 304 -
Aug 15 20:58:24 deathstar python3[1114451]: 127.0.0.1 - - [15/Aug/2021 20:58:24] "GET / HTTP/1.1" 304 -
Aug 15 20:58:25 deathstar python3[1114451]: 127.0.0.1 - - [15/Aug/2021 20:58:25] "GET / HTTP/1.1" 304 -
➜  ~ git:(master) ✗

For those that want to dive deeper into the syntax of the configuration file you should have a look at the output of systemctl --user show website.service which list all of the hidden settings that are predefined for a service. To see what you can change them to, have a look here.

Deep dive into the logs

All logs made you systemd go into the /var/log/daemon.log file by default. You can override this but I would highly advise you not to do it as there are special tools that come with systemd to inspect the logs, plus all logs in one place is quite handy for grepping. Have a look at the file and you should see a similar output.

Aug 15 20:24:44 deathstar systemd[1]: Reloading.
Aug 15 20:24:47 deathstar systemd[1]: Stopping OpenBSD Secure Shell server...
Aug 15 20:24:47 deathstar systemd[1]: ssh.service: Succeeded.
Aug 15 20:24:47 deathstar systemd[1]: Stopped OpenBSD Secure Shell server.
Aug 15 20:24:47 deathstar systemd[1]: Starting OpenBSD Secure Shell server...
Aug 15 20:24:47 deathstar systemd[1]: Started OpenBSD Secure Shell server.
Aug 15 20:57:38 deathstar systemd[585]: Started VTE child process 1114299 launched by gnome-terminal-server process 1027574.
Aug 15 20:58:02 deathstar systemd[585]: Reloading.
Aug 15 20:58:15 deathstar systemd[585]: Started Our own webserver.
Aug 15 20:58:23 deathstar python3[1114451]: 127.0.0.1 - - [15/Aug/2021 20:58:23] "GET / HTTP/1.1" 304 -
Aug 15 20:58:24 deathstar python3[1114451]: 127.0.0.1 - - [15/Aug/2021 20:58:24] "GET / HTTP/1.1" 304 -
Aug 15 20:58:24 deathstar python3[1114451]: 127.0.0.1 - - [15/Aug/2021 20:58:24] "GET / HTTP/1.1" 304 -
Aug 15 20:58:25 deathstar python3[1114451]: 127.0.0.1 - - [15/Aug/2021 20:58:25] "GET / HTTP/1.1" 304 -

Systemd comes with a specialized program to sift through it's logs called journalctl. Just invoking journalctl will give you the output of the log file in less. A very handy argument you'll probably always use is -e which scrolls to the end of the logs. As an alternative you can add --no-pager which will not pipe to less but just print to STDOUT. To only view a specific service we can add the --unit argument, followed by the service name. For example:

➜  ~ git:(master) ✗ sudo journalctl --unit ssh.service --no-pager --since "1 h 25 min ago"
-- Journal begins at Wed 2021-07-14 22:35:36 CEST, ends at Sun 2021-08-15 21:46:42 CEST. --
Aug 15 20:22:14 deathstar sshd[1110635]: Received signal 15; terminating.
Aug 15 20:22:14 deathstar systemd[1]: Stopping OpenBSD Secure Shell server...
Aug 15 20:22:14 deathstar systemd[1]: ssh.service: Succeeded.
Aug 15 20:22:14 deathstar systemd[1]: Stopped OpenBSD Secure Shell server.
Aug 15 20:22:14 deathstar systemd[1]: Starting OpenBSD Secure Shell server...
Aug 15 20:22:14 deathstar sshd[1110849]: Server listening on 0.0.0.0 port 2222.
Aug 15 20:22:14 deathstar sshd[1110849]: Server listening on :: port 2222.
Aug 15 20:22:14 deathstar systemd[1]: Started OpenBSD Secure Shell server.
Aug 15 20:24:47 deathstar systemd[1]: Stopping OpenBSD Secure Shell server...
Aug 15 20:24:47 deathstar sshd[1110849]: Received signal 15; terminating.
Aug 15 20:24:47 deathstar systemd[1]: ssh.service: Succeeded.
Aug 15 20:24:47 deathstar systemd[1]: Stopped OpenBSD Secure Shell server.
Aug 15 20:24:47 deathstar systemd[1]: Starting OpenBSD Secure Shell server...
Aug 15 20:24:47 deathstar sshd[1111233]: Server listening on 0.0.0.0 port 2200.
Aug 15 20:24:47 deathstar sshd[1111233]: Server listening on :: port 2200.
Aug 15 20:24:47 deathstar systemd[1]: Started OpenBSD Secure Shell server.
➜  ~ git:(master) ✗

To understand the --since argument I advise you to read the man systemd.time pages. An argument you'll often see suggested online is -x. It adds more verbose output to debug issues. The manpage documentation is below for reference purpose for reference purposes.

-x, --catalog
    Augment log lines with explanation texts from the message catalog. This will add explanatory help texts to log messages
    in the output where this is available. These short help texts will explain the context of an error or log event,
    possible solutions, as well as pointers to support forums, developer documentation, and any other relevant manuals. Note
    that help texts are not available for all messages, but only for selected ones. For more information on the message
    catalog, please refer to the Message Catalog Developer Documentation[5].

    Note: when attaching journalctl output to bug reports, please do not use -x.

Last but not least, the -f argument does a live stream of the log so you can debug on the fly. This can be very handy in a tmux session. For more information I highly advise the man pages with man journalctl!

A sidetrack into cron

But what if we want to run a quick script or command every day at midnight? Like an email report of the system status, or a apt update? This can also be done with systemd but the classic way of doing this is via cron. As always, have a look at man cron and when you're finished you'll know you want to read the man crontab as well.

In short, every user can have a crontab which is a list of command to execute at certain intervals. To inspect your own crontab, just execute crontab -e which will open your editor of choice. Read through the comments, it's quite self explanatory no? Only the timestamp syntax is quite annoying in my opinion but there is a handy website to help you understand it a bit better. To have a command executed every minute you add the following.

* * * * * echo "helloword" >> /tmp/coucou

The root user has his own crontab you can edit with sudo crontab -e To do an apt update every day at midnight you would add the following.

0 0 * * * apt update

I must note that this is not really the best way to accomplish automatic update and upgrades. Have a look here for better alternatives.

Systemd timers

As you can probably see, cron is a very basic but powerful way of scheduling actions. So people really like the simplicity bit for others a bit more control is desired, hence man systemd.timer. We can list all current timers with the following command.

➜  ~ git:(master) ✗ systemctl list-timers --no-pager
NEXT                       LEFT          LAST                       PASSED       UNIT                       ACTIVATES
Mon 2021-08-16 00:00:00 C… 1h 40min left Sun 2021-08-15 00:00:13 C… 22h ago      logrotate.timer            logrotate.service
Mon 2021-08-16 00:00:00 C… 1h 40min left Sun 2021-08-15 00:00:13 C… 22h ago      man-db.timer               man-db.service
Mon 2021-08-16 06:52:37 C… 8h left       Sun 2021-08-15 06:34:23 C… 15h ago      apt-daily-upgrade.timer    apt-daily-upgrade.service
Mon 2021-08-16 12:45:13 C… 14h left      Sun 2021-08-15 12:45:13 C… 9h ago       systemd-tmpfiles-clean.ti… systemd-tmpfiles-clean.ser…
Mon 2021-08-16 12:46:51 C… 14h left      Sun 2021-08-15 19:56:29 C… 2h 23min ago apt-daily.timer            apt-daily.service
Sun 2021-08-22 03:10:26 C… 6 days left   Sun 2021-08-15 03:10:52 C… 19h ago      e2scrub_all.timer          e2scrub_all.service

6 timers listed.
Pass --all to see loaded but inactive timers, too.
➜  ~ git:(master) ✗

To list your own timers you add the --user argument.

➜  ~ git:(master) ✗ systemctl --user list-timers --no-pager
NEXT LEFT LAST PASSED UNIT ACTIVATES

0 timers listed.
Pass --all to see loaded but inactive timers, too.
➜  ~ git:(master) ✗

Let's add one! In order to create a timer, we need a service that we can run so let's do that first. In the same folder as before I'll create a monitor.service file and will add the following to it.

➜  user git:(master) ✗ systemctl --user cat monitor.service
# /home/waldek/.local/share/systemd/user/monitor.service
[Unit]
Description=Doing some monitoring
Wants=monitor.timer

[Service]
Type=oneshot
ExecStart=/usr/bin/ps u

[Install]
WantedBy=multi-user.target
➜  user git:(master) ✗

Now, let's test the service.

➜  user git:(master) ✗ systemctl --user start monitor.service 
➜  user git:(master) ✗ systemctl --user status monitor.service
● monitor.service - Doing some monitoring
     Loaded: loaded (/home/waldek/.local/share/systemd/user/monitor.service; disabled; vendor preset: enabled)
     Active: inactive (dead)

Aug 15 22:29:08 deathstar ps[1122931]: waldek   1099261  0.0  0.0  16300  9880 pts/2    Ss   17:54   0:00 zsh
Aug 15 22:29:08 deathstar ps[1122931]: waldek   1101477  0.0  0.0  17468 11496 pts/3    Ss   18:31   0:06 zsh
Aug 15 22:29:08 deathstar ps[1122931]: waldek   1117521  0.0  0.0  16744 10068 pts/4    Ss   21:38   0:01 zsh
Aug 15 22:29:08 deathstar ps[1122931]: waldek   1118424  0.0  0.0  17120 10052 pts/5    Ss+  21:44   0:00 zsh
Aug 15 22:29:08 deathstar ps[1122931]: waldek   1119048  0.0  0.0  40788  7956 pts/4    S+   21:50   0:00 journalctl -f
Aug 15 22:29:08 deathstar ps[1122931]: waldek   1121204  1.9  0.1  45280 31108 pts/2    S+   22:11   0:20 vim learning_systemd.md
Aug 15 22:29:08 deathstar ps[1122931]: waldek   1122718  0.2  0.0  12848  8716 pts/6    Ss+  22:27   0:00 zsh
Aug 15 22:29:08 deathstar ps[1122931]: waldek   1122930  0.0  0.0  10072  1168 pts/3    S+   22:29   0:00 systemctl --user start monit>
Aug 15 22:29:08 deathstar systemd[585]: monitor.service: Succeeded.
Aug 15 22:29:08 deathstar systemd[585]: Finished Doing some monitoring.

OK, that seems to work well, so now we want to create the timer that will run this service every minute. In order to do this we create a monitor.timer file with the following content. The onCalendar=*-*-* *:*:00 make it run every minute. You can read more about the syntax in the man systemd.time pages.

➜  user git:(master) ✗ systemctl --user cat monitor.timer  
# /home/waldek/.local/share/systemd/user/monitor.timer
[Unit]
Description=Doing some timely monitoring
Requires=monitor.service

[Timer]
Unit=monitor.service
OnCalendar=*-*-* *:*:00

[Install]
WantedBy=timers.target
➜  user git:(master) ✗

Once the timer is in place you should start the service with systemctl --user start monitor.service. There is no need to start or enable the monitor.timer file as the link between them is in the monitor.service file via the Wants=monitor.timer configuration line. If you now watch your log in real time with journalctl -f --user-unit monitor.service you should see your service executing every minute!

Pro's and cons

The following advice was taken from the arch wiki.

As a cron replacement Although cron is arguably the most well-known job scheduler, systemd timers can be an alternative.

Benefits The main benefits of using timers come from each job having its own systemd service. Some of these benefits are:

Jobs can be easily started independently of their timers. This simplifies debugging. Each job can be configured to run in a specific environment (see systemd.exec(5)). Jobs can be attached to cgroups. Jobs can be set up to depend on other systemd units. Jobs are logged in the systemd journal for easy debugging. Caveats Some things that are easy to do with cron are difficult to do with timer units alone:

Creation: to set up a timed job with systemd you need to create two files and run systemctl commands, compared to adding a single line to a crontab. Emails: there is no built-in equivalent to cron's MAILTO for sending emails on job failure. See the next section for an example of setting up a similar functionality using OnFailure=. Also note that user timer units will only run during an active user login session by default. However, lingering can enable services to run at boot even when the user has no active login session.