From 2fcc62a90374dffc32f7738410a7026a553af400 Mon Sep 17 00:00:00 2001 From: waldek Date: Mon, 23 Aug 2021 23:10:37 +0200 Subject: [PATCH] adds udev and systemd integration --- .../learning_systemd_udev_integration.md | 359 ++++++++++++++++++ 1 file changed, 359 insertions(+) create mode 100644 modules/qualifying/learning_systemd_udev_integration.md diff --git a/modules/qualifying/learning_systemd_udev_integration.md b/modules/qualifying/learning_systemd_udev_integration.md new file mode 100644 index 0000000..e4329d6 --- /dev/null +++ b/modules/qualifying/learning_systemd_udev_integration.md @@ -0,0 +1,359 @@ +# Integrating udev rules with systemd + +As we saw in [learning udev](./learning_udev.md) we can **run** scripts or commands when certain conditions are met in regards to physical device changes. +The *classic* way to run long running actions, such as a backup, was to have `udev` trigger a script and be done with it. +This behaviour is currently discouraged and will most likely **fail** if your script take too long. +The `man udev` pages state the following. + +``` +Starting daemons or other long-running processes is not allowed; the forked processes, detached or not, will be +unconditionally killed after the event handling has finished. In order to activate long-running processes from udev +rules, provide a service unit and pull it in from a udev device using the SYSTEMD_WANTS device property. See +systemd.device(5) for details. +``` + +It's pretty clear what you can't do, but how to do it *right* is less clear. +The idea is to start a systemd service when a device is plugged in. +Udev will tell systemd to start the service but will delegate the responsibility to systemd so the request is non-blocking. + +## Writing the service + +We start by writing a small script that serves as a placeholder to prove our workflow. +It will just echo a line to `journalctl`. +No need to manage our own logging systemd as we can take full advantage of systemd's built in logging. + +``` +➜ ~ git:(master) ✗ cat backup.sh +#!/bin/bash + +echo "hello world $(date)" +exit 0 +➜ ~ git:(master) ✗ +``` + +We can now take this mini script and turn it into service by creating a `backup.service` file in `/etc/systemd/system`. +Let's do this and populate as such. + +``` +➜ ~ git:(master) ✗ cat /etc/systemd/system/backup.service +[Unit] +Description=Our own backup script + +[Service] +ExecStart=/bin/bash /home/waldek/backup.sh + +[Install] +WantedBy=default.target +➜ ~ git:(master) ✗ +``` + +Next we need to reload systemd and test the service. + +``` +➜ ~ git:(master) ✗ sudo systemctl daemon-reload +➜ ~ git:(master) ✗ sudo systemctl start backup.service +➜ ~ git:(master) ✗ sudo systemctl status backup.service +● backup.service - Our own backup script + Loaded: loaded (/etc/systemd/system/backup.service; disabled; vendor preset: enabled) + Active: inactive (dead) + +Aug 23 21:50:38 deathstar systemd[1]: Started Our own backup script. +Aug 23 21:50:38 deathstar bash[956441]: hello world Mon 23 Aug 21:50:38 CEST 2021 +Aug 23 21:50:38 deathstar systemd[1]: backup.service: Succeeded. +➜ ~ git:(master) ✗ +``` + + +## Writing the udev rule + +It seems to have worked nicely! +Now we will write the udev rule to start our service when a USB stick is plugged in. +We'll start with a very *generic* rule that triggers for each block device that is plugged in. + +``` +➜ ~ git:(master) ✗ cat /etc/udev/rules.d/99-backup_to_usb_stick.rules +SUBSYSTEM=="block", ACTION=="add", ENV{SYSTEMD_WANTS}="backup.service" +➜ ~ git:(master) ✗ +``` + +To test the rule we need to reload the udev rules and inspect the logs of systemd with journalctl. +We can monitor the logs *live* with the `-f` argument if we want. +You'll see the ourput of the placeholder script appear when you plug in any USB stick! +It will also double or even triple trigger, depending on the number of partitions you have on your stick but we know how to fix this. + +``` +➜ ~ git:(master) ✗ sudo udevadm control --reload +➜ ~ git:(master) ✗ sudo journalctl -f --unit backup.service +-- Journal begins at Wed 2021-07-14 22:35:36 CEST, ends at Mon 2021-08-23 21:59:26 CEST. -- +Aug 23 21:59:05 deathstar systemd[1]: Started Our own backup script. +Aug 23 21:59:05 deathstar bash[957988]: hello world Mon 23 Aug 21:59:05 CEST 2021 +Aug 23 21:59:05 deathstar systemd[1]: backup.service: Succeeded. +Aug 23 21:59:05 deathstar systemd[1]: Started Our own backup script. +Aug 23 21:59:05 deathstar bash[957992]: hello world Mon 23 Aug 21:59:05 CEST 2021 +Aug 23 21:59:05 deathstar systemd[1]: backup.service: Succeeded. +``` + +## Fine tuning the udev rule + +I only want my backup do be done to a very specific USB stick I trust. +When I plug it in I can inspect it's attributes and environment variables via `udevadm`. +I'll use the filesystem's UUID to identify the disk. + +``` +➜ ~ git:(master) ✗ sudo udevadm info --name=/dev/sdd1 | grep UUID +E: ID_PART_TABLE_UUID=d9f8a99f-673a-334c-af4f-18697ed888c9 +E: ID_FS_UUID=2469ffe5-e066-476d-805a-cde85a58ea3b +E: ID_FS_UUID_ENC=2469ffe5-e066-476d-805a-cde85a58ea3b +E: ID_PART_ENTRY_UUID=73a25ff8-0e0f-1147-8aaa-7976bf921ced +➜ ~ git:(master) ✗ +``` + +I modify the rule, reload udev and inspect my logs after plugging in the device. +This works like a charm! + +``` +➜ ~ git:(master) ✗ cat /etc/udev/rules.d/99-backup_to_usb_stick.rules +SUBSYSTEM=="block", ENV{ID_FS_UUID}=="2469ffe5-e066-476d-805a-cde85a58ea3b", ACTION=="add", ENV{SYSTEMD_WANTS}="backup.service" +➜ ~ git:(master) ✗ sudo udevadm control --reload +➜ ~ git:(master) ✗ sudo journalctl --unit backup.service +-- Journal begins at Wed 2021-07-14 22:35:36 CEST. -- +Aug 23 22:12:34 deathstar systemd[1]: Started Our own backup script. +Aug 23 22:12:34 deathstar bash[960433]: hello world Mon 23 Aug 22:12:34 CEST 2021 +Aug 23 22:12:34 deathstar systemd[1]: backup.service: Succeeded. +➜ ~ git:(master) ✗ +``` + +Now a **tricky detail**. +The rule works and triggers only once but as my server is a lean machine I do not have automount installed so the script will have to do this for me. +But how will the script *know* which device is the right one? +We can go multiple ways here but I can think of two different ones. + +* hardcode the UUID into the script and look for the proper device +* pass the kernel identified to the service + +Let's go for the second option as it's the more *mature* one. +The full solution requires multiple steps and we'll go over them one at time. +First we'll have to modify our rule once more by adding the following. +This passes the drive name to the service. + +``` +➜ ~ git:(master) ✗ cat /etc/udev/rules.d/99-backup_to_usb_stick.rules +SUBSYSTEM=="block", ENV{ID_FS_UUID}=="2469ffe5-e066-476d-805a-cde85a58ea3b", ACTION=="add", ENV{SYSTEMD_WANTS}="backup@$name.service" +➜ ~ git:(master) ✗ sudo udevadm control --reload +``` + +You should notice two things when looking at the changed rule. + +* the added `@` to the service +* the `$name` + +When the partition we want to mount is `/dev/sde1` the service that will get triggered will be `backup@sde1.service`. +These are called **template** services and in order to create one, you *just* have to have the `@` symbol in the service name. +We'll move our old backup.service and reload systemd as follows. + +``` +➜ ~ git:(master) ✗ sudo mv /etc/systemd/system/backup.service /etc/systemd/system/backup@.service +➜ ~ git:(master) ✗ sudo systemctl daemon-reload +``` + +Inspecting the logs while plugging in our drive now gives us the following output. + +``` +➜ ~ git:(master) ✗ sudo journalctl -f +-- Journal begins at Wed 2021-07-14 22:35:36 CEST. -- +Aug 23 22:40:38 deathstar sudo[964916]: waldek : TTY=pts/6 ; PWD=/home/waldek ; USER=root ; COMMAND=/usr/bin/mv /etc/systemd/system/backup.service /etc/systemd/system/backup@.service +Aug 23 22:40:38 deathstar sudo[964916]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000) +Aug 23 22:40:38 deathstar sudo[964916]: pam_unix(sudo:session): session closed for user root +Aug 23 22:40:49 deathstar sudo[964959]: waldek : TTY=pts/6 ; PWD=/home/waldek ; USER=root ; COMMAND=/usr/bin/systemctl daemon-reload +Aug 23 22:40:49 deathstar sudo[964959]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000) +Aug 23 22:40:49 deathstar systemd[1]: Reloading. +Aug 23 22:40:49 deathstar sudo[964959]: pam_unix(sudo:session): session closed for user root +Aug 23 22:41:45 deathstar kernel: usb 3-1.5: USB disconnect, device number 59 +Aug 23 22:41:48 deathstar sudo[965130]: waldek : TTY=pts/6 ; PWD=/home/waldek ; USER=root ; COMMAND=/usr/bin/journalctl -f +Aug 23 22:41:48 deathstar sudo[965130]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000) +Aug 23 22:41:58 deathstar kernel: usb 3-1.5: new high-speed USB device number 60 using ehci-pci +Aug 23 22:41:58 deathstar kernel: usb 3-1.5: New USB device found, idVendor=0781, idProduct=5591, bcdDevice= 1.00 +Aug 23 22:41:58 deathstar kernel: usb 3-1.5: New USB device strings: Mfr=1, Product=2, SerialNumber=3 +Aug 23 22:41:58 deathstar kernel: usb 3-1.5: Product: SanDisk 3.2Gen1 +Aug 23 22:41:58 deathstar kernel: usb 3-1.5: Manufacturer: USB +Aug 23 22:41:58 deathstar kernel: usb 3-1.5: SerialNumber: 04018a57f24e4dc0135c1f4666b5f2ca4e8e15e65eedfd157b0146cf1427157522b000000000000000000000a03e3c7e008cb3189155810739aa4777 +Aug 23 22:41:58 deathstar kernel: usb-storage 3-1.5:1.0: USB Mass Storage device detected +Aug 23 22:41:58 deathstar kernel: scsi host7: usb-storage 3-1.5:1.0 +Aug 23 22:41:59 deathstar kernel: scsi 7:0:0:0: Direct-Access USB SanDisk 3.2Gen1 1.00 PQ: 0 ANSI: 6 +Aug 23 22:41:59 deathstar kernel: sd 7:0:0:0: Attached scsi generic sg3 type 0 +Aug 23 22:41:59 deathstar kernel: sd 7:0:0:0: [sde] 60088320 512-byte logical blocks: (30.8 GB/28.7 GiB) +Aug 23 22:41:59 deathstar kernel: sd 7:0:0:0: [sde] Write Protect is off +Aug 23 22:41:59 deathstar kernel: sd 7:0:0:0: [sde] Mode Sense: 43 00 00 00 +Aug 23 22:41:59 deathstar kernel: sd 7:0:0:0: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA +Aug 23 22:41:59 deathstar kernel: sde: sde1 +Aug 23 22:41:59 deathstar kernel: sd 7:0:0:0: [sde] Attached SCSI removable disk +Aug 23 22:41:59 deathstar systemd[1]: Started Our own backup script. +Aug 23 22:41:59 deathstar bash[965178]: hello world Mon 23 Aug 22:41:59 CEST 2021 +Aug 23 22:41:59 deathstar systemd[1]: backup@sde1.service: Succeeded. +``` + +Success! +It's the last three lines that are important here. +We can see the right service get's triggered. +Now, how do we recuperate the `$name` variable in our script? + +## Fine tuning the service + +We can use the variables passed through via the service name by modifying the `backup@.service` file. +The argument used is the `%I` when calling our script. +Let's look at the final service file and reload systemd. + +``` +➜ ~ git:(master) ✗ cat /etc/systemd/system/backup@.service +[Unit] +Description=Our own backup script + +[Service] +ExecStart=/bin/bash /home/waldek/backup.sh %I + +[Install] +WantedBy=default.target +➜ ~ git:(master) ✗ sudo systemctl daemon-reload +➜ ~ git:(master) ✗ +``` + +## Fine tuning the script + +The script should now be receiving the partition location as a first argument. +We can echo it out by modifying the script and inspecting the logs via `journalctl`. + +``` +➜ ~ git:(master) ✗ cat backup.sh +#!/bin/bash + +echo "hello world $(date), will backup to $1" +exit 0 +sudo journalctl -f +-- Journal begins at Wed 2021-07-14 22:35:36 CEST. -- +Aug 23 22:49:59 deathstar kernel: sd 7:0:0:0: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA +Aug 23 22:49:59 deathstar kernel: sde: sde1 +Aug 23 22:49:59 deathstar kernel: sd 7:0:0:0: [sde] Attached SCSI removable disk +Aug 23 22:49:59 deathstar systemd[1]: Started Our own backup script. +Aug 23 22:49:59 deathstar bash[966393]: hello world Mon 23 Aug 22:49:59 CEST 2021, will backup to sde1 +Aug 23 22:49:59 deathstar systemd[1]: backup@sde1.service: Succeeded. +``` + +## Mini conclusion + +The entire process might seem a bit convoluted but it's fully in line with the Linux philosophy of separating responsibility. +All the tools are there to chain together a very robust and controlled flow of events. + +## An actual useful script + +Our placeholder script is a good proof of concept to debug the flow of operations but does not do anything useful. +Let's write a simple backup script though. +I like using `rsync` so you'll have to install it if you want to play around with this script. + +``` +➜ ~ git:(master) ✗ cat backup.sh +#!/bin/bash + +SOURCE_DIR="/home/waldek/Documents/freecad/" +MOUNT_POINT="/media/backup_usb_stick" +DST_DIR="$MOUNT_POINT/backups" + +echo "running as $(whoami) will backup to $1" +mkdir -p $MOUNT_POINT && echo "created $MOUNT_POINT" +mount /dev/$1 $MOUNT_POINT && echo "mounted $1" +rsync -av $SOURCE_DIR $DST_DIR && echo "all done" +umount /dev/$1 && echo "unmounted..." +exit 0 +➜ ~ git:(master) ✗ +``` + +Below is the output of two consecutive plug in and out's. +You can see all the files that get backed up the first time around. +The second time around no files are copied because I made no changes to any files. +This is the behaviour of the `-a` argument to `rsync`. + +``` +➜ ~ git:(master) ✗ sudo journalctl -f +-- Journal begins at Wed 2021-07-14 22:35:36 CEST. -- +Aug 23 23:04:40 deathstar sudo[969830]: waldek : TTY=pts/6 ; PWD=/home/waldek ; USER=root ; COMMAND=/usr/bin/journalctl -f +Aug 23 23:04:40 deathstar sudo[969830]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000) +Aug 23 23:04:45 deathstar kernel: usb 3-1.5: new high-speed USB device number 67 using ehci-pci +Aug 23 23:04:45 deathstar kernel: usb 3-1.5: New USB device found, idVendor=0781, idProduct=5591, bcdDevice= 1.00 +Aug 23 23:04:45 deathstar kernel: usb 3-1.5: New USB device strings: Mfr=1, Product=2, SerialNumber=3 +Aug 23 23:04:45 deathstar kernel: usb 3-1.5: Product: SanDisk 3.2Gen1 +Aug 23 23:04:45 deathstar kernel: usb 3-1.5: Manufacturer: USB +Aug 23 23:04:45 deathstar kernel: usb 3-1.5: SerialNumber: 04018a57f24e4dc0135c1f4666b5f2ca4e8e15e65eedfd157b0146cf1427157522b000000000000000000000a03e3c7e008cb3189155810739aa4777 +Aug 23 23:04:45 deathstar kernel: usb-storage 3-1.5:1.0: USB Mass Storage device detected +Aug 23 23:04:45 deathstar kernel: scsi host8: usb-storage 3-1.5:1.0 +Aug 23 23:04:46 deathstar kernel: scsi 8:0:0:0: Direct-Access USB SanDisk 3.2Gen1 1.00 PQ: 0 ANSI: 6 +Aug 23 23:04:46 deathstar kernel: sd 8:0:0:0: Attached scsi generic sg3 type 0 +Aug 23 23:04:46 deathstar kernel: sd 8:0:0:0: [sdf] 60088320 512-byte logical blocks: (30.8 GB/28.7 GiB) +Aug 23 23:04:46 deathstar kernel: sd 8:0:0:0: [sdf] Write Protect is off +Aug 23 23:04:46 deathstar kernel: sd 8:0:0:0: [sdf] Mode Sense: 43 00 00 00 +Aug 23 23:04:46 deathstar kernel: sd 8:0:0:0: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA +Aug 23 23:04:46 deathstar kernel: sdf: sdf1 +Aug 23 23:04:46 deathstar kernel: sd 8:0:0:0: [sdf] Attached SCSI removable disk +Aug 23 23:04:46 deathstar systemd[1]: Started Our own backup script. +Aug 23 23:04:46 deathstar bash[969865]: running as root will backup to sdf1 +Aug 23 23:04:46 deathstar bash[969865]: created /media/backup_usb_stick +Aug 23 23:04:46 deathstar bash[969865]: mounted sdf1 +Aug 23 23:04:46 deathstar kernel: EXT4-fs (sdf1): mounted filesystem with ordered data mode. Opts: (null) +Aug 23 23:04:46 deathstar bash[969876]: sending incremental file list +Aug 23 23:04:46 deathstar bash[969876]: created directory /media/backup_usb_stick/backups +Aug 23 23:04:46 deathstar bash[969876]: ./ +Aug 23 23:04:46 deathstar bash[969876]: arm (Meshed).stl +Aug 23 23:04:46 deathstar bash[969876]: base_pivot (Meshed).stl +Aug 23 23:04:46 deathstar bash[969876]: base_pivot_alt (Meshed).stl +Aug 23 23:04:46 deathstar bash[969876]: base_plate (Meshed).stl +Aug 23 23:04:46 deathstar bash[969876]: base_plate_alt (Meshed).stl +Aug 23 23:04:46 deathstar bash[969876]: filament_guide.FCStd +Aug 23 23:04:46 deathstar bash[969876]: filament_guide.FCStd1 +Aug 23 23:04:46 deathstar bash[969876]: filament_guide.stl +Aug 23 23:04:46 deathstar bash[969876]: ivar_riser.FCStd +Aug 23 23:04:46 deathstar bash[969876]: ivar_riser.FCStd1 +Aug 23 23:04:46 deathstar bash[969876]: ivar_riser.stl +Aug 23 23:04:46 deathstar bash[969876]: light_pot_office (Meshed).stl +Aug 23 23:04:46 deathstar bash[969876]: light_pot_office.FCStd +Aug 23 23:04:46 deathstar bash[969876]: light_pot_office.FCStd1 +Aug 23 23:04:46 deathstar bash[969876]: monitor_arm.FCStd +Aug 23 23:04:46 deathstar bash[969876]: monitor_arm.FCStd1 +Aug 23 23:04:46 deathstar bash[969876]: monitor_arm_120mm.stl +Aug 23 23:04:46 deathstar bash[969876]: picam.FCStd +Aug 23 23:04:46 deathstar bash[969876]: picam.FCStd1 +Aug 23 23:04:46 deathstar bash[969876]: vat_filter.FCStd +Aug 23 23:04:46 deathstar bash[969876]: vat_filter.FCStd1 +Aug 23 23:04:46 deathstar bash[969876]: vat_filter.stl +Aug 23 23:04:46 deathstar bash[969876]: vesa_plate (Meshed).stl +Aug 23 23:04:46 deathstar bash[969876]: sent 1,849,171 bytes received 510 bytes 3,699,362.00 bytes/sec +Aug 23 23:04:46 deathstar bash[969876]: total size is 1,847,019 speedup is 1.00 +Aug 23 23:04:46 deathstar bash[969865]: all done +Aug 23 23:04:46 deathstar bash[969865]: unmounted... +Aug 23 23:04:46 deathstar systemd[1]: backup@sdf1.service: Succeeded. +Aug 23 23:05:01 deathstar kernel: usb 3-1.5: USB disconnect, device number 67 +Aug 23 23:05:04 deathstar kernel: usb 3-1.5: new high-speed USB device number 68 using ehci-pci +Aug 23 23:05:04 deathstar kernel: usb 3-1.5: New USB device found, idVendor=0781, idProduct=5591, bcdDevice= 1.00 +Aug 23 23:05:04 deathstar kernel: usb 3-1.5: New USB device strings: Mfr=1, Product=2, SerialNumber=3 +Aug 23 23:05:04 deathstar kernel: usb 3-1.5: Product: SanDisk 3.2Gen1 +Aug 23 23:05:04 deathstar kernel: usb 3-1.5: Manufacturer: USB +Aug 23 23:05:04 deathstar kernel: usb 3-1.5: SerialNumber: 04018a57f24e4dc0135c1f4666b5f2ca4e8e15e65eedfd157b0146cf1427157522b000000000000000000000a03e3c7e008cb3189155810739aa4777 +Aug 23 23:05:04 deathstar kernel: usb-storage 3-1.5:1.0: USB Mass Storage device detected +Aug 23 23:05:04 deathstar kernel: scsi host8: usb-storage 3-1.5:1.0 +Aug 23 23:05:05 deathstar kernel: scsi 8:0:0:0: Direct-Access USB SanDisk 3.2Gen1 1.00 PQ: 0 ANSI: 6 +Aug 23 23:05:05 deathstar kernel: sd 8:0:0:0: Attached scsi generic sg3 type 0 +Aug 23 23:05:05 deathstar kernel: sd 8:0:0:0: [sdf] 60088320 512-byte logical blocks: (30.8 GB/28.7 GiB) +Aug 23 23:05:05 deathstar kernel: sd 8:0:0:0: [sdf] Write Protect is off +Aug 23 23:05:05 deathstar kernel: sd 8:0:0:0: [sdf] Mode Sense: 43 00 00 00 +Aug 23 23:05:05 deathstar kernel: sd 8:0:0:0: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA +Aug 23 23:05:05 deathstar kernel: sdf: sdf1 +Aug 23 23:05:05 deathstar kernel: sd 8:0:0:0: [sdf] Attached SCSI removable disk +Aug 23 23:05:05 deathstar systemd[1]: Started Our own backup script. +Aug 23 23:05:05 deathstar bash[969947]: running as root will backup to sdf1 +Aug 23 23:05:05 deathstar bash[969947]: created /media/backup_usb_stick +Aug 23 23:05:05 deathstar bash[969947]: mounted sdf1 +Aug 23 23:05:05 deathstar kernel: EXT4-fs (sdf1): mounted filesystem with ordered data mode. Opts: (null) +Aug 23 23:05:05 deathstar bash[969958]: sending incremental file list +Aug 23 23:05:05 deathstar bash[969958]: sent 796 bytes received 12 bytes 1,616.00 bytes/sec +Aug 23 23:05:05 deathstar bash[969958]: total size is 1,847,019 speedup is 2,285.91 +Aug 23 23:05:05 deathstar bash[969947]: all done +Aug 23 23:05:05 deathstar bash[969947]: unmounted... +Aug 23 23:05:05 deathstar systemd[1]: backup@sdf1.service: Succeeded. +```