diff --git a/modules/qualifying/assets/munin_01.png b/modules/qualifying/assets/munin_01.png new file mode 100644 index 0000000..e414d7a Binary files /dev/null and b/modules/qualifying/assets/munin_01.png differ diff --git a/modules/qualifying/assets/munin_02.png b/modules/qualifying/assets/munin_02.png new file mode 100644 index 0000000..9c5be77 Binary files /dev/null and b/modules/qualifying/assets/munin_02.png differ diff --git a/modules/qualifying/assets/munin_03.png b/modules/qualifying/assets/munin_03.png new file mode 100644 index 0000000..da1c5a9 Binary files /dev/null and b/modules/qualifying/assets/munin_03.png differ diff --git a/modules/qualifying/assets/munin_04.png b/modules/qualifying/assets/munin_04.png new file mode 100644 index 0000000..e44c125 Binary files /dev/null and b/modules/qualifying/assets/munin_04.png differ diff --git a/modules/qualifying/assets/munin_architecture.png b/modules/qualifying/assets/munin_architecture.png new file mode 100644 index 0000000..f7dfaa5 Binary files /dev/null and b/modules/qualifying/assets/munin_architecture.png differ diff --git a/modules/qualifying/assets/nginx.png b/modules/qualifying/assets/nginx.png new file mode 100644 index 0000000..b16b258 Binary files /dev/null and b/modules/qualifying/assets/nginx.png differ diff --git a/modules/qualifying/learning_centralized_system_monitoring.md b/modules/qualifying/learning_centralized_system_monitoring.md new file mode 100644 index 0000000..171d9c0 --- /dev/null +++ b/modules/qualifying/learning_centralized_system_monitoring.md @@ -0,0 +1,196 @@ +# Centralized System Monitoring + +When you have multiple servers or client computers under your control it's nice to have a centralized way of monitoring them for health or unexpected errors. +There are a few different solutions out there but we'll have a go at installing and configuring [munin](https://munin-monitoring.org/). + +## Munin + +I set one up on [this](https://munin.86thumbs.net/) domain for demonstration purposes. +You can log in with the account details I'll hand out in class. + +![munin overview](./assets/munin_01.png) + +![munin zoom](./assets/munin_02.png) + +![munin details](./assets/munin_03.png) + + +### Architecture + +Munin has a server client architecture where they refer to the server as **master** and the clients as **nodes**. +When monitoring for example ten workstations or servers you will have **one** master and on each workstation or server you'll install **one** node but in total you'll have **ten** nodes connected to your **single** master. + +![munin architecture](./assets/munin_architecture.png) + +### First installation + +#### The master + +First we'll install one munin **master** for everybody in the classroom. +The installation documentation on the official [website](http://guide.munin-monitoring.org/en/latest/installation/install.html) is very good so we'll follow that one. +The overall processes has three components. + +1. install munin +2. configure the master to query some future nodes +3. install a web server to serve the monitor pages + +##### Installing + +Installing can be done through the Debian package manager. +Do a `tab-complete` to see all available packages and note that on the **master** you need the `munin` package. + +`sudo apt install munin` + +It installs and configures itself as a systemd service we can start, stop or inspect with the following commands. + +``` +sudo systemctl start munin.service +sudo systemctl stop munin.service +sudo systemctl status munin.service +``` + +If you used `tab-complete`, as you should, you probably noticed that there are *two* munin services running. +`munin.service` is the **master** and `munin-node.service` is the **node**. +Why does the master also install the node? +Well, just so it can monitor itself and display it in the web gui. +This gives you already a good idea of what will need to be installed and configured on the nodes. + +##### Configuring + +The main configuration file for munin is located at `/etc/munin/munin.conf`. +Open up your favorite text editor and browse through it. +There is should be an entry to just one **node** that is your localhost. +This is the server monitoring itself. +On my demonstration server the configuration is as follows. + +``` +# a simple host tree +[muninserver.86thumbs.net] + address 127.0.0.1 + use_node_name yes + +[vps.86thumbs.net] + address 10.8.0.1 + use_node_name yes + +[bigone.86thumbs.net] + address 10.8.0.2 + use_node_name yes + +[hugeone.86thumbs.net] + address 192.168.0.236 + use_node_name yes + +[yacyone.86thumbs.net] + address 10.8.0.4 + use_node_name yes +``` + +For now we'll just leave it as is and move on to installing the web server. + +##### The web server + +Here you have a choice to make but any way you go, you can't really go wrong. +My web server of choice is often `nginx` and we'll install that one together because I'm most comfortable with it. +When you install your own **master** maybe try out a different one. +The documentation has templates for the following we servers. + +* [apache](https://guide.munin-monitoring.org/en/latest/example/webserver/apache-cgi.html#example-webserver-apache-cgi) +* [nginx](https://guide.munin-monitoring.org/en/latest/example/webserver/nginx-cron.html#example-webserver-nginx-cron) +* [lighttpd](https://guide.munin-monitoring.org/en/latest/example/webserver/lighttpd-proxy.html#example-webserver-lighttpd-proxy) + +Nginx is really easy to install on Debian. Just type `sudo apt install nginx` and when you get your prompt back, you'll have a web server running. +To verify it's working we can use any web browser we want. + +![it's alive!](./assets/nginx.png) + +The main configuration directory for nginx can be found at `/etc/nginx`. +Nginx is a bit different compared to most configuration we've done up until now. +The server itself is configured via the `/etc/nginx/nginx.conf` file and the `/etc/nginx/conf.d/` folder but the **websites** your server serves to the outside world are stored in `/etc/nginx/sites-available` and `/etc/nginx/sites-enabled`. +Each site should have an actual configuration file in the `/etc/nginx/sites-available` folder and if you want the lite to be *live* you should create a [symbolic link](https://stackoverflow.com/questions/39080375/nginx-symbolic-link) to this site in the `/etc/nginx/sites-enabled` folder. + +First have a look at the `/etc/nginx/sitex-available/default` file to understand how the configuration works. + +**Can you change something on the default webpage please?** + +We can add a new file in the `/etc/nginx/sites-available` directory called `munin`. +You can name this any which way you want. +For example, on my vps that runs our domain I have the following files which offer websites: + +``` +➜ sites-enabled ls -l +total 36 +-rw-r--r-- 1 root root 2049 Mar 25 20:56 86thumbs.net +-rw-r--r-- 1 root root 1813 Mar 25 12:42 gitea.86thumbs.net +-rw-r--r-- 1 root root 1646 May 24 12:00 ipcdb.86thumbs.net +lrwxrwxrwx 1 root root 50 Mar 4 22:20 jitsi.86thumbs.net.conf -> /etc/nginx/sites-available/jitsi.86thumbs.net.conf +-rw-r--r-- 1 root root 1907 Mar 25 20:55 matrix.86thumbs.net +-rw-r--r-- 1 root root 1720 Jul 8 16:49 munin.86thumbs.net +-rw-r--r-- 1 root root 1758 Mar 11 16:41 radio.86thumbs.net +-rw-r--r-- 1 root root 1881 Mar 11 16:41 riot.86thumbs.net +-rw-r--r-- 1 root root 1930 Mar 11 19:07 taskjuggler.86thumbs.net +-rw-r--r-- 1 root root 2200 Jul 7 18:50 yacy.86thumbs.net +➜ sites-enabled +``` + +As you can see, I did not follow the proper *rules* and only one file is a symbolic link towards `sites-available`. +Shame on me! +The documentation gives us a good idea of how to configure the website but it's not fully complete. +The following configuration should give you a fully functional monitoring page. + +``` +server { + listen 80; + listen [::]:80; + + location /munin/static/ { + alias /etc/munin/static/; + expires modified +1w; + } + + location /munin { + alias /var/cache/munin/www/; + expires modified +310s; + } +} +``` + +It's even more basic than the one supplied by munin itself just to illustrate a bare bones website. +If you now go to the server you should see the munin monitoring page with one **node** available, itself. + +![basic munin](./assets/munin_04.png) + +I urge you to read up a bit on nginx configuration and to look at the folders on your system that are used by nginx to serve websites. +Web servers are an essential part of any Linux system administrator's life. + +**Can you figure out how to password protect this folder so it behaves like my munin server?** + +#### The first node + +Let's add some nodes! +The nodes have to be in the same network in order to be contactable by the master. +There are nifty ways to overcome this *problem* but that is for a later date. +If you do feel like diving deeper into it, have a look [here](https://guide.munin-monitoring.org/en/latest/tutorial/monitoring-the-unreachable.html#unreachable-index). + +On any virtual machine you have around you can install the node by simply doing `sudo apt install munin-node`. +The node is now up, running and listening for incoming connections on port 4949 (by default). +It is however quite picky in accepting incoming connections! +The configuration file for the node is located at `/etc/munin/munin-node.conf`. +Open it up and have a read. +You should see two lines very similar to the one below. + +``` +allow ^127\.0\.0\.1$ +allow ^::1$ +``` + +Can you tell me what those are? +Yes, that's a **regex**! +By default the node only accepts incoming connections from the localhost. +We can add as many **masters** as we like to this list but we must respect the regex syntax. +I'll leave that up to you to figure out. + + +## Hints + +* if your dynamic zoom is not working have a look [here](https://serverfault.com/questions/615048/munin-dynamic-graph-zoom-dynazoom-not-working-nginx-php-fpm)