adds munin doc

modules/qualifying/learning_centralized_system_monitoring.md

@@ -0,0 +1,196 @@
+# Centralized System Monitoring
+
+When you have multiple servers or client computers under your control it's nice to have a centralized way of monitoring them for health or unexpected errors.
+There are a few different solutions out there but we'll have a go at installing and configuring [munin](https://munin-monitoring.org/).
+
+## Munin
+
+I set one up on [this](https://munin.86thumbs.net/) domain for demonstration purposes.
+You can log in with the account details I'll hand out in class.
+
+![munin overview](./assets/munin_01.png)
+
+![munin zoom](./assets/munin_02.png)
+
+![munin details](./assets/munin_03.png)
+
+
+### Architecture
+
+Munin has a server client architecture where they refer to the server as **master** and the clients as **nodes**.
+When monitoring for example ten workstations or servers you will have **one** master and on each workstation or server you'll install **one** node but in total you'll have **ten** nodes connected to your **single** master.
+
+![munin architecture](./assets/munin_architecture.png)
+
+### First installation
+
+#### The master
+
+First we'll install one munin **master** for everybody in the classroom.
+The installation documentation on the official [website](http://guide.munin-monitoring.org/en/latest/installation/install.html) is very good so we'll follow that one.
+The overall processes has three components.
+
+1. install munin
+2. configure the master to query some future nodes
+3. install a web server to serve the monitor pages
+
+##### Installing
+
+Installing can be done through the Debian package manager.
+Do a `tab-complete` to see all available packages and note that on the **master** you need the `munin` package.
+
+`sudo apt install munin`
+
+It installs and configures itself as a systemd service we can start, stop or inspect with the following commands.
+
+```
+sudo systemctl start munin.service
+sudo systemctl stop munin.service
+sudo systemctl status munin.service
+```
+
+If you used `tab-complete`, as you should, you probably noticed that there are *two* munin services running.
+`munin.service` is the **master** and `munin-node.service` is the **node**.
+Why does the master also install the node?
+Well, just so it can monitor itself and display it in the web gui.
+This gives you already a good idea of what will need to be installed and configured on the nodes.
+
+##### Configuring
+
+The main configuration file for munin is located at `/etc/munin/munin.conf`.
+Open up your favorite text editor and browse through it.
+There is should be an entry to just one **node** that is your localhost.
+This is the server monitoring itself.
+On my demonstration server the configuration is as follows.
+
+```
+# a simple host tree
+[muninserver.86thumbs.net]
+    address 127.0.0.1
+    use_node_name yes
+
+[vps.86thumbs.net]
+    address 10.8.0.1
+    use_node_name yes
+
+[bigone.86thumbs.net]
+    address 10.8.0.2
+    use_node_name yes
+
+[hugeone.86thumbs.net]
+    address 192.168.0.236
+    use_node_name yes
+
+[yacyone.86thumbs.net]
+    address 10.8.0.4
+    use_node_name yes
+```
+
+For now we'll just leave it as is and move on to installing the web server.
+
+##### The web server
+
+Here you have a choice to make but any way you go, you can't really go wrong.
+My web server of choice is often `nginx` and we'll install that one together because I'm most comfortable with it.
+When you install your own **master** maybe try out a different one.
+The documentation has templates for the following we servers.
+
+* [apache](https://guide.munin-monitoring.org/en/latest/example/webserver/apache-cgi.html#example-webserver-apache-cgi)
+* [nginx](https://guide.munin-monitoring.org/en/latest/example/webserver/nginx-cron.html#example-webserver-nginx-cron)
+* [lighttpd](https://guide.munin-monitoring.org/en/latest/example/webserver/lighttpd-proxy.html#example-webserver-lighttpd-proxy)
+
+Nginx is really easy to install on Debian. Just type `sudo apt install nginx` and when you get your prompt back, you'll have a web server running.
+To verify it's working we can use any web browser we want.
+
+![it's alive!](./assets/nginx.png)
+
+The main configuration directory for nginx can be found at `/etc/nginx`.
+Nginx is a bit different compared to most configuration we've done up until now.
+The server itself is configured via the `/etc/nginx/nginx.conf` file  and the `/etc/nginx/conf.d/` folder but the **websites** your server serves to the outside world are stored in `/etc/nginx/sites-available` and `/etc/nginx/sites-enabled`.
+Each site should have an actual configuration file in the `/etc/nginx/sites-available` folder and if you want the lite to be *live* you should create a [symbolic link](https://stackoverflow.com/questions/39080375/nginx-symbolic-link) to this site in the `/etc/nginx/sites-enabled` folder.
+
+First have a look at the `/etc/nginx/sitex-available/default` file to understand how the configuration works.
+
+**Can you change something on the default webpage please?**
+
+We can add a new file in the `/etc/nginx/sites-available` directory called `munin`.
+You can name this any which way you want.
+For example, on my vps that runs our domain I have the following files which offer websites:
+
+```
+➜  sites-enabled ls -l
+total 36
+-rw-r--r-- 1 root root 2049 Mar 25 20:56 86thumbs.net
+-rw-r--r-- 1 root root 1813 Mar 25 12:42 gitea.86thumbs.net
+-rw-r--r-- 1 root root 1646 May 24 12:00 ipcdb.86thumbs.net
+lrwxrwxrwx 1 root root   50 Mar  4 22:20 jitsi.86thumbs.net.conf -> /etc/nginx/sites-available/jitsi.86thumbs.net.conf
+-rw-r--r-- 1 root root 1907 Mar 25 20:55 matrix.86thumbs.net
+-rw-r--r-- 1 root root 1720 Jul  8 16:49 munin.86thumbs.net
+-rw-r--r-- 1 root root 1758 Mar 11 16:41 radio.86thumbs.net
+-rw-r--r-- 1 root root 1881 Mar 11 16:41 riot.86thumbs.net
+-rw-r--r-- 1 root root 1930 Mar 11 19:07 taskjuggler.86thumbs.net
+-rw-r--r-- 1 root root 2200 Jul  7 18:50 yacy.86thumbs.net
+➜  sites-enabled 
+```
+
+As you can see, I did not follow the proper *rules* and only one file is a symbolic link towards `sites-available`.
+Shame on me!
+The documentation gives us a good idea of how to configure the website but it's not fully complete.
+The following configuration should give you a fully functional monitoring page.
+
+```
+server {
+	listen 80;
+	listen [::]:80;
+	
+	location /munin/static/ {
+    		alias /etc/munin/static/;
+    		expires modified +1w;
+	}
+
+	location /munin {
+    		alias /var/cache/munin/www/;
+    		expires modified +310s;
+	}
+}
+```
+
+It's even more basic than the one supplied by munin itself just to illustrate a bare bones website.
+If you now go to the server you should see the munin monitoring page with one **node** available, itself.
+
+![basic munin](./assets/munin_04.png)
+
+I urge you to read up a bit on nginx configuration and to look at the folders on your system that are used by nginx to serve websites.
+Web servers are an essential part of any Linux system administrator's life.
+
+**Can you figure out how to password protect this folder so it behaves like my munin server?**
+
+#### The first node
+
+Let's add some nodes!
+The nodes have to be in the same network in order to be contactable by the master.
+There are nifty ways to overcome this *problem* but that is for a later date.
+If you do feel like diving deeper into it, have a look [here](https://guide.munin-monitoring.org/en/latest/tutorial/monitoring-the-unreachable.html#unreachable-index).
+
+On any virtual machine you have around you can install the node by simply doing `sudo apt install munin-node`.
+The node is now up, running and listening for incoming connections on port 4949 (by default).
+It is however quite picky in accepting incoming connections!
+The configuration file for the node is located at `/etc/munin/munin-node.conf`.
+Open it up and have a read.
+You should see two lines very similar to the one below.
+
+```
+allow ^127\.0\.0\.1$
+allow ^::1$
+```
+
+Can you tell me what those are?
+Yes, that's a **regex**!
+By default the node only accepts incoming connections from the localhost.
+We can add as many **masters** as we like to this list but we must respect the regex syntax.
+I'll leave that up to you to figure out.
+
+
+## Hints
+
+* if your dynamic zoom is not working have a look [here](https://serverfault.com/questions/615048/munin-dynamic-graph-zoom-dynazoom-not-working-nginx-php-fpm)