linux_course_doc/modules/qualifying/learning_centralized_system_monitoring.md

# Centralized System Monitoring

When you have multiple servers or client computers under your control it's nice to have a centralized way of monitoring them for health or unexpected errors.
There are a few different solutions out there but we'll have a go at installing and configuring [munin](https://munin-monitoring.org/).

## Munin

I set one up on [this](https://munin.86thumbs.net/) domain for demonstration purposes.
You can log in with the account details I'll hand out in class.

![munin overview](./assets/munin_01.png)

![munin zoom](./assets/munin_02.png)

![munin details](./assets/munin_03.png)


### Architecture

Munin has a server client architecture where they refer to the server as **master** and the clients as **nodes**.
When monitoring for example ten workstations or servers you will have **one** master and on each workstation or server you'll install **one** node but in total you'll have **ten** nodes connected to your **single** master.

![munin architecture](./assets/munin_architecture.png)

### First installation

#### The master

First we'll install one munin **master** for everybody in the classroom.
The installation documentation on the official [website](http://guide.munin-monitoring.org/en/latest/installation/install.html) is very good so we'll follow that one.
The overall processes has three components.

1. install munin
2. configure the master to query some future nodes
3. install a web server to serve the monitor pages

##### Installing

Installing can be done through the Debian package manager.
Do a `tab-complete` to see all available packages and note that on the **master** you need the `munin` package.

`sudo apt install munin`

It installs and configures itself as a systemd service we can start, stop or inspect with the following commands.

```
sudo systemctl start munin.service
sudo systemctl stop munin.service
sudo systemctl status munin.service
```

If you used `tab-complete`, as you should, you probably noticed that there are *two* munin services running.
`munin.service` is the **master** and `munin-node.service` is the **node**.
Why does the master also install the node?
Well, just so it can monitor itself and display it in the web gui.
This gives you already a good idea of what will need to be installed and configured on the nodes.

##### Configuring

The main configuration file for munin is located at `/etc/munin/munin.conf`.
Open up your favorite text editor and browse through it.
There is should be an entry to just one **node** that is your localhost.
This is the server monitoring itself.
On my demonstration server the configuration is as follows.

```
# a simple host tree
[muninserver.86thumbs.net]
    address 127.0.0.1
    use_node_name yes

[vps.86thumbs.net]
    address 10.8.0.1
    use_node_name yes

[bigone.86thumbs.net]
    address 10.8.0.2
    use_node_name yes

[hugeone.86thumbs.net]
    address 192.168.0.236
    use_node_name yes

[yacyone.86thumbs.net]
    address 10.8.0.4
    use_node_name yes
```

For now we'll just leave it as is and move on to installing the web server.

##### The web server

Here you have a choice to make but any way you go, you can't really go wrong.
My web server of choice is often `nginx` and we'll install that one together because I'm most comfortable with it.
When you install your own **master** maybe try out a different one.
The documentation has templates for the following we servers.

* [apache](https://guide.munin-monitoring.org/en/latest/example/webserver/apache-cgi.html#example-webserver-apache-cgi)
* [nginx](https://guide.munin-monitoring.org/en/latest/example/webserver/nginx-cron.html#example-webserver-nginx-cron)
* [lighttpd](https://guide.munin-monitoring.org/en/latest/example/webserver/lighttpd-proxy.html#example-webserver-lighttpd-proxy)

Nginx is really easy to install on Debian. Just type `sudo apt install nginx` and when you get your prompt back, you'll have a web server running.
To verify it's working we can use any web browser we want.

![it's alive!](./assets/nginx.png)

The main configuration directory for nginx can be found at `/etc/nginx`.
Nginx is a bit different compared to most configuration we've done up until now.
The server itself is configured via the `/etc/nginx/nginx.conf` file  and the `/etc/nginx/conf.d/` folder but the **websites** your server serves to the outside world are stored in `/etc/nginx/sites-available` and `/etc/nginx/sites-enabled`.
Each site should have an actual configuration file in the `/etc/nginx/sites-available` folder and if you want the lite to be *live* you should create a [symbolic link](https://stackoverflow.com/questions/39080375/nginx-symbolic-link) to this site in the `/etc/nginx/sites-enabled` folder.

First have a look at the `/etc/nginx/sitex-available/default` file to understand how the configuration works.

**Can you change something on the default webpage please?**

We can add a new file in the `/etc/nginx/sites-available` directory called `munin`.
You can name this any which way you want.
For example, on my vps that runs our domain I have the following files which offer websites:

```
➜  sites-enabled ls -l
total 36
-rw-r--r-- 1 root root 2049 Mar 25 20:56 86thumbs.net
-rw-r--r-- 1 root root 1813 Mar 25 12:42 gitea.86thumbs.net
-rw-r--r-- 1 root root 1646 May 24 12:00 ipcdb.86thumbs.net
lrwxrwxrwx 1 root root   50 Mar  4 22:20 jitsi.86thumbs.net.conf -> /etc/nginx/sites-available/jitsi.86thumbs.net.conf
-rw-r--r-- 1 root root 1907 Mar 25 20:55 matrix.86thumbs.net
-rw-r--r-- 1 root root 1720 Jul  8 16:49 munin.86thumbs.net
-rw-r--r-- 1 root root 1758 Mar 11 16:41 radio.86thumbs.net
-rw-r--r-- 1 root root 1881 Mar 11 16:41 riot.86thumbs.net
-rw-r--r-- 1 root root 1930 Mar 11 19:07 taskjuggler.86thumbs.net
-rw-r--r-- 1 root root 2200 Jul  7 18:50 yacy.86thumbs.net
➜  sites-enabled 
```

As you can see, I did not follow the proper *rules* and only one file is a symbolic link towards `sites-available`.
Shame on me!
The documentation gives us a good idea of how to configure the website but it's not fully complete.
The following configuration should give you a fully functional monitoring page.

```
server {
	listen 80;
	listen [::]:80;
	
	location /munin/static/ {
    		alias /etc/munin/static/;
    		expires modified +1w;
	}

	location /munin {
    		alias /var/cache/munin/www/;
    		expires modified +310s;
	}
}
```

It's even more basic than the one supplied by munin itself just to illustrate a bare bones website.
If you now go to the server you should see the munin monitoring page with one **node** available, itself.

![basic munin](./assets/munin_04.png)

I urge you to read up a bit on nginx configuration and to look at the folders on your system that are used by nginx to serve websites.
Web servers are an essential part of any Linux system administrator's life.

**Can you figure out how to password protect this folder so it behaves like my munin server?**

#### The first node

Let's add some nodes!
The nodes have to be in the same network in order to be contactable by the master.
There are nifty ways to overcome this *problem* but that is for a later date.
If you do feel like diving deeper into it, have a look [here](https://guide.munin-monitoring.org/en/latest/tutorial/monitoring-the-unreachable.html#unreachable-index).

On any virtual machine you have around you can install the node by simply doing `sudo apt install munin-node`.
The node is now up, running and listening for incoming connections on port 4949 (by default).
It is however quite picky in accepting incoming connections!
The configuration file for the node is located at `/etc/munin/munin-node.conf`.
Open it up and have a read.
You should see two lines very similar to the one below.

```
allow ^127\.0\.0\.1$
allow ^::1$
```

Can you tell me what those are?
Yes, that's a **regex**!
By default the node only accepts incoming connections from the localhost.
We can add as many **masters** as we like to this list but we must respect the regex syntax.
I'll leave that up to you to figure out.


## Hints

* if your dynamic zoom is not working have a look [here](https://serverfault.com/questions/615048/munin-dynamic-graph-zoom-dynazoom-not-working-nginx-php-fpm)