diff --git a/modules/qualifying/assets/cacti_01.jpg b/modules/qualifying/assets/cacti_01.jpg new file mode 100644 index 0000000..38b0855 Binary files /dev/null and b/modules/qualifying/assets/cacti_01.jpg differ diff --git a/modules/qualifying/assets/cacti_02.png b/modules/qualifying/assets/cacti_02.png new file mode 100644 index 0000000..40c9543 Binary files /dev/null and b/modules/qualifying/assets/cacti_02.png differ diff --git a/modules/qualifying/assets/cacti_03.png b/modules/qualifying/assets/cacti_03.png new file mode 100644 index 0000000..17eb144 Binary files /dev/null and b/modules/qualifying/assets/cacti_03.png differ diff --git a/modules/qualifying/assets/cacti_04.png b/modules/qualifying/assets/cacti_04.png new file mode 100644 index 0000000..f551150 Binary files /dev/null and b/modules/qualifying/assets/cacti_04.png differ diff --git a/modules/qualifying/assets/cacti_05.png b/modules/qualifying/assets/cacti_05.png new file mode 100644 index 0000000..a63f1a0 Binary files /dev/null and b/modules/qualifying/assets/cacti_05.png differ diff --git a/modules/qualifying/assets/cacti_06.png b/modules/qualifying/assets/cacti_06.png new file mode 100644 index 0000000..d71a495 Binary files /dev/null and b/modules/qualifying/assets/cacti_06.png differ diff --git a/modules/qualifying/assets/cacti_07.png b/modules/qualifying/assets/cacti_07.png new file mode 100644 index 0000000..e400b63 Binary files /dev/null and b/modules/qualifying/assets/cacti_07.png differ diff --git a/modules/qualifying/assets/cacti_08.png b/modules/qualifying/assets/cacti_08.png new file mode 100644 index 0000000..ea920fd Binary files /dev/null and b/modules/qualifying/assets/cacti_08.png differ diff --git a/modules/qualifying/assets/cacti_09.png b/modules/qualifying/assets/cacti_09.png new file mode 100644 index 0000000..2094ba7 Binary files /dev/null and b/modules/qualifying/assets/cacti_09.png differ diff --git a/modules/qualifying/assets/cacti_10.png b/modules/qualifying/assets/cacti_10.png new file mode 100644 index 0000000..c192936 Binary files /dev/null and b/modules/qualifying/assets/cacti_10.png differ diff --git a/modules/qualifying/assets/cacti_11.png b/modules/qualifying/assets/cacti_11.png new file mode 100644 index 0000000..b46c0dd Binary files /dev/null and b/modules/qualifying/assets/cacti_11.png differ diff --git a/modules/qualifying/learning_centralized_system_monitoring.md b/modules/qualifying/learning_centralized_system_monitoring.md index dec79d8..1d55817 100644 --- a/modules/qualifying/learning_centralized_system_monitoring.md +++ b/modules/qualifying/learning_centralized_system_monitoring.md @@ -198,7 +198,7 @@ To have your node auto detect which plugins should be used you can execute the f munin-node-configure --shell --families=contrib,auto | sh -x ``` -## Other operating systems +### Other operating systems Because the munin server client model is so basic you can find node software for most operating systems. I've run it successfully on [freeNAS](https://www.truenas.com/community/threads/install-munin-node.79224/#post-549934) but you can try a [windows](https://github.com/munin-monitoring/munin-node-win32) node as well. @@ -233,13 +233,7 @@ Connection closed by foreign host. waldek@muninserver:~$ ``` -## Alternatives - -There are quite a few alternatives out there to choose from. -One of the better known is [nagios](https://www.nagios.org/) which can be installed on Debian as well. -There are ways to have munin and nagios work together that are described [here](https://guide.munin-monitoring.org/en/latest/tutorial/nagios.html?highlight=nagios). - -## Notes and hints +### Notes and hints You might feel munin is too simple or basic because it does not offer any tweaking via it's web GUI but that's the main reason why I *like* and use it so much. Because it's so simple, it's not a real security risk. @@ -248,3 +242,139 @@ There are some known exploits in the CGI part of munin. * the Debian system administrator handbook has a good [section](https://debian-handbook.info/browse/stable/sect.monitoring.html#sect.munin) on munin * if your dynamic zoom is not working have a look [here](https://serverfault.com/questions/615048/munin-dynamic-graph-zoom-dynazoom-not-working-nginx-php-fpm) + +## Alternatives + +There are quite a few alternatives out there to choose from. +One of the better known is [nagios](https://www.nagios.org/) which can be installed on Debian as well. +There are ways to have munin and nagios work together that are described [here](https://guide.munin-monitoring.org/en/latest/tutorial/nagios.html?highlight=nagios). + +## Cacti + +Cacti is a very similar monitoring system compared to munin. +Where munin shines in it's simplicity, cacti can be fine-tuned host by host, monitor even more diverse clients such as Cisco routers and switches. +I personally *prefer* munin mostly because I know it better but let's have a go at installing cacti to see if we like it better or not! + +### Architecture + +The architecture for cacti is very similar to munin and can be seen below. +They serve the same **purpose** but do two main things **differently**. + +1. how cacti **fetches** the data +2. how cacti **stores** the data + +![cacti architecture](./assets/cacti_01.jpg) + +#### Fetching data + +Munin fetches it's data via a simple telnet connection. +To restrict access it defaults to blacklisting all IP's except the ones we allow in the configuration file located at `/etc/munin/munin-node.conf`. +Cacti relies on the more elaborated [SNMP](https://en.wikipedia.org/wiki/Simple_Network_Management_Protocol) protocol. +As it's a full blown protocol it can do a lot more than just fetch data and hence is more prone to [exploits](https://www.exploit-db.com/search?q=snmp). + +#### Storing data + +Munin stores it's data as simple binary files located at `/var/lib/munin/$DOMAIN_NAME`. +It's not the quickest nor the most compact way of storing data but as it's not a high demand service it can get away with it. +Cacti on the other hand uses an [SQL](https://en.wikipedia.org/wiki/SQL) database to store it's information which is a more *mature* way of dealing with data availability but comes at the cost of an other server, a mysql in this case, and more configuration and potential security threats. + +### Installing the master + +Debian has a good all-in-one package to install [cacti](https://packages.debian.org/stable/cacti). +It takes care of all the heavy lifting for you and will prompt you with some questions. +You install it with the good old package manager `sudo apt install cacti`. + +First you'll be prompted with a choice of web server to install. +You can choose any which one you like, or choose *none* if you want to use `nginx` for example. +Depending on your choice, apt will fetch the necessary packages. + +![web server](./assets/cacti_02.png) + +Once that's all done you'll be prompted to configure the database. +Here I highly advise you to choose **yes** as sql databases can be a bit of a pain to set up. + +![db choice](./assets/cacti_03.png) + +If you chose yes you'll immediately be prompted to set a password for the cacti user in the database. +This will also be the admin password you'll use to log into the web GUI. +For testing purposes you can choose a simple one but if you think you'll use the server for anything serious, you need a serious password! + +![setting the password](./assets/cacti_04.png) + +That's it, your cacti server is now up and running! +You can visit it's login page by going to the IP address of your cacti server. +If you see the default apache2 page as seen below, append `/cacti` after your IP. +You need to use the password you set during the installation, but the username for the default user is `admin`. + +![apache2 default page](./assets/cacti_05.png) + +![cacti login page](./assets/cacti_06.png) + +### Configuring the nodes + +Now that we have a server up and running it's time to add some clients to it. +Contrary to munin, cacti does not install it's *node/client* component on the server in order for you to be able to monitor *itself*. +Now, which package do we need to install on a client in order for it to be able to submit data to our server? +The answer lies in the protocol used by cacti. +The following command will install the SNMP **daemon** onto our first client, our cacti server. + +``` +sudo apt install snmpd +``` + +This is a quick install and the server should now be ready to report to itself. +Let's try to add it as a client. +In the web GUI you need to go to the `create->new device` tab. + +![new device](./assets/cacti_07.png) + +In the description you can put anything you want, but the `hostname` field needs to be the IP address of your server, which in our case is the loopback device `127.0.0.1`. +You'll see the following screen if your device was successfully added, which should be the case if you followed the isntructions. + +![successfully added new device](./assets/cacti_08.png) + +You can now inspect the data collected in the `graph` tab of the web GUI. +It should look similar to the screenshot below. +It's very possible that the graphs are *not yet* showing. +This depends a bit on whether the poller has already finished polling the SNMP daemon but this should be a one time problem fixed by waiting a couple of minutes. + +![graph](./assets/cacti_09.png) + +The procedure above is how you add clients to the server. +It's basically a two step process, but we'll see in a minute we need one extra *half step* to make it work with clients that are not the server *itself*. + +1. install `snmpd` on the **client** + * configure which IP the client listens on +2. add the client IP to the **server** + +So, let's add an other client to the master. +Take any Linux machine you have and install the snmpd package. +Once installed try to add it's IP address to the web GUI of cacti. +You'll be confronted with an error screen like the one below. +The client was **added** but there is an error message in red. +What's happening here? + +![error adding client](./assets/cacti_10.png) + +`System SNMP error - SNMP::get(): No response from 192.168.0.222` means that the client is not listening for incoming connections. +We can fix this by configuring the client to listen on **all** interfaces. +By default the SNMP daemon **only** listens on the loopback device but it's an easy fix. +Open the following file `/etc/snmp/snmp.conf` with your favorite text editor and the problem is sitting right in front of your nose. + +``` +# +# AGENT BEHAVIOUR +# + +# Listen for connections from the local system only +agentAddress udp:127.0.0.1:161 +# Listen for connections on all interfaces (both IPv4 *and* IPv6) +#agentAddress udp:161,udp6:[::1]:161 +``` + +You should comment the active `agentAddress` in exchange for the second one. +The second one makes the daemon listen on all interfaces for both ipv4 and ipv6. +If for some reason you configured your Linux kernel to only do ipv4, you'll need to remove the ipv6 entry as follows `agentAddress udp:161` otherwise the daemon will fail to restart. +If you restart the service with `sudo systemctl restart snmpd.service` you're client should be contactable by your server! + +![success](./assets/cacti_11.png)