adds cacti tutorial

modules/qualifying/learning_centralized_system_monitoring.md

@@ -198,7 +198,7 @@ To have your node auto detect which plugins should be used you can execute the f
 munin-node-configure --shell --families=contrib,auto | sh -x
 ```
 
-## Other operating systems
+### Other operating systems
 
 Because the munin server client model is so basic you can find node software for most operating systems.
 I've run it successfully on [freeNAS](https://www.truenas.com/community/threads/install-munin-node.79224/#post-549934) but you can try a [windows](https://github.com/munin-monitoring/munin-node-win32) node as well.
@@ -233,13 +233,7 @@ Connection closed by foreign host.
 waldek@muninserver:~$ 
 ```
 
-## Alternatives
-
-There are quite a few alternatives out there to choose from.
-One of the better known is [nagios](https://www.nagios.org/) which can be installed on Debian as well.
-There are ways to have munin and nagios work together that are described [here](https://guide.munin-monitoring.org/en/latest/tutorial/nagios.html?highlight=nagios).
-
-## Notes and hints
+### Notes and hints
 
 You might feel munin is too simple or basic because it does not offer any tweaking via it's web GUI but that's the main reason why I *like* and use it so much.
 Because it's so simple, it's not a real security risk.
@@ -248,3 +242,139 @@ There are some known exploits in the CGI part of munin.
 
 * the Debian system administrator handbook has a good [section](https://debian-handbook.info/browse/stable/sect.monitoring.html#sect.munin) on munin
 * if your dynamic zoom is not working have a look [here](https://serverfault.com/questions/615048/munin-dynamic-graph-zoom-dynazoom-not-working-nginx-php-fpm)
+
+## Alternatives
+
+There are quite a few alternatives out there to choose from.
+One of the better known is [nagios](https://www.nagios.org/) which can be installed on Debian as well.
+There are ways to have munin and nagios work together that are described [here](https://guide.munin-monitoring.org/en/latest/tutorial/nagios.html?highlight=nagios).
+
+## Cacti
+
+Cacti is a very similar monitoring system compared to munin.
+Where munin shines in it's simplicity, cacti can be fine-tuned host by host, monitor even more diverse clients such as Cisco routers and switches.
+I personally *prefer* munin mostly because I know it better but let's have a go at installing cacti to see if we like it better or not!
+
+### Architecture
+
+The architecture for cacti is very similar to munin and can be seen below.
+They serve the same **purpose** but do two main things **differently**.
+
+1. how cacti **fetches** the data
+2. how cacti **stores** the data
+
+![cacti architecture](./assets/cacti_01.jpg)
+
+#### Fetching data
+
+Munin fetches it's data via a simple telnet connection. 
+To restrict access it defaults to blacklisting all IP's except the ones we allow in the configuration file located at `/etc/munin/munin-node.conf`.
+Cacti relies on the more elaborated [SNMP](https://en.wikipedia.org/wiki/Simple_Network_Management_Protocol) protocol.
+As it's a full blown protocol it can do a lot more than just fetch data and hence is more prone to [exploits](https://www.exploit-db.com/search?q=snmp).
+
+#### Storing data
+
+Munin stores it's data as simple binary files located at `/var/lib/munin/$DOMAIN_NAME`.
+It's not the quickest nor the most compact way of storing data but as it's not a high demand service it can get away with it.
+Cacti on the other hand uses an [SQL](https://en.wikipedia.org/wiki/SQL) database to store it's information which is a more *mature* way of dealing with data availability but comes at the cost of an other server, a mysql in this case, and more configuration and potential security threats.
+
+### Installing the master
+
+Debian has a good all-in-one package to install [cacti](https://packages.debian.org/stable/cacti).
+It takes care of all the heavy lifting for you and will prompt you with some questions.
+You install it with the good old package manager `sudo apt install cacti`.
+
+First you'll be prompted with a choice of web server to install.
+You can choose any which one you like, or choose *none* if you want to use `nginx` for example.
+Depending on your choice, apt will fetch the necessary packages.
+
+![web server](./assets/cacti_02.png)
+
+Once that's all done you'll be prompted to configure the database.
+Here I highly advise you to choose **yes** as sql databases can be a bit of a pain to set up.
+
+![db choice](./assets/cacti_03.png)
+
+If you chose yes you'll immediately be prompted to set a password for the cacti user in the database.
+This will also be the admin password you'll use to log into the web GUI.
+For testing purposes you can choose a simple one but if you think you'll use the server for anything serious, you need a serious password!
+
+![setting the password](./assets/cacti_04.png)
+
+That's it, your cacti server is now up and running!
+You can visit it's login page by going to the IP address of your cacti server.
+If you see the default apache2 page as seen below, append `/cacti` after your IP.
+You need to use the password you set during the installation, but the username for the default user is `admin`.
+
+![apache2 default page](./assets/cacti_05.png)
+
+![cacti login page](./assets/cacti_06.png)
+
+### Configuring the nodes
+
+Now that we have a server up and running it's time to add some clients to it.
+Contrary to munin, cacti does not install it's *node/client* component on the server in order for you to be able to monitor *itself*.
+Now, which package do we need to install on a client in order for it to be able to submit data to our server?
+The answer lies in the protocol used by cacti.
+The following command will install the SNMP **daemon** onto our first client, our cacti server.
+
+```
+sudo apt install snmpd
+```
+
+This is a quick install and the server should now be ready to report to itself.
+Let's try to add it as a client.
+In the web GUI you need to go to the `create->new device` tab.
+
+![new device](./assets/cacti_07.png)
+
+In the description you can put anything you want, but the `hostname` field needs to be the IP address of your server, which in our case is the loopback device `127.0.0.1`.
+You'll see the following screen if your device was successfully added, which should be the case if you followed the isntructions.
+
+![successfully added new device](./assets/cacti_08.png)
+
+You can now inspect the data collected in the `graph` tab of the web GUI.
+It should look similar to the screenshot below.
+It's very possible that the graphs are *not yet* showing.
+This depends a bit on whether the poller has already finished polling the SNMP daemon but this should be a one time problem fixed by waiting a couple of minutes.
+
+![graph](./assets/cacti_09.png)
+
+The procedure above is how you add clients to the server.
+It's basically a two step process, but we'll see in a minute we need one extra *half step* to make it work with clients that are not the server *itself*.
+
+1. install `snmpd` on the **client**
+	* configure which IP the client listens on
+2. add the client IP to the **server**
+
+So, let's add an other client to the master.
+Take any Linux machine you have and install the snmpd package.
+Once installed try to add it's IP address to the web GUI of cacti.
+You'll be confronted with an error screen like the one below.
+The client was **added** but there is an error message in red.
+What's happening here?
+
+![error adding client](./assets/cacti_10.png)
+
+`System SNMP error - SNMP::get(): No response from 192.168.0.222` means that the client is not listening for incoming connections.
+We can fix this by configuring the client to listen on **all** interfaces.
+By default the SNMP daemon **only** listens on the loopback device but it's an easy fix.
+Open the following file `/etc/snmp/snmp.conf` with your favorite text editor and the problem is sitting right in front of your nose.
+
+```
+#
+#  AGENT BEHAVIOUR
+#
+
+#  Listen for connections from the local system only
+agentAddress  udp:127.0.0.1:161
+#  Listen for connections on all interfaces (both IPv4 *and* IPv6)
+#agentAddress udp:161,udp6:[::1]:161
+```
+
+You should comment the active `agentAddress` in exchange for the second one.
+The second one makes the daemon listen on all interfaces for both ipv4 and ipv6.
+If for some reason you configured your Linux kernel to only do ipv4, you'll need to remove the ipv6 entry as follows `agentAddress udp:161` otherwise the daemon will fail to restart.
+If you restart the service with `sudo systemctl restart snmpd.service` you're client should be contactable by your server!
+
+![success](./assets/cacti_11.png)