linux_course_doc/modules/resources/solution_raspberrypi.md

# The FTP challenge

## Getting the file

The first step to solving this exersise is to download the CSV file to your Raspberry.
For those wondering what on earth a CSV file is I invite you to a detailed [read](https://en.wikipedia.org/wiki/Comma-separated_values) but to make a long story short it stands for *comma separated values* and is one of the most basic ways to structure data.
You can use Libreoffice calc to open it and you'll quickly understand how it works.

Now, to get the file onto our Raspberry PI we need to download it from the webserver.
One way would be to use the `wget` program to do so.
We know *where* the server is by it's IP address, plus we also know the *filename*.
Putting these two together we can construct the following line.

```bash
wget 172.30.6.96/accounts.csv
```

This will download the file to the directory we're in and will save it as `accounts.csv`.
You can change the output filename it you want, just have a look at the `wget` options via `wget --help` or our trusty `man wget`.

## Extracting the data we need

A quick `cat accounts.csv` gives us the following output:

```bash
EMAIL,LASTNAME,FIRSTNAME,MATRIX,GITEA,TEAM
1h.lust.hugo@gmail.com,Lust,Hugo,@hugo_lust:86thumbs.net,https://gitea.86thumbs.net/Hugo,red
ticus@kraland.net,krstev,vladimir,@vl4dd:86thumbs.net,https://gitea.86thumbs.net/vl4dd,blue
adamd@outlook.be,Adam,David,@adamd73:matrix.org,https://gitea.86thumbs.net/adamd,red
nicohawai@gmail.com,Perez,Nicolas,@hawai:86thumbs.net,https://gitea.86thumbs.net/Hawai,blue
nicolas.wattripont@gmail.com,Wattripont,Nicolas,@wawa142:86thumbs.net,https://gitea.86thumbs.net/wawa142,red
laurentdelvigne@hotmail.com,Delvigne,Laurent,@ldelvigne:86thumbs.net,https://gitea.86thumbs.net/ldelvigne,blue
sselcukaslan@gmail.com,Aslan,Selçuk,@slck:86thumbs.net,https://gitea.86thumbs.net/selcuk,blue
Sarah24886@hotmail.com,Rmiki,Sarah,@sarahrm95:matrix.org,https://gitea.86thumbs.net/sarahrm95,blue
knoppixs@hotmail.com,Abbamoulay,Abdellah,@knoppixs:86thumbs.net,https://gitea.86thumbs.net/Abdellah,red
JonathanDechief@hotmail.com,Dechief,Jonathan,@elewene:matrix.org,https://gitea.86thumbs.net/Elewene,red
51207@etu.he2b.be,,Aliou,@aliou:86thumbs.net,https://gitea.86thumbs.net/aliou,blue
```

### The $USERNAME

To extract the `$USERNAME` we're interested in the fourth collumn which has the MATRIX login handle.
We can extract only this collumn by using `cut` with `,` as a delimiter.

```bash
cat accounts.csv | cut -d "," -f 4
```

This leaves us with only the MATRIX login handles which is a good start but there is still a bit too much information .
We need to drop the first line, which is the *header* of the CSV file, plus crop between the `@` and the `:`.
These two operations can be done is multiple ways but I suggest this modification.
It is not the most elegant solution but it uses only tools you have used this far.
The `tail` command drops the *header* line, and the two `cut` commands trim the username to just what we need.

```bash
cat accounts.csv | cut -d "," -f 4 | tail -n +2 | cut -d ":" -f 1 | cut -d "@" -f 2
```

Done!
We have all the usernames we need.
We can save this to a file by redirecting the STDOUT to a file with the following command.

```bash
cat accounts.csv | cut -d "," -f 4 | tail -n +2 | cut -d ":" -f 1 | cut -d "@" -f 2 > usernames.list
```

### The $PASSWORD

To extract the password we need to combine two field from the CSV file.
A *really* good command line program to accieve this is `awk`.
Don't forget the man pages!

```bash
cat accounts.csv | awk -F "," '{print $2 $3}' | tail -n +2
```

If you feel like making the password complexer, you can try to add in extra data into the `awk` command, or even append random numbers to the end.

```bash
cat accounts.csv | awk -F "," '{print $2 "_helloworld_" $3}' | tail -n +2
```

We can now save these passwords to a file in the same way as before.

### The $GROUP

This is an *easy* one because it requires no real modification of the field.

```bash
cat accounts.csv | tail -n +2 | cut -d "," -f 6 > groups.list
```

## Using this information to create accounts

We now have command that extracts the information we need, plus three separate files that contain all the information.
This is a good moment to try and write a very simple script.
I'll first try to do it without a loop but you'll quickly see it's a *lot* easier with a loop in there.
Remember that `$1` represents the first argument on the command line so that when calling our script with `./script.sh 3` we'll get the username, password and group for the third user.
A combination of `head | tail` is a [classic](https://stackoverflow.com/questions/6022384/bash-tool-to-get-nth-line-from-a-file) way of selection only one file from a file.
Last but not least, don't forget to add execution permissions to this script with `chmod`.

```bash
#!/bin/bash

LINE=$1

head -$1 usernames.list | tail -1
head -$1 passwords.list | tail -1
head -$1 groups.list | tail -1
```

We can then *copy/paste* this information to create the following line.
We'll be prompted to paste in the proper information.

```bash
sudo adduser $USERNAME
```

Needless to say this is a labour intensive operation that we can automate with a mini script.

### Putting it together as a script

Brace yourselves a bit but I promise it's worth it!
The only thing we have not seen is how to save the output of a command into a variable.
This can be done with the `$(...)` syntax.
I know it looks a bit cryptic but an example speaks more than words.

```bash
NOW=$(date)
echo $NOW
```

With this in mind, the following code should make sense.
We're doing the exact same thing but saving the output of each command into a variable.
At the last line we *use* the variables to create a message.

```bash
#!/bin/bash

LINE=$1

USERNAME=$(head -$1 usernames.list | tail -1)
PASSWORD=$(head -$1 passwords.list | tail -1)
GROUP=$(head -$1 groups.list | tail -1)

echo "user: $USERNAME password: $PASSWORD group: $GROUP"
```

This just output's all information onto one line, but why not *use* this information to actually create the accounts?
A counterpart to the `adduser` program you're used to using, there is also `useradd` which is better suited for scripting purposes.
By default `useradd` is very *barebones* and does not create a home directory for the user but a quick look at the `man useradd` pages tells us we can use the `-m` flag to do so. 
This tells us the the command `useradd $USERNAME -m` will create a user for us with his/her own home directory.
A [google search](https://linux.die.net/man/8/chpasswd) pointed me to `chpasswd` to set passwords from within a script.
The line to set the password will be `echo $USERNAME:$PASSWORD | chpasswd`.
This gives us the following script.

```bash
#!/bin/bash

LINE=$1

USERNAME=$(head -$1 usernames.list | tail -1)
PASSWORD=$(head -$1 passwords.list | tail -1)
GROUP=$(head -$1 groups.list | tail -1)

echo "adding user: $USERNAME"
useradd $USERNAME -m

echo "setting password: $PASSWORD for $USERNAME"
echo $USERNAME:$PASSWORD | chpasswd
```

You probably noticed I did not add the users to their *red/blue* groups.
We can add them on the `useradd` line with the `-G` flag but it would fail if the group does not exist yet.
The `groupadd` command will add a group to the system and if we add the `-f` flag to it will do so without complaining if the group already exists.
This way we can just execute that line each time without worrying wether the group exists or not.

```bash 
#!/bin/bash

LINE=$1

USERNAME=$(head -$1 usernames.list | tail -1)
PASSWORD=$(head -$1 passwords.list | tail -1)
GROUP=$(head -$1 groups.list | tail -1)

echo "making sure $GROUP exists..."
groupadd -f $GROUP

echo "adding user: $USERNAME"
useradd $USERNAME -m -G $GROUP

echo "setting password: $PASSWORD for $USERNAME"
echo $USERNAME:$PASSWORD | chpasswd
```

Those who switched to the newly created user noticed that the shell is *very* basic one.
You can find out which shell these new accounts use by looking at the `/etc/passwd` file.
There are multiple ways to sort this problem but a look at the `man useradd` pages tells us we can use the `-s` flag to set the shell we want for the user.
We probably want to use `/bin/bash` for this option!

```bash 
#!/bin/bash

LINE=$1

USERNAME=$(head -$1 usernames.list | tail -1)
PASSWORD=$(head -$1 passwords.list | tail -1)
GROUP=$(head -$1 groups.list | tail -1)

echo "making sure $GROUP exists..."
groupadd -f $GROUP

echo "adding user: $USERNAME"
useradd $USERNAME -m -G $GROUP -s "/bin/bash"

echo "setting password: $PASSWORD for $USERNAME"
echo $USERNAME:$PASSWORD | chpasswd

echo "adding $USERNAME to $GROUP"
```

This is getting pretty close to perfect!
We can now run through all of the lines of the file one by one and automatically create the proper user, password and group combinations.
To know how many accounts we have to create we can use `wc -l accounts.csv` and then just run the script, incrementing the number each time.

```bash
sudo ./script.sh 1
sudo ./script.sh 2
sudo ./script.sh 3
echo "etc..."
```

### Taking it further as an extra challenge

If we want to automate the entire thing we'll need a loop to *loop through* every line in the `accounts.csv` file.
Bash loops are for a further class but I'll leave you with this quick example for those who feel like messing around.
Remember the *oneliners* we constructed at the beginning to extract the relevant information from the line!
Don't worry if this looks to complicated at the moment, we'll do this exersise again when we're looking into [bash scripting](https://ryanstutorials.net/bash-scripting-tutorial/).

```bash
#!/bin/bash

FILE=$1
LINES=$(cat $1)

for LINE in $LINES;
do
	echo $LINE
done
```

## Setting up the fileserver

### FTP

A quick [google](https://likegeeks.com/ftp-server-linux/) tells us `vsftpd` is a popular *FileTransferProtocol* server.
As expected you can find it in the main Debian repositories.
You know how to install it by now!

```bash 
sudo apt search vsftpd
```

To know *where* to configure the server we can look at the `man vsftpd` pages.
Scroll all the way to the bottom to see which files it uses to configure itself.
The configuration file itself is very *verbose* and should explain itself.

### SFTP

TODO