* What's the deal with [userspace and kernelspace](https://unix.stackexchange.com/questions/137820/whats-the-difference-of-the-userland-vs-the-kernel)?
* Why is Debian called [the universal operating system](https://www.reddit.com/r/debian/comments/22j0wf/so_why_debian_is_called_the_universal_operating/)?
In order to run virtual machines, or VM's, we need a **host** program.
One of the most popular ones out there is called `virtualbox`.
It should be installed on your machine but in case it's not you can download it [here](https://www.virtualbox.org/wiki/Downloads).
If you **expand** the section below you'll see a step by step walk-through of a VM creation in virtualbox with some notes on each step with best practice pointers.
1. Most of the time you'll want dynamic size meaning the disk **file** will only take up as much **space** as it needs.
For example, if you set a size of 50GB for your disk and your OS plus personal files take up 12GB, the actual space this image takes op on the physical disk of your **host** will be about 12GB.
1. By default your machine will have only **one** CPU core but you can add more via the settings.
This is something you can change whenever you need more processing power but the same rule as with the RAM applies, it's not a magical way to add resources to your machine.
I'll install a full blown and modern graphical Debian machine and I would like you to **not** do it yourself but take **notes** on each step so you'll be able to reference your notes later **when** I ask you to install a machine yourself.
Below is a gain a step by step walk through with some of my tips but a more details guide can be found in the [Debian Administrator Handbook](https://debian-handbook.info/browse/stable/sect.installation-steps.html).
This book is a real *bible* of information and I highly advise you to read through it.
My most important notes would be the following.
* **Read** each section carefully and you'll know what to do.
* Don't set a root password but use `sudo` instead.
* Don't install a graphical environment if you don't need one.
It's a lot easier to add one later than to remove it.
* Don't forget to install `grub` at the end otherwise your installation will not be able to boot.
You *can* recover from this mistake with [supergrub2](https://www.supergrubdisk.org/super-grub2-disk/) but that's for an other day.
Once these are installed have a look at the running services and programs via `htop` and compare it to both your graphical installation **and** your graphical install but running in `multi-user.target`.
Now, I don't think we have sufficiently *proven* that `date` is a full blown program so let's dig a bit deeper.
```
waldek@hellodebian:~$ which vlc
/usr/bin/vlc
waldek@hellodebian:~$ which date
/usr/bin/date
waldek@hellodebian:~$ which which
/usr/bin/which
```
There is quite bit to *unpack* in the example above.
First, what on earth is `which`?
Well, it's *also* a program and it's sole purpose in life is to tell you **where** a program is located on your system.
Because `which` by itself does not make a lot of sense it needs what we call an **argument**.
Here the argument is the name of the program we want to know it's location of.
The existence of arguments is the second big thing we discovered here.
The third new thing we can observe here is what we call **paths**, meaning locations on the system.
For example, `vlc` is a *binary* program located in a **folder** called `bin` which is located in a folder called `usr` which is at the *root* of your system.
If this sounds complicated, don't worry, we'll go into detail a bit later.
Now that we know **where** some of our programs are located, let's find out **what** they are.
The methodology is the same as with `which` but we'll use an other program called `file` who's purpose in life is to tell more about the **content** of a certain file.
Logically, `file` needs an argument and this argument is the **path** to the file you want to inspect.
```
waldek@hellodebian:~$ file /usr/bin/vlc
/usr/bin/vlc: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=51c40f8234213415771b3a344cab25a140543f8a, for GNU/Linux 3.2.0, stripped
waldek@hellodebian:~$ file /usr/bin/date
/usr/bin/date: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=b740f054aaef6a85aff024858c914c7eae70a6a5, for GNU/Linux 3.2.0, stripped
waldek@hellodebian:~$ file /usr/bin/which
/usr/bin/which: POSIX shell script, ASCII text executable
waldek@hellodebian:~$ file $(which which) # this is some command line kung fu...
/usr/bin/which: POSIX shell script, ASCII text executable
waldek@hellodebian:~$
```
Here we learn that both `vlc` and `date` are **executables**, compiled for an x86-64 system.
I would say they are both created *equally* no?
Both are actual programs.
But what about `which`?
It's also an executable but not compiled, it's a *POSIX shell script*.
Executing a file, or program, means you take this file and tell the computer it needs to execute the actions that are stored in the file.
Compiled programs contain actual [instructions](https://en.wikipedia.org/wiki/Instruction_set_architecture#Instructions) the computer understands out of the box.
This means that programs that are compiled are always compiled for a specify architecture which in our case is x86-64.
Interpreted programs are not compiled to machine code but when we run them each line is passed to an appropriate **interpreter** and executed **line by line**.
The first line of a script is often the path to the interpreter that *understands* the code that will follow.
Popular interpreted languages are `bash`, `sh`, `python`, `php`, ...
We can take it *one* more step forward and peak into the *content* of the files.
A nice little program to do this is called `head` who's purpose in life is to show the first few lines of a file.
!"<22><><EFBFBD><EFBFBD>H<EFBFBD>H<EFBFBD><48>DESKTOP_STARTUP_ID--no-ignore-config--media-library--dbusvlcorg.VideoLAN.VLCVLC/3.0.16VLC media playerglobalhotkeys,noneVLC is not supposed to be run as root. Sorry.
If you need to use real-time priorities and/or privileged TCP ports
you can use %s-wrapper (make sure it is Set-UID root and
```
And now of `which`.
```
waldek@hellodebian:~$ head /usr/bin/which
#! /bin/sh
set -ef
if test -n "$KSH_VERSION"; then
puts() {
print -r -- "$*"
}
else
puts() {
printf '%s\n' "$*"
waldek@hellodebian:~$
```
Notice how the `vlc` file is mostly **not human readable** but we can still make out some keywords?
The `which` file however is perfectly readable!
The first line of the `which` file is `#! /bin/sh` which is the path to the interpreter.
The `#!` is called a [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)).
We can dig into this to learn more about this mysterious `sh`.
```
waldek@hellodebian:~$ file /bin/sh
/bin/sh: symbolic link to dash
waldek@hellodebian:~$ which dash
/usr/bin/dash
waldek@hellodebian:~$ file /usr/bin/dash
/usr/bin/dash: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=cb6911fd56559717336c938bef1ce479b0a85b35, for GNU/Linux 3.2.0, stripped
waldek@hellodebian:~$
```
Surprise surprise!
The *interpreter* is a compiled program!
When you think about it it does make sense because at the end of the day the computer *only* understands machine instructions.
So when you execute a script each line of code is converted to machine instructions on the spot.
This makes scripts easier to write but slower at execution.
Linux does not rely as much on file extensions as windows does.
We could even say it often does not care about them.
Due to the decentralized nature of Linux development, some programs *do* care about them while others don't.
It's a bit of hit and miss but I advise you to read [this](https://askubuntu.com/questions/803434/do-file-extensions-have-any-purpose-in-linux) post on AskUbuntu for a more detailed explanation.
There are multiple ways to go to the same location on your system.
You can always go either in an **absolute** way, or in *multiple***relative** ways.
The root of your system is `/` and to list what we can find there we can use `ls`.
```
waldek@hellodebian:~$ cd /
waldek@hellodebian:/$ ls
bin dev home initrd.img.old lib32 libx32 media opt root sbin sys usr vmlinuz
boot etc initrd.img lib lib64 lost+found mnt proc run srv tmp var vmlinuz.old
waldek@hellodebian:/$
```
From here we can go back home in multiple ways.
```
waldek@hellodebian:/$ cd
waldek@hellodebian:~$ cd -
/
waldek@hellodebian:/$ cd home/waldek/
waldek@hellodebian:~$ cd -
/
waldek@hellodebian:/$ cd /home/waldek/
waldek@hellodebian:~$
```
The first one is the handy shortcut we learned, and we go back to the root of our system with the `cd -` shortcut.
The following two manipulations look very similar but there is a subtle difference.
The first one `home/waldek` is a **relative** path, and the second one `/home/waldek` is an **absolute** one.
Relative paths depend on where you are located, absolute ones always point to the same location.
An example.
```
waldek@hellodebian:~$ cd /usr/bin/
waldek@hellodebian:/usr/bin$ cd /home/waldek/
waldek@hellodebian:~$ cd -
/usr/bin
waldek@hellodebian:/usr/bin$ cd home/waldek
-bash: cd: home/waldek: No such file or directory
waldek@hellodebian:/usr/bin$
```
The second command fails because from where I'm standing, there is no folder called `home/waldek`!
I can however still go to my home in a relative way but I need to *go back* a few directories first.
```
waldek@hellodebian:/usr/bin$ cd ../../home/waldek/
waldek@hellodebian:~$ pwd
/home/waldek
waldek@hellodebian:~$
```
The `..` means go **back** one directory so in our example we go back **two** directories, which brings us to the `/` of our system and from there we go **up** to `home` and then `waldek`.
The `ls` program is one of the most used commands.
It's like a very basic *file explorer* in a graphical environment and just like it's graphical counter part you can use it to display it's data in different ways.
Think of the *details* versus *thumbnail* views in your favorite explorer.
But how do you do this?
We can **add flags or options** to the program to modify it's behavior.
Have a look at the output below.
```
waldek@hellodebian:~$ ls
Desktop Documents Downloads Music Pictures Public Templates Videos
waldek@hellodebian:~$ ls -l
total 32
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Desktop
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Documents
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Downloads
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Music
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Pictures
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Public
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Templates
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Videos
The `-l` and `-a` additions are what we call *flags* or *options*.
Aren't they just extra *arguments*?
There is no real rule to the naming of this concept but [this](https://unix.stackexchange.com/questions/285575/whats-the-difference-between-a-flag-an-option-and-an-argument) is a rather interesting take.
We can combine arguments if we want, and some programs even allow *concatenation* of arguments.
Have a look below to understand what I mean by that.
```
waldek@hellodebian:~$ ls -a -l
total 112
drwxr-xr-x 16 waldek waldek 4096 Feb 21 18:12 .
drwxr-xr-x 3 root root 4096 Feb 18 12:49 ..
-rw------- 1 waldek waldek 2772 Feb 22 10:45 .bash_history
-rw-r--r-- 1 waldek waldek 220 Feb 18 12:49 .bash_logout
-rw-r--r-- 1 waldek waldek 3526 Feb 18 12:49 .bashrc
drwx------ 16 waldek waldek 4096 Feb 18 14:24 .cache
drwx------ 20 waldek waldek 4096 Feb 18 14:23 .config
drwx------ 3 waldek waldek 4096 Feb 18 12:52 .dbus
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Desktop
-rw-r--r-- 1 waldek waldek 35 Feb 18 13:33 .dmrc
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Documents
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Downloads
drwx------ 2 waldek waldek 4096 Feb 18 15:07 .gnupg
drwxr-xr-x 3 waldek waldek 4096 Feb 18 12:50 .local
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Music
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Pictures
-rw-r--r-- 1 waldek waldek 807 Feb 18 12:49 .profile
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Public
-rw------- 1 waldek waldek 11 Feb 21 18:12 .python_history
drwx------ 2 waldek waldek 4096 Feb 21 12:45 .ssh
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Templates
-rw-r----- 1 waldek waldek 5 Feb 18 15:06 .vboxclient-clipboard.pid
-rw-r----- 1 waldek waldek 5 Feb 18 15:06 .vboxclient-display-svga-x11.pid
-rw-r----- 1 waldek waldek 5 Feb 18 15:06 .vboxclient-draganddrop.pid
-rw-r----- 1 waldek waldek 5 Feb 18 15:06 .vboxclient-seamless.pid
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Videos
-rw------- 1 waldek waldek 56 Feb 18 13:33 .Xauthority
-rw------- 1 waldek waldek 2701 Feb 18 13:33 .xsession-errors
waldek@hellodebian:~$ ls -la
total 112
drwxr-xr-x 16 waldek waldek 4096 Feb 21 18:12 .
drwxr-xr-x 3 root root 4096 Feb 18 12:49 ..
-rw------- 1 waldek waldek 2772 Feb 22 10:45 .bash_history
-rw-r--r-- 1 waldek waldek 220 Feb 18 12:49 .bash_logout
-rw-r--r-- 1 waldek waldek 3526 Feb 18 12:49 .bashrc
drwx------ 16 waldek waldek 4096 Feb 18 14:24 .cache
drwx------ 20 waldek waldek 4096 Feb 18 14:23 .config
drwx------ 3 waldek waldek 4096 Feb 18 12:52 .dbus
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Desktop
-rw-r--r-- 1 waldek waldek 35 Feb 18 13:33 .dmrc
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Documents
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Downloads
drwx------ 2 waldek waldek 4096 Feb 18 15:07 .gnupg
drwxr-xr-x 3 waldek waldek 4096 Feb 18 12:50 .local
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Music
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Pictures
-rw-r--r-- 1 waldek waldek 807 Feb 18 12:49 .profile
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Public
-rw------- 1 waldek waldek 11 Feb 21 18:12 .python_history
drwx------ 2 waldek waldek 4096 Feb 21 12:45 .ssh
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Templates
-rw-r----- 1 waldek waldek 5 Feb 18 15:06 .vboxclient-clipboard.pid
-rw-r----- 1 waldek waldek 5 Feb 18 15:06 .vboxclient-display-svga-x11.pid
-rw-r----- 1 waldek waldek 5 Feb 18 15:06 .vboxclient-draganddrop.pid
-rw-r----- 1 waldek waldek 5 Feb 18 15:06 .vboxclient-seamless.pid
drwxr-xr-x 2 waldek waldek 4096 Feb 18 12:50 Videos
-rw------- 1 waldek waldek 56 Feb 18 13:33 .Xauthority
-rw------- 1 waldek waldek 2701 Feb 18 13:33 .xsession-errors
waldek@hellodebian:~$
```
# Getting help
## Options
OK, so we have *a lot* of programs we can use on the command line, plus we can change how they work by adding obscure characters.
How on earth can we *discover* what a program can and can't do?
Introducing the **most important flag**: `--help`.
Most programs come with builtin help to explain you how to use the program.
Often, but not always, this help can be displayed on the terminal by adding the `--help` flag.
Let's investigate this.
```
waldek@hellodebian:~$ ls --help
Usage: ls [OPTION]... [FILE]...
List information about the FILEs (the current directory by default).
Sort entries alphabetically if none of -cftuvSUX nor --sort is specified.
Mandatory arguments to long options are mandatory for short options too.
-a, --all do not ignore entries starting with .
-A, --almost-all do not list implied . and ..
--author with -l, print the author of each file
-b, --escape print C-style escapes for nongraphic characters
--block-size=SIZE with -l, scale sizes by SIZE when printing them;
e.g., '--block-size=M'; see SIZE format below
-B, --ignore-backups do not list implied entries ending with ~
-c with -lt: sort by, and show, ctime (time of last
modification of file status information);
with -l: show ctime and sort by name;
otherwise: sort by ctime, newest first
-C list entries by columns
--color[=WHEN] colorize the output; WHEN can be 'always' (default
if omitted), 'auto', or 'never'; more info below
-d, --directory list directories themselves, not their contents
-D, --dired generate output designed for Emacs' dired mode
-f do not sort, enable -aU, disable -ls --color
-F, --classify append indicator (one of */=>@|) to entries
--file-type likewise, except do not append '*'
--format=WORD across -x, commas -m, horizontal -x, long -l,
single-column -1, verbose -l, vertical -C
--full-time like -l --time-style=full-iso
-g like -l, but do not list owner
--group-directories-first
group directories before files;
can be augmented with a --sort option, but any
use of --sort=none (-U) disables grouping
-G, --no-group in a long listing, don't print group names
-h, --human-readable with -l and -s, print sizes like 1K 234M 2G etc.
--si likewise, but use powers of 1000 not 1024
-H, --dereference-command-line
follow symbolic links listed on the command line
--dereference-command-line-symlink-to-dir
follow each command line symbolic link
that points to a directory
--hide=PATTERN do not list implied entries matching shell PATTERN
(overridden by -a or -A)
--hyperlink[=WHEN] hyperlink file names; WHEN can be 'always'
(default if omitted), 'auto', or 'never'
--indicator-style=WORD append indicator with style WORD to entry names:
none (default), slash (-p),
file-type (--file-type), classify (-F)
-i, --inode print the index number of each file
-I, --ignore=PATTERN do not list implied entries matching shell PATTERN
-k, --kibibytes default to 1024-byte blocks for disk usage;
used only with -s and per directory totals
-l use a long listing format
-L, --dereference when showing file information for a symbolic
link, show information for the file the link
references rather than for the link itself
-m fill width with a comma separated list of entries
-n, --numeric-uid-gid like -l, but list numeric user and group IDs
-N, --literal print entry names without quoting
-o like -l, but do not list group information
-p, --indicator-style=slash
append / indicator to directories
-q, --hide-control-chars print ? instead of nongraphic characters
--show-control-chars show nongraphic characters as-is (the default,
unless program is 'ls' and output is a terminal)
-Q, --quote-name enclose entry names in double quotes
--quoting-style=WORD use quoting style WORD for entry names:
literal, locale, shell, shell-always,
shell-escape, shell-escape-always, c, escape
(overrides QUOTING_STYLE environment variable)
-r, --reverse reverse order while sorting
-R, --recursive list subdirectories recursively
-s, --size print the allocated size of each file, in blocks
-S sort by file size, largest first
--sort=WORD sort by WORD instead of name: none (-U), size (-S),
time (-t), version (-v), extension (-X)
--time=WORD change the default of using modification times;
access time (-u): atime, access, use;
change time (-c): ctime, status;
birth time: birth, creation;
with -l, WORD determines which time to show;
with --sort=time, sort by WORD (newest first)
--time-style=TIME_STYLE time/date format with -l; see TIME_STYLE below
-t sort by time, newest first; see --time
-T, --tabsize=COLS assume tab stops at each COLS instead of 8
-u with -lt: sort by, and show, access time;
with -l: show access time and sort by name;
otherwise: sort by access time, newest first
-U do not sort; list entries in directory order
-v natural sort of (version) numbers within text
-w, --width=COLS set output width to COLS. 0 means no limit
-x list entries by lines instead of by columns
-X sort alphabetically by entry extension
-Z, --context print any security context of each file
-1 list one file per line. Avoid '\n' with -q or -b
--help display this help and exit
--version output version information and exit
The SIZE argument is an integer and optional unit (example: 10K is 10*1024).
Units are K,M,G,T,P,E,Z,Y (powers of 1024) or KB,MB,... (powers of 1000).
Binary prefixes can be used, too: KiB=K, MiB=M, and so on.
The TIME_STYLE argument can be full-iso, long-iso, iso, locale, or +FORMAT.
FORMAT is interpreted like in date(1). If FORMAT is FORMAT1<newline>FORMAT2,
then FORMAT1 applies to non-recent files and FORMAT2 to recent files.
TIME_STYLE prefixed with 'posix-' takes effect only outside the POSIX locale.
Also the TIME_STYLE environment variable sets the default style to use.
Using color to distinguish file types is disabled both by default and
with --color=never. With --color=auto, ls emits color codes only when
standard output is connected to a terminal. The LS_COLORS environment
variable can change the settings. Use the dircolors command to set it.
Exit status:
0 if OK,
1 if minor problems (e.g., cannot access subdirectory),
2 if serious trouble (e.g., cannot access command-line argument).
GNU coreutils online help: <https://www.gnu.org/software/coreutils/>
Full documentation <https://www.gnu.org/software/coreutils/ls>
or available locally via: info '(coreutils) ls invocation'
waldek@hellodebian:~$
```
The output above teaches us that the options `-la` change the output so it becomes:
*`-a, --all do not ignore entries starting with .`
*`-l use a long listing format`
To *not ignore entries starting with .* we can use two different flags, `-a` or `--all`.
The former is called short arguments and the latter long ones.
The long ones are more *verbose* so it's often easier to understand what a line *does* just by reading it.
Note that `ls` only has one way of showing the help by adding `--help` whereas the help for `htop` can be shown with both `-h` or `--help`.
This inconsistency is sadly a byproduct of the decentralized nature of Linux.
To add to this mess some programs use *single dash long form* (`-help`) but those programs are rather rare.
## Manuals
The `--help` flag is tremendously useful as it gives you an overview of how you can modify a programs behaviour, kind of like settings, but it doesn't always explain you what a program is *for*.
The is where the manuals come into play.
I see the `--help` flag as a *cheat sheet* to quickly discover options and the man pages as a reference.
We can read the manual by calling the `man` program with an argument.
This argument is the *name* of the program you want to read the manual for.
Same idea as with `which`.
```
waldek@hellodebian:~$ man ls
```
You're now in a program called `less`, which is a [pager](https://en.wikipedia.org/wiki/Terminal_pager).
You can scroll the manual with the arrows of `vi` based navigation (which we'll get into later).
If you hit `h` you'll see the builtin documentation of `less` and press `q` to quit (twice if you're in the help section).
This brings you back to your prompt.
It's worth reading the [wikipedia entry](https://en.wikipedia.org/wiki/Man_page) of the `man` command to understand it's history and operation.
The output below illustrates the *tree* of a terminal that's reading a manual.
In a `bash` shell you invoke `man ls` which in turn runs the `pager` program to display the content.
If you're intrigued by this `pstree` program go and read the manual!
```
waldek@hellodebian:~$ pstree 10845 -a
bash
└─man ls
└─pager
waldek@hellodebian:~$
```
Not every command has a manual.
For example, `cd` does not have one.
This is kind of logical because `cd` is a builtin.
To learn more about builtin commands you can use the `help` program.
On it's own it gives you all the builtin command you can invoke and with an argument it outputs the help about that specific command.
```
waldek@hellodebian:~$ help
GNU bash, version 5.1.4(1)-release (x86_64-pc-linux-gnu)
These shell commands are defined internally. Type `help' to see this list.
Type `help name' to find out more about the function `name'.
Use `info bash' to find out more about the shell in general.
Use `man -k' or `info' to find out more about commands not in this list.
A star (*) next to a name means that the command is disabled.
job_spec [&] history [-c] [-d offset] [n] or history -anrw [filename] or history -ps>
(( expression )) if COMMANDS; then COMMANDS; [ elif COMMANDS; then COMMANDS; ]... [ else>
. filename [arguments] jobs [-lnprs] [jobspec ...] or jobs -x command [args]
: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [>
