waldek
/
linux_introduction
1
0
Fork 0
Code Issues 1 Pull Requests Projects Releases Wiki Activity

Merge branch 'master' of ssh://86thumbs.net:3022/waldek/linux_introduction

This commit is contained in:
waldek 2022-06-28 20:21:50 +02:00
parent eddafa6545 fe76dac62a
commit 005e727434
3 changed files with 613 additions and 15 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

490
advanced/learning_shells.md
View File

@ -1,14 +1,498 @@
# `bash` login
TODO
What happens when we log into a server or when we open up a terminal with `bash` running in it?
As always the manual can tell us quite a bit.
Quite often you'll find a list of configuration files used by a program near the end of the manual.
Sometimes not, your millage may vary but here is the files section of the `bash` manual.
```
FILES
/bin/bash
The bash executable
/etc/profile
The systemwide initialization file, executed for login shells
/etc/bash.bashrc
The systemwide per-interactive-shell startup file
/etc/bash.bash.logout
The systemwide login shell cleanup file, executed when a login shell exits
~/.bash_profile
The personal initialization file, executed for login shells
~/.bashrc
The individual per-interactive-shell startup file
~/.bash_logout
The individual login shell cleanup file, executed when a login shell exits
~/.inputrc
Individual readline initialization file
```
I created a minimal container to play around a bit.
This is what I have in my home directory.
```
waldek@tester:~$ ls -la
total 24
drwxr-xr-x 2 waldek waldek 4096 Jun 15 15:49 .
drwxr-xr-x 3 root root 4096 Jun 15 15:47 ..
-rw------- 1 waldek waldek 39 Jun 15 15:50 .bash_history
-rw-r--r-- 1 waldek waldek 220 Jun 15 15:47 .bash_logout
-rw-r--r-- 1 waldek waldek 3526 Jun 15 15:47 .bashrc
-rw-r--r-- 1 waldek waldek 807 Jun 15 15:47 .profile
waldek@tester:~$
```
Let's have a look at them one by one.
First the `.bash_history`.
This contains a history of all the commands I ran on this computer!
Quite handy for when we forget how to do something but we know we've done it before.
You can use `grep` to search the file but there is also a shortcut you can use, `ctrl-r`.
```
waldek@tester:~$ cat .bash_history
vim .profile
su
ls -la
exit
su
logout
waldek@tester:~$
```
Next up is `.bash_logout`.
It's purpose is written in the comment on the first line!
This file gets executed when a login shell exits.
Is there a difference between a *shell* and a *login shell*?
I'll let you ponder this a bit and we'll come back to this later.
```
waldek@tester:~$ cat .bash_logout
# ~/.bash_logout: executed by bash(1) when login shell exits.
# when leaving the console clear the screen to increase privacy
if [ "$SHLVL" = 1 ]; then
[ -x /usr/bin/clear_console ] && /usr/bin/clear_console -q
fi
waldek@tester:~$
```
Now the `.bashrc` file.
I know it's a pretty long file so I'll just show the `head`.
```
waldek@tester:~$ head .bashrc
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples
# If not running interactively, don't do anything
case $- in
*i*) ;;
*) return;;
esac
waldek@tester:~$
```
Aha!
The comment on the first line mentions *non-login shells* so there must be a difference!
This is a very important configuration file and I urge you to read it, especially the comments.
What kind of content is in this file?
Last but not least `.profile`.
```
waldek@tester:~$ cat .profile
# ~/.profile: executed by the command interpreter for login shells.
# This file is not read by bash(1), if ~/.bash_profile or ~/.bash_login
# exists.
# see /usr/share/doc/bash/examples/startup-files for examples.
# the files are located in the bash-doc package.
# the default umask is set in /etc/profile; for setting the umask
# for ssh logins, install and configure the libpam-umask package.
#umask 022
# if running bash
if [ -n "$BASH_VERSION" ]; then
# include .bashrc if it exists
if [ -f "$HOME/.bashrc" ]; then
. "$HOME/.bashrc"
fi
fi
# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
PATH="$HOME/bin:$PATH"
fi
# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/.local/bin" ] ; then
PATH="$HOME/.local/bin:$PATH"
fi
waldek@tester:~$
```
## which files are sources when
TODO
So, we have different files that are sourced at different times.
Let's try to play around a bit.
I added some comments to them so we get a clear idea which are sourced when.
```
waldek@tester:~$ tail -n 1 .bashrc .profile .bash_logout
==> .bashrc <==
echo I am .bachrc
==> .profile <==
echo I am .profile
==> .bash_logout <==
echo I am .bash_logout
waldek@tester:~$
```
When I log in on a tty I get the following output.
This is an actual interactive login so we source both `.bashrc` and `.profile`.
You might ask yourself where the *other* output comes from.
That's up to you to find out!
```
Debian GNU/Linux 11 tester pts/1
tester login: waldek
Password:
Linux tester 5.10.0-15-amd64 #1 SMP Debian 5.10.120-1 (2022-06-09) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Jun 15 16:03:17 CEST 2022 on pts/1
I am .bachrc
I am .profile
waldek@tester:~$
```
What if we `su` to ourselves?
```
waldek@tester:~$ su waldek
Password:
I am .bachrc
waldek@tester:~$
```
The `.profile` is not sourced!
We can however change the behaviour of `su` with arguments.
A quick look at the `man su` tells us that.
```
-, -l, --login
Start the shell as a login shell with an environment similar to a real login:
o clears all the environment variables except TERM and variables specified by --whitelist-environment
o initializes the environment variables HOME, SHELL, USER, LOGNAME, and PATH
o changes to the target user's home directory
o sets argv[0] of the shell to '-' in order to make the shell a login shell
```
So we can do the following.
```
waldek@tester:~$ su - waldek
Password:
I am .bachrc
I am .profile
waldek@tester:~$
```
If we `exit` this session we get the following.
```
waldek@tester:~$ exit
logout
I am .bash_logout
waldek@tester:~$
```
Ah, there is out `.bash_logout` message!
And if we exit the tty session we get this.
```
waldek@tester:~$ exit
logout
I am .bash_logout
Debian GNU/Linux 11 tester pts/1
tester login:
```
When we launch a bash in a bash we get the following.
But when we launch a script, none of our files get sourced!
```
waldek@tester:~$ bash
I am .bachrc
waldek@tester:~$ exit
exit
waldek@tester:~$ bash script.sh
I'm a script
waldek@tester:~$
```
What about `ssh` connections?
I'm running the server on port 2222 because it's a contained with host networking.
It's a bit too early to go into too much detail but by then end of the course you'll fully understand what's happening!
```
waldek@tester:~$ ssh localhost -p 2222
waldek@localhost's password:
Linux tester 5.10.0-15-amd64 #1 SMP Debian 5.10.120-1 (2022-06-09) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Jun 15 16:29:26 2022 from ::1
I am .bachrc
I am .profile
waldek@tester:~$ exit
logout
I am .bash_logout
Connection to localhost closed.
waldek@tester:~$
```
What if we run a command over `ssh`?
Then none of these files are sourced!
```
waldek@tester:~$ ssh localhost -p 2222 ls -la
waldek@localhost's password:
total 36
drwxr-xr-x 3 waldek waldek 4096 Jun 15 16:29 .
drwxr-xr-x 3 root root 4096 Jun 15 15:47 ..
-rw------- 1 waldek waldek 649 Jun 15 16:27 .bash_history
-rw-r--r-- 1 waldek waldek 243 Jun 15 16:19 .bash_logout
-rw-r--r-- 1 waldek waldek 3545 Jun 15 16:20 .bashrc
-rw-r--r-- 1 waldek waldek 826 Jun 15 16:17 .profile
drwx------ 2 waldek waldek 4096 Jun 15 16:29 .ssh
-rw------- 1 waldek waldek 3141 Jun 15 16:20 .viminfo
-rw-r--r-- 1 waldek waldek 34 Jun 15 16:18 script.sh
waldek@tester:~$
```
# A clean slate
I deleted all the content of our three files.
```
waldek@tester:~$ cat .bashrc .bash_logout .profile
waldek@tester:~$
```
We don't see any differences but our shell is *less* powerful.
For example, autocomplete does not work anymore on things like `systemctl`.
We now have a `bash` shell that has only sourced actual content from the following files.
* `/etc/profile`
* `/etc/bash.bashrc`
* `/etc/bash.bash.logout` if it exists
As these are system files we can only edit them as `root`.
I went a head and added a comment `#` in front of every line.
When I now log in on a tty I get the following.
```
Debian GNU/Linux 11 tester pts/1
tester login: waldek
Password:
Linux tester 5.10.0-15-amd64 #1 SMP Debian 5.10.120-1 (2022-06-09) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Jun 15 16:44:05 CEST 2022 on pts/1
-bash-5.1$
```
This is a fully barebones `bash` shell, no bells and whistles at all.
You can see our prompt has changed to a super simple one, no more username nor hostname nor path.
Tab complete works, but only for programs and paths, not for arguments.
# Prompt customization
TODO
Form this basic session we can learn quite a bit.
The most obvious change is our prompt.
The fact that it changed means we can modify it too!
Let's discover how.
From the manual we can conclude that `$PSX` are the variables that hold the prompt.
```
PS0 The value of this parameter is expanded (see PROMPTING below) and displayed by interactive shells after reading a command and before
the command is executed.
PS1 The value of this parameter is expanded (see PROMPTING below) and used as the primary prompt string. The default value is ``\s-\v\$
''.
PS2 The value of this parameter is expanded as with PS1 and used as the secondary prompt string. The default is ``> ''.
PS3 The value of this parameter is used as the prompt for the select command (see SHELL GRAMMAR above).
PS4 The value of this parameter is expanded as with PS1 and the value is printed before each command bash displays during an execution
trace. The first character of the expanded value of PS4 is replicated multiple times, as necessary, to indicate multiple levels of
indirection. The default is ``+ ''.
```
```
-bash-5.1$ echo $PS
$PS1 $PS2 $PS4
-bash-5.1$ echo $PS1
\s-\v\$
-bash-5.1$
```
Your prompt is defined by the `$PS1` variable.
By changing this we can change it's behaviour.
```
-bash-5.1$ PS1="helloworld"
helloworldls
script.sh
helloworldcd /
helloworldls
bin boot dev etc home lib lib32 lib64 libx32 media mnt opt proc root run sbin srv sys tmp usr var
helloworld
```
This is *far* from useful but it's just to prove a point.
Let's try to get our username back.
Again, by reading the manual we can see all variables we can use in a prompt.
I include the **PROMPTING** section for completeness.
```
PROMPTING
When executing interactively, bash displays the primary prompt PS1 when it is ready to read a command, and the secondary prompt PS2 when it
needs more input to complete a command. Bash displays PS0 after it reads a command but before executing it. Bash displays PS4 as described
above before tracing each command when the -x option is enabled. Bash allows these prompt strings to be customized by inserting a number of
backslash-escaped special characters that are decoded as follows:
\a an ASCII bell character (07)
\d the date in "Weekday Month Date" format (e.g., "Tue May 26")
\D{format}
the format is passed to strftime(3) and the result is inserted into the prompt string; an empty format results in a locale-
specific time representation. The braces are required
\e an ASCII escape character (033)
\h the hostname up to the first `.'
\H the hostname
\j the number of jobs currently managed by the shell
\l the basename of the shell's terminal device name
\n newline
\r carriage return
\s the name of the shell, the basename of $0 (the portion following the final slash)
\t the current time in 24-hour HH:MM:SS format
\T the current time in 12-hour HH:MM:SS format
\@ the current time in 12-hour am/pm format
\A the current time in 24-hour HH:MM format
\u the username of the current user
\v the version of bash (e.g., 2.00)
\V the release of bash, version + patch level (e.g., 2.00.0)
\w the current working directory, with $HOME abbreviated with a tilde (uses the value of the PROMPT_DIRTRIM variable)
\W the basename of the current working directory, with $HOME abbreviated with a tilde
\! the history number of this command
\# the command number of this command
\$ if the effective UID is 0, a #, otherwise a $
\nnn the character corresponding to the octal number nnn
\\ a backslash
\[ begin a sequence of non-printing characters, which could be used to embed a terminal control sequence into the prompt
\] end a sequence of non-printing characters
The command number and the history number are usually different: the history number of a command is its position in the history list, which
may include commands restored from the history file (see HISTORY below), while the command number is the position in the sequence of com‐
mands executed during the current shell session. After the string is decoded, it is expanded via parameter expansion, command substitution
or contain characters special to word expansion.
```
So, to get our username back we need the `\u` variable.
```
helloworldPS1="\u"
waldekls
bin boot dev etc home lib lib32 lib64 libx32 media mnt opt proc root run sbin srv sys tmp usr var
waldek
```
That *glue* between our username and command is rather annoying, let's modify that a bit.
```
waldekPS1="\u -> "
waldek -> ls
bin boot dev etc home lib lib32 lib64 libx32 media mnt opt proc root run sbin srv sys tmp usr var
waldek ->
```
Ah, much better!
Now, what time is it?
```
waldek -> PS1="\u \t -> "
waldek 17:02:53 -> ls
bin boot dev etc home lib lib32 lib64 libx32 media mnt opt proc root run sbin srv sys tmp usr var
waldek 17:02:56 ->
```
And where am I?
```
waldek@tester 17:04:10 -> PS1="\u@\H \t -> "
waldek@tester 17:04:19 -> ls
bin boot dev etc home lib lib32 lib64 libx32 media mnt opt proc root run sbin srv sys tmp usr var
waldek@tester 17:04:20 ->
```
And in which directory?
```
waldek@tester 17:04:20 -> PS1="\u@\H:\w \t -> "
waldek@tester:/ 17:05:06 -> cd /var/log/
waldek@tester:/var/log 17:05:13 ->
```
What about the *other* `$PS` variables?
Well, if we want to know the exact time when we launch a command we can modify the `$PS0` variable.
```
waldek@tester:/var/log -> PS0="it is \t \n"
waldek@tester:/var/log -> ls
it is 17:06:48
alternatives.log auth.log btmp dpkg.log journal messages runit user.log
apt bootstrap.log daemon.log faillog lastlog private syslog wtmp
waldek@tester:/var/log -> cd
it is 17:06:58
waldek@tester:~ ->
```
## Decoding the *base* prompt
On a modern install your `$PS1` will probably be as one of the two below.
They are actually the *same*, just one has **colors** in it, the other one not!
Customizing colors of a prompt is a bit of a nightmare but well worth the exercise!
* `PS1='\u@\h:\w\$ '`
* `PS1='\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '']]]]'`
# Alternative shells

56
essential/introduction_to_administration.md
View File

@ -571,7 +571,61 @@ steve@debian:/home/waldek$
## `umask`
TODO
When you create a **new** file of directory, there are created with default permissions according to the `umask`.
```
waldek@helloworld:~$ whatis umask
umask (2) - set file mode creation mask
waldek@helloworld:~$ umask
0022
waldek@helloworld:~$ touch new_file && ls -l new_file
-rw-r--r-- 1 waldek waldek 0 Jun 15 15:02 new_file
waldek@helloworld:~$ mkdir new_folder && ls -la new_folder
total 8
drwxr-xr-x 2 waldek waldek 4096 Jun 15 15:03 .
drwxr-xr-x 59 waldek waldek 4096 Jun 15 15:03 ..
waldek@helloworld:~$
```
The `new_file` has a octal permission of `644` and the `new_folder` `755`.
I can change my `umask` as follows.
```
waldek@helloworld:~$ rm -r new_f*
waldek@helloworld:~$ umask 000
waldek@helloworld:~$ touch new_file && ls -l new_file
-rw-rw-rw- 1 waldek waldek 0 Jun 15 15:08 new_file
waldek@helloworld:~$ mkdir new_folder && ls -la new_folder
total 8
drwxrwxrwx 2 waldek waldek 4096 Jun 15 15:08 .
drwxr-xr-x 59 waldek waldek 4096 Jun 15 15:08 ..
waldek@helloworld:~$
```
The permissions changed!
The `new_file` now has a octal permission of `666` and the `new_folder` `777`.
Do you see a pattern here?
The `umask` value is subtracted from the [POSIX](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap01.html#tag_17_01_01_04) defined base permissions which are `666` for a file and `777` for a directory.
One more example to drive it home.
```
waldek@helloworld:~$ rm new_f*
rm: cannot remove 'new_folder': Is a directory
waldek@helloworld:~$ rm -r new_f*
waldek@helloworld:~$ umask 777
waldek@helloworld:~$ touch new_file && ls -l new_file
---------- 1 waldek waldek 0 Jun 15 15:11 new_file
waldek@helloworld:~$ mkdir new_folder && ls -la new_folder
ls: cannot open directory 'new_folder': Permission denied
waldek@helloworld:~$
```
Can you explain me why we get a `Permission denied` message?
Some of you might have noticed the `umask` returns **4** digits.
This is because there is an additional permission bit we have not covered yet.
If you want to dig into it I suggest [this](https://wiki.debian.org/Permissions#Section_2:_UNIX_permissions_explained) page on the Debian wiki where I got the following quote from.
> The first of the 4 octal digits which represent permissions contains the setuid and setgid bits. These can be used to override some of the defaults described above but it is not worth getting into details other than to note that the user private groups project collaboration idiom (see below) depends on the behavior of the setgid bit.
## Essential programs

82
essential/introduction_to_the_commandline.md
View File

@ -984,8 +984,7 @@ waldek@hellodebian:~/Documents$
## Links
TODO
TODO - symlinks
# Editors
@ -1188,7 +1187,7 @@ waldek@debian:~$
```
You'll probably be a bit confused by the `|` part of the commands above.
Don't worry, give it a few hours paragraphs and it will become fully transparent!
Don't worry, give it a few hours and it will hopefully become fully transparent!
Next, there is a very high probability that the commands above don't immediately work for you.
This is because `locate` functions with a database which needs to be updated from time to time.
@ -1330,14 +1329,14 @@ waldek@debian:~$
Adding the `--color` argument to `grep` will make the matched patterns jump out with a color, depending on the color scheme of your terminal.
## Wildcards and regular expressions
## Wild cards and regular expressions
Both are quite related and for simplicities sake you can view regular expressions as wildcards on steroids.
A more detailed explaination can be found [here](https://unix.stackexchange.com/questions/57957/how-do-regular-expressions-differ-from-wildcards-used-to-filter-files).
Both are quite related and for simplicities sake you can view regular expressions as wild cards on steroids.
A more detailed explanation can be found [here](https://unix.stackexchange.com/questions/57957/how-do-regular-expressions-differ-from-wildcards-used-to-filter-files).
### Wildcards
### Wild cards
The syntax for wildcards is rater simple and also goes by the name of [globbing](https://en.wikipedia.org/wiki/Globbing) of [filename expansion](https://www.gnu.org/software/bash/manual/html_node/Filename-Expansion.html).
The syntax for wild cards is rater simple and also goes by the name of [globbing](https://en.wikipedia.org/wiki/Globbing) of [filename expansion](https://www.gnu.org/software/bash/manual/html_node/Filename-Expansion.html).
The most important rules to remember are the following.
| character | meaning |
@ -1382,7 +1381,7 @@ waldek@helloworld:~$
I hid a lot of words in [this](../assets/find_words_in_here.txt) file.
How many can you find?
What do they all have in common?
Can you count the occurences?
Can you count the occurrences?
* [exercise from Linux long](https://gitea.86thumbs.net/waldek/linux_course_doc/src/branch/master/modules/qualifying/learning_regex.md)
@ -1392,7 +1391,34 @@ TODO
# Pipes and redirects
TODO basic overview
I mentioned before that `bash` only knows *characters*.
It was kind of a lie.
`bash` only knows **bytes** at it's input and output.
If these bytes can be interpreted as *characters* you'll see readable output on your terminal, if not you'll see some gibberish.
An example.
```
waldek@helloworld:~$ echo "$(dd if=/dev/urandom bs=64 count=1 status=none)"
<EFBFBD>B<EFBFBD><EFBFBD>-<2D>j<EFBFBD><6A><EFBFBD><EFBFBD><EFBFBD><EFBFBD>s<EFBFBD>Ɠ8<C693><38>J<EFBFBD>!
<20>ד<EFBFBD>7d<37>/<2F>X<EFBFBD><58><EFBFBD><EFBFBD>!@{<7B>{R<><52><EFBFBD><EFBFBD><EFBFBD><EFBFBD>J):<3A>sB<73><42>+c<>1<EFBFBD>
waldek@helloworld:~$
```
Recognise this *gibberish*?
It's very similar to the output we got when reading a binary file no?
The command above generates 64 random bytes and print them to the screen.
We can do the same for a string and can use `pv` to get some statistics.
```
waldek@helloworld:~$ whatis pv
pv (1) - monitor the progress of data through a pipe
waldek@helloworld:~$ echo "hello world" | pv
hello world
12.0 B 0:00:00 [ 116KiB/s ] [ <=> ]
waldek@helloworld:~$
```
Don't worry about the `|` symbol, that's up next!
## Redirecting
@ -1640,7 +1666,41 @@ waldek@debian:~$ grep -R waldek /etc/ &> files_with_my_name
waldek@debian:~$
```
TODO - input redirection
### Input redirection
This is a bit more exotic but widely used in scripting.
I personally don't use it that often on the command line.
But we can demonstrate it's behaviour with a simple TCP server.
In a **first** terminal I run the following command.
If you try this out you'll see it *hang* after hitting enter, this is normal, the server is now listening for incoming connections
```
waldek@helloworld:~$ cat message
hello world!
I'm a text file...
waldek@helloworld:~$ cat < message
hello world!
I'm a text file...
waldek@helloworld:~$ nc -l -p 9999 < message && echo "done serving the file"
done serving the file
waldek@helloworld:~$
```
In a **second** terminal I now connect to this server.
I receive the content of the text file here and when I hit `ctrl-c` I break the connection.
The first terminal will now continue and say `done serving the file`.
```
waldek@helloworld:~$ nc localhost 9999
hello world!
I'm a text file...
^C
waldek@helloworld:~$
```
## Piping