From 10605055d679c2ed485adcb8322d13b89bcf6199 Mon Sep 17 00:00:00 2001 From: waldek Date: Wed, 15 Jun 2022 15:38:52 +0200 Subject: [PATCH 1/2] adds umask and input redirect --- essential/introduction_to_administration.md | 56 ++++++++++++- essential/introduction_to_the_commandline.md | 82 +++++++++++++++++--- 2 files changed, 126 insertions(+), 12 deletions(-) diff --git a/essential/introduction_to_administration.md b/essential/introduction_to_administration.md index 91488ce..aecb0e3 100644 --- a/essential/introduction_to_administration.md +++ b/essential/introduction_to_administration.md @@ -571,7 +571,61 @@ steve@debian:/home/waldek$ ## `umask` -TODO +When you create a **new** file of directory, there are created with default permissions according to the `umask`. + +``` +waldek@helloworld:~$ whatis umask +umask (2) - set file mode creation mask +waldek@helloworld:~$ umask +0022 +waldek@helloworld:~$ touch new_file && ls -l new_file +-rw-r--r-- 1 waldek waldek 0 Jun 15 15:02 new_file +waldek@helloworld:~$ mkdir new_folder && ls -la new_folder +total 8 +drwxr-xr-x 2 waldek waldek 4096 Jun 15 15:03 . +drwxr-xr-x 59 waldek waldek 4096 Jun 15 15:03 .. +waldek@helloworld:~$ +``` + +The `new_file` has a octal permission of `644` and the `new_folder` `755`. +I can change my `umask` as follows. + +``` +waldek@helloworld:~$ rm -r new_f* +waldek@helloworld:~$ umask 000 +waldek@helloworld:~$ touch new_file && ls -l new_file +-rw-rw-rw- 1 waldek waldek 0 Jun 15 15:08 new_file +waldek@helloworld:~$ mkdir new_folder && ls -la new_folder +total 8 +drwxrwxrwx 2 waldek waldek 4096 Jun 15 15:08 . +drwxr-xr-x 59 waldek waldek 4096 Jun 15 15:08 .. +waldek@helloworld:~$ +``` + +The permissions changed! +The `new_file` now has a octal permission of `666` and the `new_folder` `777`. +Do you see a pattern here? +The `umask` value is subtracted from the [POSIX](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap01.html#tag_17_01_01_04) defined base permissions which are `666` for a file and `777` for a directory. +One more example to drive it home. + +``` +waldek@helloworld:~$ rm new_f* +rm: cannot remove 'new_folder': Is a directory +waldek@helloworld:~$ rm -r new_f* +waldek@helloworld:~$ umask 777 +waldek@helloworld:~$ touch new_file && ls -l new_file +---------- 1 waldek waldek 0 Jun 15 15:11 new_file +waldek@helloworld:~$ mkdir new_folder && ls -la new_folder +ls: cannot open directory 'new_folder': Permission denied +waldek@helloworld:~$ +``` + +Can you explain me why we get a `Permission denied` message? +Some of you might have noticed the `umask` returns **4** digits. +This is because there is an additional permission bit we have not covered yet. +If you want to dig into it I suggest [this](https://wiki.debian.org/Permissions#Section_2:_UNIX_permissions_explained) page on the Debian wiki where I got the following quote from. + +> The first of the 4 octal digits which represent permissions contains the setuid and setgid bits. These can be used to override some of the defaults described above but it is not worth getting into details other than to note that the user private groups project collaboration idiom (see below) depends on the behavior of the setgid bit. ## Essential programs diff --git a/essential/introduction_to_the_commandline.md b/essential/introduction_to_the_commandline.md index 9649d30..1c2c01d 100644 --- a/essential/introduction_to_the_commandline.md +++ b/essential/introduction_to_the_commandline.md @@ -984,8 +984,7 @@ waldek@hellodebian:~/Documents$ ## Links -TODO - +TODO - symlinks # Editors @@ -1188,7 +1187,7 @@ waldek@debian:~$ ``` You'll probably be a bit confused by the `|` part of the commands above. -Don't worry, give it a few hours paragraphs and it will become fully transparent! +Don't worry, give it a few hours and it will hopefully become fully transparent! Next, there is a very high probability that the commands above don't immediately work for you. This is because `locate` functions with a database which needs to be updated from time to time. @@ -1330,14 +1329,14 @@ waldek@debian:~$ Adding the `--color` argument to `grep` will make the matched patterns jump out with a color, depending on the color scheme of your terminal. -## Wildcards and regular expressions +## Wild cards and regular expressions -Both are quite related and for simplicities sake you can view regular expressions as wildcards on steroids. -A more detailed explaination can be found [here](https://unix.stackexchange.com/questions/57957/how-do-regular-expressions-differ-from-wildcards-used-to-filter-files). +Both are quite related and for simplicities sake you can view regular expressions as wild cards on steroids. +A more detailed explanation can be found [here](https://unix.stackexchange.com/questions/57957/how-do-regular-expressions-differ-from-wildcards-used-to-filter-files). -### Wildcards +### Wild cards -The syntax for wildcards is rater simple and also goes by the name of [globbing](https://en.wikipedia.org/wiki/Globbing) of [filename expansion](https://www.gnu.org/software/bash/manual/html_node/Filename-Expansion.html). +The syntax for wild cards is rater simple and also goes by the name of [globbing](https://en.wikipedia.org/wiki/Globbing) of [filename expansion](https://www.gnu.org/software/bash/manual/html_node/Filename-Expansion.html). The most important rules to remember are the following. | character | meaning | @@ -1382,7 +1381,7 @@ waldek@helloworld:~$ I hid a lot of words in [this](../assets/find_words_in_here.txt) file. How many can you find? What do they all have in common? -Can you count the occurences? +Can you count the occurrences? * [exercise from Linux long](https://gitea.86thumbs.net/waldek/linux_course_doc/src/branch/master/modules/qualifying/learning_regex.md) @@ -1392,7 +1391,34 @@ TODO # Pipes and redirects -TODO basic overview +I mentioned before that `bash` only knows *characters*. +It was kind of a lie. +`bash` only knows **bytes** at it's input and output. +If these bytes can be interpreted as *characters* you'll see readable output on your terminal, if not you'll see some gibberish. +An example. + +``` +waldek@helloworld:~$ echo "$(dd if=/dev/urandom bs=64 count=1 status=none)" +�B��-�j������s�Ɠ8��J�! + �ד�7d�/�X����!@{�{R������J):�sB��+c�1� +waldek@helloworld:~$ +``` + +Recognise this *gibberish*? +It's very similar to the output we got when reading a binary file no? +The command above generates 64 random bytes and print them to the screen. +We can do the same for a string and can use `pv` to get some statistics. + +``` +waldek@helloworld:~$ whatis pv +pv (1) - monitor the progress of data through a pipe +waldek@helloworld:~$ echo "hello world" | pv +hello world +12.0 B 0:00:00 [ 116KiB/s ] [ <=> ] +waldek@helloworld:~$ +``` + +Don't worry about the `|` symbol, that's up next! ## Redirecting @@ -1640,7 +1666,41 @@ waldek@debian:~$ grep -R waldek /etc/ &> files_with_my_name waldek@debian:~$ ``` -TODO - input redirection +### Input redirection + +This is a bit more exotic but widely used in scripting. +I personally don't use it that often on the command line. +But we can demonstrate it's behaviour with a simple TCP server. + +In a **first** terminal I run the following command. +If you try this out you'll see it *hang* after hitting enter, this is normal, the server is now listening for incoming connections + +``` +waldek@helloworld:~$ cat message +hello world! +I'm a text file... + +waldek@helloworld:~$ cat < message +hello world! +I'm a text file... + +waldek@helloworld:~$ nc -l -p 9999 < message && echo "done serving the file" +done serving the file +waldek@helloworld:~$ +``` + +In a **second** terminal I now connect to this server. +I receive the content of the text file here and when I hit `ctrl-c` I break the connection. +The first terminal will now continue and say `done serving the file`. + +``` +waldek@helloworld:~$ nc localhost 9999 +hello world! +I'm a text file... + +^C +waldek@helloworld:~$ +``` ## Piping From fe76dac62a9b6a3b74ce26081261f39d9b168c24 Mon Sep 17 00:00:00 2001 From: waldek Date: Wed, 15 Jun 2022 17:19:33 +0200 Subject: [PATCH 2/2] first documentation of shell modding --- advanced/learning_shells.md | 490 +++++++++++++++++++++++++++++++++++- 1 file changed, 487 insertions(+), 3 deletions(-) diff --git a/advanced/learning_shells.md b/advanced/learning_shells.md index ef5c416..d06a48f 100644 --- a/advanced/learning_shells.md +++ b/advanced/learning_shells.md @@ -1,14 +1,498 @@ # `bash` login -TODO +What happens when we log into a server or when we open up a terminal with `bash` running in it? +As always the manual can tell us quite a bit. +Quite often you'll find a list of configuration files used by a program near the end of the manual. +Sometimes not, your millage may vary but here is the files section of the `bash` manual. + +``` +FILES + /bin/bash + The bash executable + /etc/profile + The systemwide initialization file, executed for login shells + /etc/bash.bashrc + The systemwide per-interactive-shell startup file + /etc/bash.bash.logout + The systemwide login shell cleanup file, executed when a login shell exits + ~/.bash_profile + The personal initialization file, executed for login shells + ~/.bashrc + The individual per-interactive-shell startup file + ~/.bash_logout + The individual login shell cleanup file, executed when a login shell exits + ~/.inputrc + Individual readline initialization file + +``` + +I created a minimal container to play around a bit. +This is what I have in my home directory. + +``` +waldek@tester:~$ ls -la +total 24 +drwxr-xr-x 2 waldek waldek 4096 Jun 15 15:49 . +drwxr-xr-x 3 root root 4096 Jun 15 15:47 .. +-rw------- 1 waldek waldek 39 Jun 15 15:50 .bash_history +-rw-r--r-- 1 waldek waldek 220 Jun 15 15:47 .bash_logout +-rw-r--r-- 1 waldek waldek 3526 Jun 15 15:47 .bashrc +-rw-r--r-- 1 waldek waldek 807 Jun 15 15:47 .profile +waldek@tester:~$ +``` + +Let's have a look at them one by one. +First the `.bash_history`. +This contains a history of all the commands I ran on this computer! +Quite handy for when we forget how to do something but we know we've done it before. +You can use `grep` to search the file but there is also a shortcut you can use, `ctrl-r`. + +``` +waldek@tester:~$ cat .bash_history +vim .profile +su +ls -la +exit +su +logout +waldek@tester:~$ +``` + +Next up is `.bash_logout`. +It's purpose is written in the comment on the first line! +This file gets executed when a login shell exits. +Is there a difference between a *shell* and a *login shell*? +I'll let you ponder this a bit and we'll come back to this later. + +``` +waldek@tester:~$ cat .bash_logout +# ~/.bash_logout: executed by bash(1) when login shell exits. + +# when leaving the console clear the screen to increase privacy + +if [ "$SHLVL" = 1 ]; then + [ -x /usr/bin/clear_console ] && /usr/bin/clear_console -q +fi +waldek@tester:~$ +``` + +Now the `.bashrc` file. +I know it's a pretty long file so I'll just show the `head`. + +``` +waldek@tester:~$ head .bashrc +# ~/.bashrc: executed by bash(1) for non-login shells. +# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc) +# for examples + +# If not running interactively, don't do anything +case $- in + *i*) ;; + *) return;; +esac + +waldek@tester:~$ +``` + +Aha! +The comment on the first line mentions *non-login shells* so there must be a difference! +This is a very important configuration file and I urge you to read it, especially the comments. +What kind of content is in this file? + +Last but not least `.profile`. + +``` +waldek@tester:~$ cat .profile +# ~/.profile: executed by the command interpreter for login shells. +# This file is not read by bash(1), if ~/.bash_profile or ~/.bash_login +# exists. +# see /usr/share/doc/bash/examples/startup-files for examples. +# the files are located in the bash-doc package. + +# the default umask is set in /etc/profile; for setting the umask +# for ssh logins, install and configure the libpam-umask package. +#umask 022 + +# if running bash +if [ -n "$BASH_VERSION" ]; then + # include .bashrc if it exists + if [ -f "$HOME/.bashrc" ]; then + . "$HOME/.bashrc" + fi +fi + +# set PATH so it includes user's private bin if it exists +if [ -d "$HOME/bin" ] ; then + PATH="$HOME/bin:$PATH" +fi + +# set PATH so it includes user's private bin if it exists +if [ -d "$HOME/.local/bin" ] ; then + PATH="$HOME/.local/bin:$PATH" +fi +waldek@tester:~$ +``` ## which files are sources when -TODO +So, we have different files that are sourced at different times. +Let's try to play around a bit. +I added some comments to them so we get a clear idea which are sourced when. + +``` +waldek@tester:~$ tail -n 1 .bashrc .profile .bash_logout +==> .bashrc <== +echo I am .bachrc + +==> .profile <== +echo I am .profile + +==> .bash_logout <== +echo I am .bash_logout +waldek@tester:~$ +``` + +When I log in on a tty I get the following output. +This is an actual interactive login so we source both `.bashrc` and `.profile`. +You might ask yourself where the *other* output comes from. +That's up to you to find out! + +``` +Debian GNU/Linux 11 tester pts/1 + +tester login: waldek +Password: +Linux tester 5.10.0-15-amd64 #1 SMP Debian 5.10.120-1 (2022-06-09) x86_64 + +The programs included with the Debian GNU/Linux system are free software; +the exact distribution terms for each program are described in the +individual files in /usr/share/doc/*/copyright. + +Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent +permitted by applicable law. +Last login: Wed Jun 15 16:03:17 CEST 2022 on pts/1 +I am .bachrc +I am .profile +waldek@tester:~$ +``` + +What if we `su` to ourselves? + +``` +waldek@tester:~$ su waldek +Password: +I am .bachrc +waldek@tester:~$ +``` + +The `.profile` is not sourced! +We can however change the behaviour of `su` with arguments. +A quick look at the `man su` tells us that. + +``` + -, -l, --login + Start the shell as a login shell with an environment similar to a real login: + + o clears all the environment variables except TERM and variables specified by --whitelist-environment + + o initializes the environment variables HOME, SHELL, USER, LOGNAME, and PATH + + o changes to the target user's home directory + + o sets argv[0] of the shell to '-' in order to make the shell a login shell + +``` + +So we can do the following. + +``` +waldek@tester:~$ su - waldek +Password: +I am .bachrc +I am .profile +waldek@tester:~$ +``` + +If we `exit` this session we get the following. + +``` +waldek@tester:~$ exit +logout +I am .bash_logout +waldek@tester:~$ +``` + +Ah, there is out `.bash_logout` message! +And if we exit the tty session we get this. + +``` +waldek@tester:~$ exit +logout +I am .bash_logout + +Debian GNU/Linux 11 tester pts/1 + +tester login: +``` + +When we launch a bash in a bash we get the following. +But when we launch a script, none of our files get sourced! + +``` +waldek@tester:~$ bash +I am .bachrc +waldek@tester:~$ exit +exit +waldek@tester:~$ bash script.sh +I'm a script +waldek@tester:~$ +``` + +What about `ssh` connections? +I'm running the server on port 2222 because it's a contained with host networking. +It's a bit too early to go into too much detail but by then end of the course you'll fully understand what's happening! + +``` +waldek@tester:~$ ssh localhost -p 2222 +waldek@localhost's password: +Linux tester 5.10.0-15-amd64 #1 SMP Debian 5.10.120-1 (2022-06-09) x86_64 + +The programs included with the Debian GNU/Linux system are free software; +the exact distribution terms for each program are described in the +individual files in /usr/share/doc/*/copyright. + +Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent +permitted by applicable law. +Last login: Wed Jun 15 16:29:26 2022 from ::1 +I am .bachrc +I am .profile +waldek@tester:~$ exit +logout +I am .bash_logout +Connection to localhost closed. +waldek@tester:~$ +``` + +What if we run a command over `ssh`? +Then none of these files are sourced! + +``` +waldek@tester:~$ ssh localhost -p 2222 ls -la +waldek@localhost's password: +total 36 +drwxr-xr-x 3 waldek waldek 4096 Jun 15 16:29 . +drwxr-xr-x 3 root root 4096 Jun 15 15:47 .. +-rw------- 1 waldek waldek 649 Jun 15 16:27 .bash_history +-rw-r--r-- 1 waldek waldek 243 Jun 15 16:19 .bash_logout +-rw-r--r-- 1 waldek waldek 3545 Jun 15 16:20 .bashrc +-rw-r--r-- 1 waldek waldek 826 Jun 15 16:17 .profile +drwx------ 2 waldek waldek 4096 Jun 15 16:29 .ssh +-rw------- 1 waldek waldek 3141 Jun 15 16:20 .viminfo +-rw-r--r-- 1 waldek waldek 34 Jun 15 16:18 script.sh +waldek@tester:~$ +``` + +# A clean slate + +I deleted all the content of our three files. + +``` +waldek@tester:~$ cat .bashrc .bash_logout .profile + + + +waldek@tester:~$ +``` + +We don't see any differences but our shell is *less* powerful. +For example, autocomplete does not work anymore on things like `systemctl`. +We now have a `bash` shell that has only sourced actual content from the following files. + +* `/etc/profile` +* `/etc/bash.bashrc` +* `/etc/bash.bash.logout` if it exists + +As these are system files we can only edit them as `root`. +I went a head and added a comment `#` in front of every line. +When I now log in on a tty I get the following. + +``` +Debian GNU/Linux 11 tester pts/1 + +tester login: waldek +Password: +Linux tester 5.10.0-15-amd64 #1 SMP Debian 5.10.120-1 (2022-06-09) x86_64 + +The programs included with the Debian GNU/Linux system are free software; +the exact distribution terms for each program are described in the +individual files in /usr/share/doc/*/copyright. + +Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent +permitted by applicable law. +Last login: Wed Jun 15 16:44:05 CEST 2022 on pts/1 +-bash-5.1$ +``` + +This is a fully barebones `bash` shell, no bells and whistles at all. +You can see our prompt has changed to a super simple one, no more username nor hostname nor path. +Tab complete works, but only for programs and paths, not for arguments. # Prompt customization -TODO +Form this basic session we can learn quite a bit. +The most obvious change is our prompt. +The fact that it changed means we can modify it too! +Let's discover how. + +From the manual we can conclude that `$PSX` are the variables that hold the prompt. + +``` + PS0 The value of this parameter is expanded (see PROMPTING below) and displayed by interactive shells after reading a command and before + the command is executed. + PS1 The value of this parameter is expanded (see PROMPTING below) and used as the primary prompt string. The default value is ``\s-\v\$ + ''. + PS2 The value of this parameter is expanded as with PS1 and used as the secondary prompt string. The default is ``> ''. + PS3 The value of this parameter is used as the prompt for the select command (see SHELL GRAMMAR above). + PS4 The value of this parameter is expanded as with PS1 and the value is printed before each command bash displays during an execution + trace. The first character of the expanded value of PS4 is replicated multiple times, as necessary, to indicate multiple levels of + indirection. The default is ``+ ''. +``` + +``` +-bash-5.1$ echo $PS +$PS1 $PS2 $PS4 +-bash-5.1$ echo $PS1 +\s-\v\$ +-bash-5.1$ +``` + +Your prompt is defined by the `$PS1` variable. +By changing this we can change it's behaviour. + +``` +-bash-5.1$ PS1="helloworld" +helloworldls +script.sh +helloworldcd / +helloworldls +bin boot dev etc home lib lib32 lib64 libx32 media mnt opt proc root run sbin srv sys tmp usr var +helloworld +``` + +This is *far* from useful but it's just to prove a point. +Let's try to get our username back. +Again, by reading the manual we can see all variables we can use in a prompt. +I include the **PROMPTING** section for completeness. + +``` +PROMPTING + When executing interactively, bash displays the primary prompt PS1 when it is ready to read a command, and the secondary prompt PS2 when it + needs more input to complete a command. Bash displays PS0 after it reads a command but before executing it. Bash displays PS4 as described + above before tracing each command when the -x option is enabled. Bash allows these prompt strings to be customized by inserting a number of + backslash-escaped special characters that are decoded as follows: + \a an ASCII bell character (07) + \d the date in "Weekday Month Date" format (e.g., "Tue May 26") + \D{format} + the format is passed to strftime(3) and the result is inserted into the prompt string; an empty format results in a locale- + specific time representation. The braces are required + \e an ASCII escape character (033) +\h the hostname up to the first `.' + \H the hostname + \j the number of jobs currently managed by the shell + \l the basename of the shell's terminal device name + \n newline + \r carriage return + \s the name of the shell, the basename of $0 (the portion following the final slash) + \t the current time in 24-hour HH:MM:SS format + \T the current time in 12-hour HH:MM:SS format + \@ the current time in 12-hour am/pm format + \A the current time in 24-hour HH:MM format + \u the username of the current user + \v the version of bash (e.g., 2.00) + \V the release of bash, version + patch level (e.g., 2.00.0) + \w the current working directory, with $HOME abbreviated with a tilde (uses the value of the PROMPT_DIRTRIM variable) + \W the basename of the current working directory, with $HOME abbreviated with a tilde + \! the history number of this command + \# the command number of this command + \$ if the effective UID is 0, a #, otherwise a $ + \nnn the character corresponding to the octal number nnn + \\ a backslash + \[ begin a sequence of non-printing characters, which could be used to embed a terminal control sequence into the prompt + \] end a sequence of non-printing characters + + The command number and the history number are usually different: the history number of a command is its position in the history list, which + may include commands restored from the history file (see HISTORY below), while the command number is the position in the sequence of com‐ + mands executed during the current shell session. After the string is decoded, it is expanded via parameter expansion, command substitution + or contain characters special to word expansion. +``` + +So, to get our username back we need the `\u` variable. + +``` +helloworldPS1="\u" +waldekls +bin boot dev etc home lib lib32 lib64 libx32 media mnt opt proc root run sbin srv sys tmp usr var +waldek +``` + +That *glue* between our username and command is rather annoying, let's modify that a bit. + +``` +waldekPS1="\u -> " +waldek -> ls +bin boot dev etc home lib lib32 lib64 libx32 media mnt opt proc root run sbin srv sys tmp usr var +waldek -> +``` + +Ah, much better! +Now, what time is it? + +``` +waldek -> PS1="\u \t -> " +waldek 17:02:53 -> ls +bin boot dev etc home lib lib32 lib64 libx32 media mnt opt proc root run sbin srv sys tmp usr var +waldek 17:02:56 -> +``` + +And where am I? + +``` +waldek@tester 17:04:10 -> PS1="\u@\H \t -> " +waldek@tester 17:04:19 -> ls +bin boot dev etc home lib lib32 lib64 libx32 media mnt opt proc root run sbin srv sys tmp usr var +waldek@tester 17:04:20 -> +``` + +And in which directory? + +``` +waldek@tester 17:04:20 -> PS1="\u@\H:\w \t -> " +waldek@tester:/ 17:05:06 -> cd /var/log/ +waldek@tester:/var/log 17:05:13 -> +``` + +What about the *other* `$PS` variables? +Well, if we want to know the exact time when we launch a command we can modify the `$PS0` variable. + +``` +waldek@tester:/var/log -> PS0="it is \t \n" +waldek@tester:/var/log -> ls +it is 17:06:48 +alternatives.log auth.log btmp dpkg.log journal messages runit user.log +apt bootstrap.log daemon.log faillog lastlog private syslog wtmp +waldek@tester:/var/log -> cd +it is 17:06:58 +waldek@tester:~ -> +``` + +## Decoding the *base* prompt + +On a modern install your `$PS1` will probably be as one of the two below. +They are actually the *same*, just one has **colors** in it, the other one not! +Customizing colors of a prompt is a bit of a nightmare but well worth the exercise! + +* `PS1='\u@\h:\w\$ '` +* `PS1='\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '']]]]'` # Alternative shells