# Scripting ## Bash A `bash` script is a sequence of command that are executed one by one. Most of the time we just execute one command and wait for the result to then make a decision and execute an other command. We can however create a sequence of commands on the command line. ``` (baseline-) ➜ ~ echo hello world hello world (baseline-) ➜ ~ date Wed 16 Mar 2022 07:06:02 PM CET (baseline-) ➜ ~ cal March 2022 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 (baseline-) ➜ ~ echo hello world date cal hello world date cal (baseline-) ➜ ~ echo hello world; date; cal hello world Wed 16 Mar 2022 07:06:19 PM CET March 2022 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 ``` A very visually similar result, but completely different operation, can be obtained by replacing the `;` with `&` or `&&`. `&` will launch a new process and send it to the background, `&&` will **evaluate** the return status of your process and only continue **if** the status was `0`, or in other words successful. ``` (baseline-) ➜ ~ echo hello world & date & cal [1] 3075524 hello world [2] 3075525 [1] - 3075524 done echo hello world Wed 16 Mar 2022 07:06:33 PM CET [2] + 3075525 done date March 2022 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 (baseline-) ➜ ~ echo hello world && date && cal hello world Wed 16 Mar 2022 07:06:40 PM CET March 2022 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 (baseline-) ➜ ~ ``` This should make you think we can create quite complicated logical flows in `bash` and you're right! But first things first, let's create our first script! ``` (baseline-) ➜ bash file test.sh test.sh: Bourne-Again shell script, ASCII text executable (baseline-) ➜ bash ls -l test.sh -rwxr-xr-x 1 waldek waldek 39 Mar 16 19:13 test.sh (baseline-) ➜ bash cat test.sh #!/bin/bash echo hello world cal date (baseline-) ➜ bash ./test.sh hello world March 2022 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Wed 16 Mar 2022 07:27:22 PM CET (baseline-) ➜ bash bash test.sh hello world March 2022 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Wed 16 Mar 2022 07:27:25 PM CET (baseline-) ➜ bash ``` If you observe the output above you can conclude multiple things. * the `test.sh` file is a simple text file * it has executable permissions for the owner, group and others * we can execute the sequence of commands in the script in two ways * `./test.sh` * `bash test.sh` * the content is a series of commands but with a *weird* first line That first line is called a [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) and is a way of explaining which interpreter understands the lines that follow. If you venture out into the python universe you'll encounter the same norm but with a different path to an interpretor, often `/bin/python3` or `/usr/bin/env python3`. The later is a sort of shortcut that points to the local python installation, even if it's not in a standard location. A shebang is not necessary for a script to function but is **highly** advised. The `env` program is actually very interesting! Let's try it by itself. ``` ➜ ~ env PAGER=less LANGUAGE=en_US:en GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/022a5a78_fe95_445c_9b33_f6dcb35cfd27 LANG=en_US.UTF-8 DISPLAY=:0 SWAYSOCK=/run/user/1000/sway-ipc.1000.1565.sock WAYLAND_DISPLAY=wayland-0 AUTOSWITCH_VERSION=3.4.0 HUSHLOGIN=FALSE USER=waldek OLDPWD=/home/waldek/bin/bash HOME=/home/waldek MOZ_ENABLE_WAYLAND=1 VIRTUAL_ENV=/home/waldek/.virtualenvs/baseline- DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-acMe3wyyoo,guid=ad1233c6039551be866f78ae61f911b2 XDG_VTNR=1 XDG_SEAT=seat0 I3SOCK=/run/user/1000/sway-ipc.1000.1565.sock LESS=-R _=/usr/bin/env LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36: AUTOSWITCH_DEFAULTENV=baseline VTE_VERSION=6203 LSCOLORS=Gxfxcxdxbxegedabagacad ZSH=/home/waldek/.oh-my-zsh AUTOSWITCH_FILE=.venv MAIL=/var/mail/waldek LOGNAME=waldek PS1=(baseline-) %(?:%{%}➜ :%{%}➜ ) %{$fg[cyan]%}%c%{$reset_color%} $(git_prompt_info) GDK_BACKEND=wayland PATH=/home/waldek/.virtualenvs/baseline-/bin:/home/waldek/.cargo/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/waldek/.local/bin:/home/waldek/bin/python/:/home/waldek/.local/bin:/home/waldek/bin/python/ XDG_RUNTIME_DIR=/run/user/1000 QT_QPA_PLATFORM=wayland XDG_SESSION_ID=2 XDG_SESSION_TYPE=wayland FPATH=/home/waldek/.oh-my-zsh/functions:/home/waldek/.oh-my-zsh/completions:/home/waldek/.oh-my-zsh/cache/completions:/home/waldek/.oh-my-zsh/custom/plugins/autoswitch_virtualenv:/home/waldek/.oh-my-zsh/plugins/git:/home/waldek/.oh-my-zsh/functions:/home/waldek/.oh-my-zsh/completions:/home/waldek/.oh-my-zsh/cache/completions:/usr/local/share/zsh/site-functions:/usr/share/zsh/vendor-functions:/usr/share/zsh/vendor-completions:/usr/share/zsh/functions/Calendar:/usr/share/zsh/functions/Chpwd:/usr/share/zsh/functions/Completion:/usr/share/zsh/functions/Completion/AIX:/usr/share/zsh/functions/Completion/BSD:/usr/share/zsh/functions/Completion/Base:/usr/share/zsh/functions/Completion/Cygwin:/usr/share/zsh/functions/Completion/Darwin:/usr/share/zsh/functions/Completion/Debian:/usr/share/zsh/functions/Completion/Linux:/usr/share/zsh/functions/Completion/Mandriva:/usr/share/zsh/functions/Completion/Redhat:/usr/share/zsh/functions/Completion/Solaris:/usr/share/zsh/functions/Completion/Unix:/usr/share/zsh/functions/Completion/X:/usr/share/zsh/functions/Completion/Zsh:/usr/share/zsh/functions/Completion/openSUSE:/usr/share/zsh/functions/Exceptions:/usr/share/zsh/functions/MIME:/usr/share/zsh/functions/Math:/usr/share/zsh/functions/Misc:/usr/share/zsh/functions/Newuser:/usr/share/zsh/functions/Prompts:/usr/share/zsh/functions/TCP:/usr/share/zsh/functions/VCS_Info:/usr/share/zsh/functions/VCS_Info/Backends:/usr/share/zsh/functions/Zftp:/usr/share/zsh/functions/Zle:/usr/share/zsh:/usr/share/zsh SHELL=/usr/bin/zsh AUTOSWITCH_SILENT=1 EDITOR=vim MOTD_SHOWN=pam SHLVL=2 GNOME_TERMINAL_SERVICE=:1.10 COLORTERM=truecolor XCURSOR_SIZE=24 TERM=xterm-256color PWD=/home/waldek XDG_SESSION_CLASS=user ➜ ~ ``` What is all of this? Well, the are the current **environment variables** defined in your shell. You'll probably recognize some of them. Let's play around with what we have defined. ``` ➜ ~ echo SHELL SHELL ➜ ~ echo $SHELL /usr/bin/zsh ➜ ~ echo $USER waldek ➜ ~ echo $HOME /home/waldek ➜ ~ echo $ zsh: do you wish to see all 223 possibilities (56 lines)? ``` Yes, tab complete works! Your variables will be slightly different but that's not an issue at all. ``` ➜ ~ echo $ ! EUID MAILPATH SWAYSOCK # fg manpath TERM * FG MANPATH termcap - fg_bold module_path terminfo ? fg_no_bold MODULE_PATH TIMEFMT @ fignore modules TMPPREFIX _ FIGNORE MOTD_SHOWN TRY_BLOCK_ERROR $ fpath MOZ_ENABLE_WAYLAND TRY_BLOCK_INTERRUPT 0 FPATH nameddirs TTY aliases funcfiletrace NULLCMD TTYIDLE ARGC FUNCNEST OLDPWD UID argv funcsourcetrace OPTARG USER AUTOSWITCH_DEFAULTENV funcstack OPTIND userdirs AUTOSWITCH_FILE functions options usergroups AUTOSWITCH_SILENT functions_source OSTYPE USERNAME AUTOSWITCH_VERSION functrace PAGER VENDOR bg FX parameters VIRTUAL_ENV BG galiases patchars VTE_VERSION bg_bold GDK_BACKEND _patcomps watch bg_no_bold GID path WATCH bold_color GNOME_TERMINAL_SCREEN PATH WATCHFMT builtins GNOME_TERMINAL_SERVICE pipestatus WAYLAND_DISPLAY cdpath histchars plugins widgets CDPATH HISTCHARS _postpatcomps WORDCHARS color HISTCMD PPID XCURSOR_SIZE COLORTERM HISTFILE precmd_functions XDG_RUNTIME_DIR colour history preexec_functions XDG_SEAT COLUMNS historywords prompt XDG_SESSION_CLASS commands HISTSIZE PROMPT XDG_SESSION_ID _compautos HOME PROMPT2 XDG_SESSION_TYPE _comp_dumpfile HOST PROMPT3 XDG_VTNR COMPLETION_WAITING_DOTS HUSHLOGIN PROMPT4 zle_bracketed_paste _comp_options I3SOCK PS1 ZLS_COLORS comppostfuncs jobdirs PS2 ZSH compprefuncs jobstates PS3 ZSH_ARGZERO _comps jobtexts PS4 ZSH_CACHE_DIR _comp_setup key psvar ZSH_COMPDUMP CPUTYPE KEYBOARD_HACK PSVAR ZSH_CUSTOM d keymaps PWD zsh_eval_context DBUS_SESSION_BUS_ADDRESS KEYTIMEOUT QT_QPA_PLATFORM ZSH_EVAL_CONTEXT debian_missing_features LANG RANDOM ZSH_NAME dirstack langinfo READNULLCMD ZSH_PATCHLEVEL dis_aliases LANGUAGE reset_color zsh_scheduled_events dis_builtins _lastcomp reswords ZSH_SUBSHELL dis_functions LESS saliases ZSH_THEME dis_functions_source LINENO __savecursor ZSH_THEME_GIT_PROMPT_CLEAN dis_galiases LINES SAVEHIST ZSH_THEME_GIT_PROMPT_DIRTY dis_patchars LISTMAX SCREEN_NO ZSH_THEME_GIT_PROMPT_PREFIX DISPLAY LOGCHECK __searching ZSH_THEME_GIT_PROMPT_SUFFIX dis_reswords LOGNAME _services ZSH_THEME_RUBY_PROMPT_PREFIX dis_saliases LS_COLORS SHELL ZSH_THEME_RUBY_PROMPT_SUFFIX EDITOR LSCOLORS SHLVL ZSH_THEME_RVM_PROMPT_OPTIONS EGID MACHTYPE SHORT_HOST ZSH_THEME_TERM_TAB_TITLE_IDLE EPOCHREALTIME MAIL signals ZSH_THEME_TERM_TITLE_IDLE EPOCHSECONDS MAILCHECK SPROMPT ZSH_VERSION epochtime mailpath status ➜ ~ echo $ ``` We can create our own variables as follows. Notice how an undefined variable does not throw an error. This is very typical for shell scripting, python on the other hand would crash over an undefined variable. ``` waldek@helloworld:~$ echo $USER waldek waldek@helloworld:~$ echo $name waldek@helloworld:~$ name="wouter gordts" waldek@helloworld:~$ echo $name wouter gordts waldek@helloworld:~$ ``` If you open up a new shell this `$name` variable will not be defined because variables are local to each instance of `bash` that is running. This can be observed as follows. ``` waldek@helloworld:~$ name="wouter gordts" waldek@helloworld:~$ echo $name wouter gordts waldek@helloworld:~$ bash waldek@helloworld:~$ echo $name waldek@helloworld:~$ exit exit waldek@helloworld:~$ export name waldek@helloworld:~$ bash waldek@helloworld:~$ echo $name wouter gordts waldek@helloworld:~$ exit exit waldek@helloworld:~$ ``` ### Using variables to store the output of command Bash only really knows *characters*, both for sending and receiving. We can temporarily store the output of a command using variables. The syntax is a bit tricky at first but quickly becomes quite natural. We can try this out on the command line. Next we'll write a small script to leverage the power of variables and pipes. ``` waldek@metal:~$ grep $USER /etc/passwd waldek:x:1000:1000:waldek,,,:/home/local/waldek:/bin/zsh waldek@metal:~$ my_name=$(grep $USER /etc/passwd) waldek@metal:~$ echo $my_name waldek:x:1000:1000:waldek,,,:/home/local/waldek:/bin/zsh waldek@metal:~$ ``` A little but more complicated. ``` waldek@metal:~$ count=$(grep "/home" /etc/passwd | wc -l) waldek@metal:~$ msg="there are $count users on this machine" waldek@metal:~$ echo $msg there are 4 users on this machine waldek@metal:~$ ``` Now a small script. ```bash #!/bin/bash FULLNAME="wouter gordts" CITY="Brussels" echo "this script was written by $FULLNAME in $CITY" IP=$(ip a | grep -v "127.0.0.1" | grep -o -E "[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\/[[:digit:]]{1,3}") TIME=$(date +%X) DAY=$(date +%A) YEAR=$(date +%Y) echo "this computer has $IP as IP address" echo "it is $TIME and we are a $DAY in $YEAR" ``` Which if we run it gives us the following output. ``` waldek@metal:~$ bash test.sh this script was written by wouter gordts in Brussels this computer has 10.1.12.53/24 as IP address it is 02:17:02 PM and we are a Tuesday in 2022 waldek@metal:~$ ``` ### Getting input into the script #### With `read` Observe the output of the following *program*. It's not really complicated but it will demonstrate we can do arithmetic in bash scripts as well! ``` waldek@metal:~$ bash test.sh In which year where you born? 1986 your are probably around 36... waldek@metal:~$ ``` The output above was generated with the following code. The **two** things to notice are the `read year` line and the `$(( $this_year - $year ))`. The former offers the possibility to **prompt** the user for input, the latter performs a mathematical calculation with two numbers. ```bash #!/bin/bash echo "In which year where you born?" read year this_year=$(date +%Y) echo "your are probably around $(( $this_year - $year ))..." waldek@metal:~$ ``` #### With command line arguments We can create a similar behaviour but with command line arguments. By doing so we don't have to answer any questions the script poses at runtime. If we create a script that will run for a long time, it doesn't require any interaction mid way! All necessary information is supplied by at execution time. ``` waldek@metal:~$ bash test.sh 1986 your are probably around 36... waldek@metal:~$ ``` ```bash #!/bin/bash year=$1 this_year=$(date +%Y) echo "your are probably around $(( $this_year - $year ))..." ``` The magic behind is the `$1` variable. This variable represents the *first* argument on the command line. Knowing this, what would `$4` mean? Indeed, the *fourth* argument... # Coding Challenge - output the exact output below ``` waldek@metal:~$ bash test.sh hello world 1986 35 foo bar linux rulez... hello waldek, my name is test.sh you supplied 8 arguments on the command line... here are all of them on one line: hello world 1986 35 foo bar linux rulez... waldek@metal:~$ ```
Spoiler warning! ```bash #!/bin/bash echo "hello $USER, my name is $0" echo "you supplied $# arguments on the command line..." echo "here are all of them on one line: $@" ```
### What is a Bash Script - What are they, how do they work and how to run them. [Ryan's tutorials](https://ryanstutorials.net/bash-scripting-tutorial/bash-script.php) ### Variables - Store data temporarily for later use. [Ryan's tutorials](https://ryanstutorials.net/bash-scripting-tutorial/bash-variables.php) Write a program that prints information about your computer such as: * the hostname * the FQDN * number of cpus * type of cpu * amount of RAM Write a program that prints information about a given user such as: * name * UID * their default shell * groups they are a member of * number of files in their home directory * amount of disk space they use ### Input - Different ways to supply data and directions to your Bash script. [Ryan's tutorials](https://ryanstutorials.net/bash-scripting-tutorial/bash-input.php) ### Arithmetic - Perform various arithmetic operations in your Bash script. [Ryan's tutorials](https://ryanstutorials.net/bash-scripting-tutorial/bash-arithmitic.php) ### If Statements - How to make decisions within your Bash script. [Ryan's tutorials](https://ryanstutorials.net/bash-scripting-tutorial/bash-if-statements.php) Write a script that takes one argument which is a filepath. The program should print out what type of file this is, and if it is readable, print the first and last 5 lines. ### Loops - A variety of ways to perform repetitive tasks. [Ryan's tutorials](https://ryanstutorials.net/bash-scripting-tutorial/bash-loops.php) Write a script that sets all you cpu's to a desired governor. ### Functions - Reuse code to make life easier. [Ryan's tutorials](https://ryanstutorials.net/bash-scripting-tutorial/bash-functions.php) ### User Interface - Make your scripts user friendly. [Ryan's tutorials](https://ryanstutorials.net/bash-scripting-tutorial/bash-user-interfaces.php) ## Python [This](https://gitea.86thumbs.net/waldek/python_course_doc) repository has a twenty day course to learn python written by me. The main file you need to follow is [this](https://gitea.86thumbs.net/waldek/python_course_doc/src/branch/master/learning_python3.md) one. Some practical exercises can be found [here](https://gitea.86thumbs.net/waldek/linux_course_doc/src/branch/master/modules/qualifying/exercise_python.md) together with the needed [source files](https://gitea.86thumbs.net/waldek/linux_course_doc/src/branch/master/modules/qualifying/assets/files.tar.gz). ## Vim as an IDE I made a tutorial on the essentials of [vim customization](https://gitea.86thumbs.net/waldek/linux_course_doc/src/branch/master/modules/qualifying/learning_vim_configuration.md). My real world configuration can be found at [this](https://gitea.86thumbs.net/waldek/vimrc) repository.