diff --git a/assets/io_01.dia b/assets/io_01.dia new file mode 100644 index 0000000..ad5652e Binary files /dev/null and b/assets/io_01.dia differ diff --git a/assets/io_01.png b/assets/io_01.png new file mode 100644 index 0000000..e0b3fc9 Binary files /dev/null and b/assets/io_01.png differ diff --git a/assets/io_02.dia b/assets/io_02.dia new file mode 100644 index 0000000..6b408da Binary files /dev/null and b/assets/io_02.dia differ diff --git a/assets/io_02.png b/assets/io_02.png new file mode 100644 index 0000000..85ccf99 Binary files /dev/null and b/assets/io_02.png differ diff --git a/assets/io_03.dia b/assets/io_03.dia new file mode 100644 index 0000000..ec2d106 Binary files /dev/null and b/assets/io_03.dia differ diff --git a/assets/io_03.png b/assets/io_03.png new file mode 100644 index 0000000..5af8868 Binary files /dev/null and b/assets/io_03.png differ diff --git a/assets/io_04.dia b/assets/io_04.dia new file mode 100644 index 0000000..b8ff8fe Binary files /dev/null and b/assets/io_04.dia differ diff --git a/assets/io_04.png b/assets/io_04.png new file mode 100644 index 0000000..3bbb5e0 Binary files /dev/null and b/assets/io_04.png differ diff --git a/essential/introduction_to_the_commandline.md b/essential/introduction_to_the_commandline.md index c484528..53ff2f2 100644 --- a/essential/introduction_to_the_commandline.md +++ b/essential/introduction_to_the_commandline.md @@ -1137,7 +1137,7 @@ For the eager ones: * find all files in `/usr` that contain the word *hippop* or *hiphop* * find all Belgian email addresses in `/usr` -## Searching inside files +# Searching inside files `locate` and `find` are used to search **for** files. To search for content **inside** files we can use an other very powerful tool called `grep`. @@ -1229,7 +1229,7 @@ whippoorwills waldek@debian:~$ ``` -Adding the `--color` argument to `grep will make the matched patterns jump out with a color, depending on the color scheme of your terminal. +Adding the `--color` argument to `grep` will make the matched patterns jump out with a color, depending on the color scheme of your terminal. ## Wildcards and regular expressions @@ -1284,4 +1284,304 @@ Can you count the occurences? * [exercise from Linux long](https://gitea.86thumbs.net/waldek/linux_course_doc/src/branch/master/modules/qualifying/learning_regex.md) + # Pipes and redirects + +TODO basic overview + +## Redirecting + +![IO bash](../assets/io_01.png) + +| name | stream ID | description | +| --- | --- | --- | +| STDIN | 0 | standard input, by default the keyboard | +| STDOUT | 1 | standard output, by default your terminal | +| STDERR | 2 | standard error, by default your terminal | + +We can however **redirect** the input and output's! +This is done with the `>` character. +An example... + +``` +waldek@debian:~$ ls -l +total 0 +waldek@debian:~$ date +Tue 10 May 2022 12:23:13 PM CEST +waldek@debian:~$ date > date_now.txt +waldek@debian:~$ ls -l +total 4 +-rw-r--r-- 1 waldek waldek 33 May 10 12:23 date_now.txt +waldek@debian:~$ cat date_now.txt +Tue 10 May 2022 12:23:25 PM CEST +waldek@debian:~$ +``` + +![IO bash](../assets/io_02.png) + +It's very important to realize that the file you redirect too will be overwritten! +So if you run the above `date > date_now.txt` again, the file will show a more recent date. +We can also [append](https://en.wikipedia.org/wiki/Append) to an existing file by ising the `>>`. +This redirection will add the date to the end of the file if the file exists, otherwise it will just create one. + +``` +waldek@debian:~$ date >> date_now.txt +waldek@debian:~$ date >> date_now.txt +waldek@debian:~$ date >> date_now.txt +waldek@debian:~$ cat date_now.txt +Tue 10 May 2022 12:23:25 PM CEST +Tue 10 May 2022 12:27:23 PM CEST +Tue 10 May 2022 12:27:24 PM CEST +Tue 10 May 2022 12:27:25 PM CEST +waldek@debian:~$ +``` + +We can also choose which of the two outputs, STDOUT or STDERR, we want to redirect. +This is done by adding the **stream ID** to the redirection. +Observe the following output where I search for my username in each file located at `/etc`. + +``` +waldek@debian:~$ grep -R waldek /etc/ +/etc/group-:cdrom:x:24:waldek +/etc/group-:floppy:x:25:waldek +/etc/group-:sudo:x:27:waldek +/etc/group-:audio:x:29:waldek +/etc/group-:dip:x:30:waldek +/etc/group-:video:x:44:waldek +/etc/group-:plugdev:x:46:waldek +/etc/group-:netdev:x:108:waldek +/etc/group-:waldek:x:1000: +grep: /etc/sv/ssh/log/supervise: No such file or directory +grep: /etc/sv/ssh/supervise: No such file or directory +/etc/passwd-:waldek:x:1000:1000:waldek,,,:/home/waldek:/bin/bash +grep: /etc/gshadow: Permission denied +/etc/subuid:waldek:100000:65536 +grep: /etc/security/opasswd: Permission denied +grep: /etc/gshadow-: Permission denied +grep: /etc/sudoers.d/README: Permission denied +grep: /etc/.pwd.lock: Permission denied +grep: /etc/runit/runsvdir/default/ssh/log/supervise: No such file or directory +grep: /etc/runit/runsvdir/default/ssh/supervise: No such file or directory +grep: /etc/ssh/ssh_host_ecdsa_key: Permission denied +grep: /etc/ssh/ssh_host_ed25519_key: Permission denied +grep: /etc/ssh/ssh_host_rsa_key: Permission denied +grep: /etc/shadow-: Permission denied +grep: /etc/ssl/private: Permission denied +/etc/group:cdrom:x:24:waldek +/etc/group:floppy:x:25:waldek +/etc/group:sudo:x:27:waldek +/etc/group:audio:x:29:waldek +/etc/group:dip:x:30:waldek +/etc/group:video:x:44:waldek +/etc/group:plugdev:x:46:waldek +/etc/group:netdev:x:108:waldek +/etc/group:waldek:x:1000: +grep: /etc/shadow: Permission denied +grep: /etc/sudoers: Permission denied +/etc/subgid:waldek:100000:65536 +/etc/passwd:waldek:x:1000:1000:waldek,,,:/home/waldek:/bin/bash +waldek@debian:~$ +``` + +We can see quite a few `Permission denied` lines which are errors. +Because STDERR is connected to our terminal by default, we also see them printed on the screen. +Let's play around with the above command a bit. +First a simple redirection. + +``` +waldek@debian:~$ grep -R waldek /etc/ > files_with_my_name +grep: /etc/sv/ssh/log/supervise: No such file or directory +grep: /etc/sv/ssh/supervise: No such file or directory +grep: /etc/gshadow: Permission denied +grep: /etc/security/opasswd: Permission denied +grep: /etc/gshadow-: Permission denied +grep: /etc/sudoers.d/README: Permission denied +grep: /etc/.pwd.lock: Permission denied +grep: /etc/runit/runsvdir/default/ssh/log/supervise: No such file or directory +grep: /etc/runit/runsvdir/default/ssh/supervise: No such file or directory +grep: /etc/ssh/ssh_host_ecdsa_key: Permission denied +grep: /etc/ssh/ssh_host_ed25519_key: Permission denied +grep: /etc/ssh/ssh_host_rsa_key: Permission denied +grep: /etc/shadow-: Permission denied +grep: /etc/ssl/private: Permission denied +grep: /etc/shadow: Permission denied +grep: /etc/sudoers: Permission denied +``` + +We only see the errors now! +Where did our actual output go? + +``` +waldek@debian:~$ ls -l +total 4 +-rw-r--r-- 1 waldek waldek 722 May 10 12:34 files_with_my_name +waldek@debian:~$ cat files_with_my_name +/etc/group-:cdrom:x:24:waldek +/etc/group-:floppy:x:25:waldek +/etc/group-:sudo:x:27:waldek +/etc/group-:audio:x:29:waldek +/etc/group-:dip:x:30:waldek +/etc/group-:video:x:44:waldek +/etc/group-:plugdev:x:46:waldek +/etc/group-:netdev:x:108:waldek +/etc/group-:waldek:x:1000: +/etc/passwd-:waldek:x:1000:1000:waldek,,,:/home/waldek:/bin/bash +/etc/subuid:waldek:100000:65536 +/etc/group:cdrom:x:24:waldek +/etc/group:floppy:x:25:waldek +/etc/group:sudo:x:27:waldek +/etc/group:audio:x:29:waldek +/etc/group:dip:x:30:waldek +/etc/group:video:x:44:waldek +/etc/group:plugdev:x:46:waldek +/etc/group:netdev:x:108:waldek +/etc/group:waldek:x:1000: +/etc/subgid:waldek:100000:65536 +/etc/passwd:waldek:x:1000:1000:waldek,,,:/home/waldek:/bin/bash +waldek@debian:~$ +``` + +If we want to do the *reverse* we can redirect the STDERR by using `2>`. +An example... + +``` +waldek@debian:~$ grep -R waldek /etc/ 2> files_with_my_name +/etc/group-:cdrom:x:24:waldek +/etc/group-:floppy:x:25:waldek +/etc/group-:sudo:x:27:waldek +/etc/group-:audio:x:29:waldek +/etc/group-:dip:x:30:waldek +/etc/group-:video:x:44:waldek +/etc/group-:plugdev:x:46:waldek +/etc/group-:netdev:x:108:waldek +/etc/group-:waldek:x:1000: +/etc/passwd-:waldek:x:1000:1000:waldek,,,:/home/waldek:/bin/bash +/etc/subuid:waldek:100000:65536 +/etc/group:cdrom:x:24:waldek +/etc/group:floppy:x:25:waldek +/etc/group:sudo:x:27:waldek +/etc/group:audio:x:29:waldek +/etc/group:dip:x:30:waldek +/etc/group:video:x:44:waldek +/etc/group:plugdev:x:46:waldek +/etc/group:netdev:x:108:waldek +/etc/group:waldek:x:1000: +/etc/subgid:waldek:100000:65536 +/etc/passwd:waldek:x:1000:1000:waldek,,,:/home/waldek:/bin/bash +waldek@debian:~$ +``` + +And the file content. + +``` +waldek@debian:~$ cat files_with_my_name +grep: /etc/sv/ssh/log/supervise: No such file or directory +grep: /etc/sv/ssh/supervise: No such file or directory +grep: /etc/gshadow: Permission denied +grep: /etc/security/opasswd: Permission denied +grep: /etc/gshadow-: Permission denied +grep: /etc/sudoers.d/README: Permission denied +grep: /etc/.pwd.lock: Permission denied +grep: /etc/runit/runsvdir/default/ssh/log/supervise: No such file or directory +grep: /etc/runit/runsvdir/default/ssh/supervise: No such file or directory +grep: /etc/ssh/ssh_host_ecdsa_key: Permission denied +grep: /etc/ssh/ssh_host_ed25519_key: Permission denied +grep: /etc/ssh/ssh_host_rsa_key: Permission denied +grep: /etc/shadow-: Permission denied +grep: /etc/ssl/private: Permission denied +grep: /etc/shadow: Permission denied +grep: /etc/sudoers: Permission denied +waldek@debian:~$ +``` + +Now the errors are not *that* interesting too us so we can send this output to a *black hole* or *trash* location. +This is done by sending STDERR to `/dev/null`. +You'll see this redirection in a lot of places! +Try it out when you're searching for files with `find` and set the start of the search at the root of your system `/`. + +``` +waldek@debian:~$ grep -R waldek /etc/ 2> /dev/null +/etc/group-:cdrom:x:24:waldek +/etc/group-:floppy:x:25:waldek +/etc/group-:sudo:x:27:waldek +/etc/group-:audio:x:29:waldek +/etc/group-:dip:x:30:waldek +/etc/group-:video:x:44:waldek +/etc/group-:plugdev:x:46:waldek +/etc/group-:netdev:x:108:waldek +/etc/group-:waldek:x:1000: +/etc/passwd-:waldek:x:1000:1000:waldek,,,:/home/waldek:/bin/bash +/etc/subuid:waldek:100000:65536 +/etc/group:cdrom:x:24:waldek +/etc/group:floppy:x:25:waldek +/etc/group:sudo:x:27:waldek +/etc/group:audio:x:29:waldek +/etc/group:dip:x:30:waldek +/etc/group:video:x:44:waldek +/etc/group:plugdev:x:46:waldek +/etc/group:netdev:x:108:waldek +/etc/group:waldek:x:1000: +/etc/subgid:waldek:100000:65536 +/etc/passwd:waldek:x:1000:1000:waldek,,,:/home/waldek:/bin/bash +waldek@debian:~$ cat /dev/null +waldek@debian:~$ +``` + +If you want to redirect **both** outputs to a file you can use the `&>` syntax. +The file will then contain **everything** that would be printed out on the terminal. + +``` +waldek@debian:~$ grep -R waldek /etc/ &> files_with_my_name +waldek@debian:~$ +``` + +TODO - input redirection + +## Piping + +A *very* powerful fundamental concept of the command line is are **pipes** which are created with the `|` character. +Their purpose is to send the STDOUT of one process to the STDIN of a following. +They might seem similar to redirection but they are fundamentally different. +A redirect sends **to** or **from** a **file** while a pipe sends to and from a **process**. + +![IO bash](../assets/io_03.png) + +As always, an example should clarify it better. + +``` +waldek@debian:~$ ls -l /etc/* | grep "d.conf" +-rw-r--r-- 1 root root 6169 Feb 27 2021 /etc/sudo_logsrvd.conf +-rw-r--r-- 1 root root 3289 Mar 13 2021 sshd_config +drwxr-xr-x 2 root root 4096 Mar 13 2021 sshd_config.d +-rw-r--r-- 1 root root 1052 Mar 20 20:55 journald.conf +-rw-r--r-- 1 root root 1145 Mar 20 20:55 logind.conf +-rw-r--r-- 1 root root 609 Feb 2 2021 networkd.conf +-rw-r--r-- 1 root root 943 Mar 20 20:55 resolved.conf +-rw-r--r-- 1 root root 677 Mar 20 20:55 timesyncd.conf +waldek@debian:~$ +``` + +In the example above we send the output of `ls -l /etc/*` to `grep "d.conf"`. +We go from *a lot* of output to *less*. +Pipes are a very powerful tool to do text manipulations! + +# Text manipulation + +| command | description | +| --- | --- | +| cut | remove sections from each line of files | +| sort | sort lines of text files | +| uniq | report or omit repeated lines | +| grep | print lines that match patterns | +| tr | translate or delete characters | +| sed | stream editor for filtering and transforming text | +| nl | number lines of files | +| wc | print newline, word, and byte counts for each file | +| seq | print a sequence of numbers | +| diff | compare files | +| cat | concatenate files and print on the standard output | +| tac | concatenate and print files in reverse | +| comm | compare two sorted files line by line | +| shuf | generate random permutations | + + diff --git a/essential/solutions.md b/essential/solutions.md new file mode 100644 index 0000000..668325a --- /dev/null +++ b/essential/solutions.md @@ -0,0 +1,2 @@ +`find / -iname "*ssh*" -size -10c 2> /dev/null` +`grep -R -n "hip.op" /usr/ 2> /dev/null`