continues grep and file manipulation

essential/introduction_to_the_commandline.md

@@ -1137,7 +1137,7 @@ For the eager ones:
 * find all files in `/usr` that contain the word *hippop* or *hiphop*
 * find all Belgian email addresses in `/usr`
 
-## Searching inside files
+# Searching inside files
 
 `locate` and `find` are used to search **for** files.
 To search for content **inside** files we can use an other very powerful tool called `grep`.
@@ -1229,7 +1229,7 @@ whippoorwills
 waldek@debian:~$
 ```
 
-Adding the `--color` argument to `grep will make the matched patterns jump out with a color, depending on the color scheme of your terminal.
+Adding the `--color` argument to `grep` will make the matched patterns jump out with a color, depending on the color scheme of your terminal.
 
 ## Wildcards and regular expressions
 
@@ -1284,4 +1284,304 @@ Can you count the occurences?
 
 * [exercise from Linux long](https://gitea.86thumbs.net/waldek/linux_course_doc/src/branch/master/modules/qualifying/learning_regex.md)
 
+
 # Pipes and redirects
+
+TODO basic overview
+
+## Redirecting
+
+![IO bash](../assets/io_01.png)
+
+| name | stream ID | description |
+| --- | --- | --- |
+| STDIN | 0 | standard input, by default the keyboard |
+| STDOUT | 1 | standard output, by default your terminal |
+| STDERR | 2 | standard error, by default your terminal |
+
+We can however **redirect** the input and output's!
+This is done with the `>` character.
+An example...
+
+```
+waldek@debian:~$ ls -l
+total 0
+waldek@debian:~$ date
+Tue 10 May 2022 12:23:13 PM CEST
+waldek@debian:~$ date > date_now.txt
+waldek@debian:~$ ls -l
+total 4
+-rw-r--r-- 1 waldek waldek 33 May 10 12:23 date_now.txt
+waldek@debian:~$ cat date_now.txt 
+Tue 10 May 2022 12:23:25 PM CEST
+waldek@debian:~$ 
+```
+
+![IO bash](../assets/io_02.png)
+
+It's very important to realize that the file you redirect too will be overwritten!
+So if you run the above `date > date_now.txt` again, the file will show a more recent date.
+We can also [append](https://en.wikipedia.org/wiki/Append) to an existing file by ising the `>>`.
+This redirection will add the date to the end of the file if the file exists, otherwise it will just create one.
+
+```
+waldek@debian:~$ date >> date_now.txt
+waldek@debian:~$ date >> date_now.txt
+waldek@debian:~$ date >> date_now.txt
+waldek@debian:~$ cat date_now.txt 
+Tue 10 May 2022 12:23:25 PM CEST
+Tue 10 May 2022 12:27:23 PM CEST
+Tue 10 May 2022 12:27:24 PM CEST
+Tue 10 May 2022 12:27:25 PM CEST
+waldek@debian:~$ 
+```
+
+We can also choose which of the two outputs, STDOUT or STDERR, we want to redirect.
+This is done by adding the **stream ID** to the redirection.
+Observe the following output where I search for my username in each file located at `/etc`.
+
+```
+waldek@debian:~$ grep -R waldek /etc/
+/etc/group-:cdrom:x:24:waldek
+/etc/group-:floppy:x:25:waldek
+/etc/group-:sudo:x:27:waldek
+/etc/group-:audio:x:29:waldek
+/etc/group-:dip:x:30:waldek
+/etc/group-:video:x:44:waldek
+/etc/group-:plugdev:x:46:waldek
+/etc/group-:netdev:x:108:waldek
+/etc/group-:waldek:x:1000:
+grep: /etc/sv/ssh/log/supervise: No such file or directory
+grep: /etc/sv/ssh/supervise: No such file or directory
+/etc/passwd-:waldek:x:1000:1000:waldek,,,:/home/waldek:/bin/bash
+grep: /etc/gshadow: Permission denied
+/etc/subuid:waldek:100000:65536
+grep: /etc/security/opasswd: Permission denied
+grep: /etc/gshadow-: Permission denied
+grep: /etc/sudoers.d/README: Permission denied
+grep: /etc/.pwd.lock: Permission denied
+grep: /etc/runit/runsvdir/default/ssh/log/supervise: No such file or directory
+grep: /etc/runit/runsvdir/default/ssh/supervise: No such file or directory
+grep: /etc/ssh/ssh_host_ecdsa_key: Permission denied
+grep: /etc/ssh/ssh_host_ed25519_key: Permission denied
+grep: /etc/ssh/ssh_host_rsa_key: Permission denied
+grep: /etc/shadow-: Permission denied
+grep: /etc/ssl/private: Permission denied
+/etc/group:cdrom:x:24:waldek
+/etc/group:floppy:x:25:waldek
+/etc/group:sudo:x:27:waldek
+/etc/group:audio:x:29:waldek
+/etc/group:dip:x:30:waldek
+/etc/group:video:x:44:waldek
+/etc/group:plugdev:x:46:waldek
+/etc/group:netdev:x:108:waldek
+/etc/group:waldek:x:1000:
+grep: /etc/shadow: Permission denied
+grep: /etc/sudoers: Permission denied
+/etc/subgid:waldek:100000:65536
+/etc/passwd:waldek:x:1000:1000:waldek,,,:/home/waldek:/bin/bash
+waldek@debian:~$ 
+```
+
+We can see quite a few `Permission denied` lines which are errors.
+Because STDERR is connected to our terminal by default, we also see them printed on the screen.
+Let's play around with the above command a bit.
+First a simple redirection.
+
+```
+waldek@debian:~$ grep -R waldek /etc/ > files_with_my_name
+grep: /etc/sv/ssh/log/supervise: No such file or directory
+grep: /etc/sv/ssh/supervise: No such file or directory
+grep: /etc/gshadow: Permission denied
+grep: /etc/security/opasswd: Permission denied
+grep: /etc/gshadow-: Permission denied
+grep: /etc/sudoers.d/README: Permission denied
+grep: /etc/.pwd.lock: Permission denied
+grep: /etc/runit/runsvdir/default/ssh/log/supervise: No such file or directory
+grep: /etc/runit/runsvdir/default/ssh/supervise: No such file or directory
+grep: /etc/ssh/ssh_host_ecdsa_key: Permission denied
+grep: /etc/ssh/ssh_host_ed25519_key: Permission denied
+grep: /etc/ssh/ssh_host_rsa_key: Permission denied
+grep: /etc/shadow-: Permission denied
+grep: /etc/ssl/private: Permission denied
+grep: /etc/shadow: Permission denied
+grep: /etc/sudoers: Permission denied
+```
+
+We only see the errors now!
+Where did our actual output go?
+
+```
+waldek@debian:~$ ls -l
+total 4
+-rw-r--r-- 1 waldek waldek 722 May 10 12:34 files_with_my_name
+waldek@debian:~$ cat files_with_my_name 
+/etc/group-:cdrom:x:24:waldek
+/etc/group-:floppy:x:25:waldek
+/etc/group-:sudo:x:27:waldek
+/etc/group-:audio:x:29:waldek
+/etc/group-:dip:x:30:waldek
+/etc/group-:video:x:44:waldek
+/etc/group-:plugdev:x:46:waldek
+/etc/group-:netdev:x:108:waldek
+/etc/group-:waldek:x:1000:
+/etc/passwd-:waldek:x:1000:1000:waldek,,,:/home/waldek:/bin/bash
+/etc/subuid:waldek:100000:65536
+/etc/group:cdrom:x:24:waldek
+/etc/group:floppy:x:25:waldek
+/etc/group:sudo:x:27:waldek
+/etc/group:audio:x:29:waldek
+/etc/group:dip:x:30:waldek
+/etc/group:video:x:44:waldek
+/etc/group:plugdev:x:46:waldek
+/etc/group:netdev:x:108:waldek
+/etc/group:waldek:x:1000:
+/etc/subgid:waldek:100000:65536
+/etc/passwd:waldek:x:1000:1000:waldek,,,:/home/waldek:/bin/bash
+waldek@debian:~$ 
+```
+
+If we want to do the *reverse* we can redirect the STDERR by using `2>`.
+An example...
+
+```
+waldek@debian:~$ grep -R waldek /etc/ 2> files_with_my_name
+/etc/group-:cdrom:x:24:waldek
+/etc/group-:floppy:x:25:waldek
+/etc/group-:sudo:x:27:waldek
+/etc/group-:audio:x:29:waldek
+/etc/group-:dip:x:30:waldek
+/etc/group-:video:x:44:waldek
+/etc/group-:plugdev:x:46:waldek
+/etc/group-:netdev:x:108:waldek
+/etc/group-:waldek:x:1000:
+/etc/passwd-:waldek:x:1000:1000:waldek,,,:/home/waldek:/bin/bash
+/etc/subuid:waldek:100000:65536
+/etc/group:cdrom:x:24:waldek
+/etc/group:floppy:x:25:waldek
+/etc/group:sudo:x:27:waldek
+/etc/group:audio:x:29:waldek
+/etc/group:dip:x:30:waldek
+/etc/group:video:x:44:waldek
+/etc/group:plugdev:x:46:waldek
+/etc/group:netdev:x:108:waldek
+/etc/group:waldek:x:1000:
+/etc/subgid:waldek:100000:65536
+/etc/passwd:waldek:x:1000:1000:waldek,,,:/home/waldek:/bin/bash
+waldek@debian:~$ 
+```
+
+And the file content.
+
+```
+waldek@debian:~$ cat files_with_my_name 
+grep: /etc/sv/ssh/log/supervise: No such file or directory
+grep: /etc/sv/ssh/supervise: No such file or directory
+grep: /etc/gshadow: Permission denied
+grep: /etc/security/opasswd: Permission denied
+grep: /etc/gshadow-: Permission denied
+grep: /etc/sudoers.d/README: Permission denied
+grep: /etc/.pwd.lock: Permission denied
+grep: /etc/runit/runsvdir/default/ssh/log/supervise: No such file or directory
+grep: /etc/runit/runsvdir/default/ssh/supervise: No such file or directory
+grep: /etc/ssh/ssh_host_ecdsa_key: Permission denied
+grep: /etc/ssh/ssh_host_ed25519_key: Permission denied
+grep: /etc/ssh/ssh_host_rsa_key: Permission denied
+grep: /etc/shadow-: Permission denied
+grep: /etc/ssl/private: Permission denied
+grep: /etc/shadow: Permission denied
+grep: /etc/sudoers: Permission denied
+waldek@debian:~$ 
+```
+
+Now the errors are not *that* interesting too us so we can send this output to a *black hole* or *trash* location.
+This is done by sending STDERR to `/dev/null`.
+You'll see this redirection in a lot of places!
+Try it out when you're searching for files with `find` and set the start of the search at the root of your system `/`.
+
+```
+waldek@debian:~$ grep -R waldek /etc/ 2> /dev/null 
+/etc/group-:cdrom:x:24:waldek
+/etc/group-:floppy:x:25:waldek
+/etc/group-:sudo:x:27:waldek
+/etc/group-:audio:x:29:waldek
+/etc/group-:dip:x:30:waldek
+/etc/group-:video:x:44:waldek
+/etc/group-:plugdev:x:46:waldek
+/etc/group-:netdev:x:108:waldek
+/etc/group-:waldek:x:1000:
+/etc/passwd-:waldek:x:1000:1000:waldek,,,:/home/waldek:/bin/bash
+/etc/subuid:waldek:100000:65536
+/etc/group:cdrom:x:24:waldek
+/etc/group:floppy:x:25:waldek
+/etc/group:sudo:x:27:waldek
+/etc/group:audio:x:29:waldek
+/etc/group:dip:x:30:waldek
+/etc/group:video:x:44:waldek
+/etc/group:plugdev:x:46:waldek
+/etc/group:netdev:x:108:waldek
+/etc/group:waldek:x:1000:
+/etc/subgid:waldek:100000:65536
+/etc/passwd:waldek:x:1000:1000:waldek,,,:/home/waldek:/bin/bash
+waldek@debian:~$ cat /dev/null 
+waldek@debian:~$ 
+```
+
+If you want to redirect **both** outputs to a file you can use the `&>` syntax.
+The file will then contain **everything** that would be printed out on the terminal.
+
+```
+waldek@debian:~$ grep -R waldek /etc/ &>  files_with_my_name 
+waldek@debian:~$ 
+```
+
+TODO - input redirection
+
+## Piping
+
+A *very* powerful fundamental concept of the command line is are **pipes** which are created with the `|` character.
+Their purpose is to send the STDOUT of one process to the STDIN of a following.
+They might seem similar to redirection but they are fundamentally different.
+A redirect sends **to** or **from** a **file**  while a pipe sends to and from a **process**.
+
+![IO bash](../assets/io_03.png)
+
+As always, an example should clarify it better.
+
+```
+waldek@debian:~$ ls -l /etc/* | grep "d.conf"
+-rw-r--r-- 1 root root    6169 Feb 27  2021 /etc/sudo_logsrvd.conf
+-rw-r--r-- 1 root root   3289 Mar 13  2021 sshd_config
+drwxr-xr-x 2 root root   4096 Mar 13  2021 sshd_config.d
+-rw-r--r-- 1 root root 1052 Mar 20 20:55 journald.conf
+-rw-r--r-- 1 root root 1145 Mar 20 20:55 logind.conf
+-rw-r--r-- 1 root root  609 Feb  2  2021 networkd.conf
+-rw-r--r-- 1 root root  943 Mar 20 20:55 resolved.conf
+-rw-r--r-- 1 root root  677 Mar 20 20:55 timesyncd.conf
+waldek@debian:~$
+```
+
+In the example above we send the output of `ls -l /etc/*` to `grep "d.conf"`.
+We go from *a lot* of output to *less*.
+Pipes are a very powerful tool to do text manipulations!
+
+# Text manipulation
+
+| command | description |
+| --- | --- |
+| cut | remove sections from each line of files |
+| sort | sort lines of text files |
+| uniq | report or omit repeated lines |
+| grep | print lines that match patterns |
+| tr | translate or delete characters |
+| sed | stream editor for filtering and transforming text |
+| nl | number lines of files |
+| wc | print newline, word, and byte counts for each file |
+| seq | print a sequence of numbers |
+| diff | compare files |
+| cat | concatenate files and print on the standard output |
+| tac | concatenate and print files in reverse |
+| comm | compare two sorted files line by line |
+| shuf | generate random permutations |
+
+

essential/solutions.md

@@ -0,0 +1,2 @@
+`find / -iname "*ssh*" -size -10c 2> /dev/null`
+`grep -R -n "hip.op" /usr/ 2> /dev/null`