First of all, this will be a bit painful but as with `vim` once you overcome the initial learning curve you start to see the potential regular expressions bring to the table.
To make matters even worse, there are multiple *flavors* of regexes.
An overview and comparison between different flavors can be found on [wikipedia](https://en.wikipedia.org/wiki/Comparison_of_regular-expression_engines).
Don't see this as a reason *not* to learn some basic expressions though, a little experience goes a long way.
## What are they?
> A regular expression (shortened as regex or regexp;[1] also referred to as rational expression[2][3]) is a sequence of characters that specifies a search pattern. Usually such patterns are used by string-searching algorithms for "find" or "find and replace" operations on strings, or for input validation. It is a technique developed in theoretical computer science and formal language theory.
You can see regular expressions as find (and replace) on steroids.
As a practical example, I used *a lot* of regular expressions to clean up the multiple choice LPI questionnaires.
This was done in `vim` so I used the vim flavor regex but it's not too much different from the main one you should know, `grep`.
From a practical system administer point of view you'll probably use regexes in this order:
1. with `grep`
2. with `sed` (went copy pasting commands found online)
3. with `vim`
4. with a scripting language such as `python3`
## How to learn them?
Some tips and pointers before we head into the actual syntax.
### Vim
There is a setting in `vim` that is disabled by default but highly advised to learn vim regexes.
By setting `set incsearch` in your `~/.vimrc` or in the **expert** command line vim will highlight whatever matches the pattern you're searching for.
This can be a tremendous help when building complex patterns.
### Grep
By default `grep` only interprets basic regular expressions.
If you want, or more likely *need* to use [extended](https://www.gnu.org/software/grep/manual/html_node/Basic-vs-Extended.html) expressions you should use `grep -E` or `egrep` instead.
For completeness's sake I should mention there is a third *version* of `grep` invoke with `grep -P` that interprets the patterns as [perl regex](https://perldoc.perl.org/perlre).
One of the advantages of perl regexes is reverse matching.
Download the following [file](./assets/regex_03.txt) which is an American English dictionary we will use to test out some basic patterns.
Fire up a terminal and find all words that contain the string `abba`.
You already know how to do this with `grep`, nothing really special here.
`grep 'abba' regex_03.txt` should give give you a list containing 16 words.
Notice the color output in you terminal.
What if we only want the words containing `abba` but who *also* start with `s`?
This can be done with the following line `grep 's.*abba' regex_03.txt`.
You should have 6 matches left.
How about the same ones, but they have to end with an `s`?
You guessed it, `grep 's.*abba.*s' regex_03.txt`, get's the job done.
Now, what are those special characters?
They are *like* [wildcards in bash](https://ryanstutorials.net/linuxtutorial/wildcards.php) but on steroids.
In the example above, the `.` represents almost any character, and the `*` means as **many** times as we want.
They are both very powerful, and broad, matching patterns that are part of the [fundamental structure](https://www.gnu.org/software/grep/manual/html_node/Fundamental-Structure.html#Fundamental-Structure) of regex.
They work well but sometime we need to be a bit more precise, which we'll get to in a bit.
Second little example with the same list.
What if we want all words with the letter `a` in them?
Easy, `grep 'a' regex_03.txt` should do the trick but 52849 results are a bit much.
Let's trim it back to only the words having `aa` in them?
`grep 'aa' regex_03.txt` brings it down to 65 matches.
Nice.
Now, how about only the ones that *start* with `aa`?
This can be done with an other core concept of regex called [anchoring](https://www.gnu.org/software/grep/manual/html_node/Anchoring.html#Anchoring) and can be done as such `grep '^aa' regex_03.txt`.
Here the `^` signifies the **start** of the line.
Last one, I promise, what if we want of those three only the ones with `s` at the end?
Right, `grep '^aa.*s$' regex_03.txt` should do the trick where `$` means **ends** with.
### Character classes and bracket expansion
Let's find, in the following [file](./assets/regex_04.txt) all words starting with a capital case letter.
This can be done in multiple ways.
First we try one you *should* remember from your bandit days.
Does `[a-zA-Z]` ring a bell?
You used it to do ROT13 rotation somewhere around level 12.
`grep '^[A-Z].*' regex_04.txt` greps out all words starting with a capital.
Wonderful, but a bit too many words.
Let's limit it to only words that have punctuation in them.
This, `grep '^[A-Z][[:alnum:]]*[[:punct:]].*' regex_04.txt`, which is *very* cryptic does the job.
Let's break it down a bit:
*`^[A-Z]` matches all capital letters
*`[[:alnum:]]*` all letters and number, as many times as we want
*`[[:punct:]]` matches any punctuation, but just **once**
*`.*` matches almost any character as many times as we want
In the above pattern you see *two* different forms of character classes, the `[A-Z]` and the `[[:alnum:]]`.
Worth noting is that you can negate an expansion by putting a `^` inside it as such `grep '^[^A-Z].*' regex_03.txt` (note I used the first list because the animals list has no words starting with a capital letter).
This is basically the same as `grep '^[a-z].*' regex_03.txt`.
### The backslash character and special expressions
If we want to match only animals that contain multiple words we can use [special expressions](https://www.gnu.org/software/grep/manual/html_node/The-Backslash-Character-and-Special-Expressions.html#The-Backslash-Character-and-Special-Expressions).
Have a look at the output of this grep pattern `grep '\s[[:alpha:]]*' regex_04.txt`.
Here we notice we can select only parts of the lines too!
A handy flag we can add to the `grep` command is `-o` which will only print the part that matches (in red).
If we only want single words coming out of the search we can do this like so `grep -o '\s[a-zA-Z]*\>' regex_04.txt`.
Notice how it starts with an empty space?
let's break this one down.
*`\s` matches whitespace
*`[a-zA-Z]*` matches the entire alphabet as many times as we want
*`\>` matches the end of the word (not really needed here)
You can fine very good cheat sheets online, like [this](https://cheatography.com/davechild/cheat-sheets/regular-expressions/) one, so have a look around.
