Regular Expressions (RegEx)
Regular Expressions are a syntax for writing patterns to match for. Lot of symbols mean something allowing you to write complex rules in a very short string
Last updated
Regular Expressions are a syntax for writing patterns to match for. Lot of symbols mean something allowing you to write complex rules in a very short string
Last updated
Regular Expressions (RegEx) are a way of writing patterns that many languages understand. Almost every language has some library or way to work with regular expressions, and they are really useful for quickly finding something.
The syntax for RegEx may be hard to read at first. There is not really a way to make comments, and they are supposed to be very compact. But after working with them for a bit and understanding all the rules, you can quickly understand what a RegEx does. One great site that I always use for testing and creating Regular Expressions is RegExr:
Many languages have some library or native way to interpret Regular Expressions. Here are two examples:
To search through files or command output with these regular expressions, you can use Grep which supports advanced regular expressions using the -P
option (and use '
single quotes to avoid escaping issues).
Lots of code editors (IDEs) also allow you to search through your code using regular expressions. This can be really powerful in combination with the Replacing feature to transform a pattern in your code without doing everything by hand. You can often enable this feature by clicking a .*
button.
.
Any character except newline
\w
\d
\s
Word, digit, whitespace
\W
\D
\S
Not word, digit, whitespace
[abc]
Any of a, b, or c
[^abc]
Not a, b, or c
[a-g]
Character between a & g
^abc$
Start / end of the string
\b
\B
Word, not-word boundary
\.
\*
\\
Escaped special characters
\t
\n
\r
Tab, linefeed, carriage return
(abc)
Capture group
\1
Backreference to group #1
(?:abc)
Non-capturing group
(?=abc)
Positive lookahead
(?!abc)
Negative lookahead
a*a+a?
0 or more, 1 or more, 0 or 1
a{5}a{2,}
Exactly five, two or more
a{1,3}
Between one & three
a+?a{2,}?
Match as few as possible
ab|cd
Match ab or cd
Regular Expressions can also be used to replace matches with something. Using groups with ()
around parts of the pattern, you can even include groups back in the replacement. This is really useful for changing specific things around or in your pattern, without doing it manually. Here are the variables you can use in the replacement string:
$&
: Full match
$1
: First group
$2
: Second group
etc.
You can use the $&
anywhere in your replacement string to insert the full match. This is useful if you want to add some characters around the match, instead of changing it. You can also get any groups with $n
, where n
is the number of the group in your search pattern.
Regular Expressions are often used in two possible ways: finding a string embedded in some text, or matching the whole text against some pattern. It can be easy to mistakenly match anywhere in the text instead of the whole text, which is what this common bypass is about. The ^
and $
characters "anchor" the pattern to the start or end of the string, see some examples below:
As you can imagine, it's easy to forget the ^
and $
characters. If you do, any string will partially match. That causes the center of the string to be validated, but the surrounding text can still contain any special characters. Below is an example that attempts to validate a URL:
The above can be exploited with a javascript:
URL, while it may seem the https:
or http:
protocol is required, it may be anywhere in the string:
This will trigger Cross-Site Scripting (XSS) and passes the regex using the valid URL in the comment after it.
Even with correctly adding ^
and $
around the whole RegEx, some specific syntax using |
may still cause it to be vulnerable. This "OR" operator creates multiple possible patterns that the string may match, but it should always use ()
parentheses to define where the options start and end. Otherwise, it defaults to the whole RegEx including things like ^
in the first pattern and $
in the last pattern. See the examples below:
In programming languages you'll often see a distinction between match and full match as two different functions. The full match function will make sure the entire input string is the match that comes out of the given RegEx.
Sometimes, partial matches can be intentional, but without a clear delimiter between the validated and unvalidated input. Take the following check, for example:
While it looks to be correctly validating the domain, this "ends with" check can be bypassed by simply adding characters before trusted.com
to form another registerable domain:
The same can sometimes happen with "starts with" checks by creating a subdomain that matches:
If you've read the rules carefully, you'll know that .
means any character, but sometimes, developers forget this when trying to match a string that should contain a literal dot. In domains, for example:
While trusted.com
may not be under your control, subXtrusted.com
or with any other character in place of the X
could be registerable to hijack a domain that passes this RegEx.
Note that this can match any character except newlines, unless the explicit "dotall" (/s
) flag is set.
The character set syntax of [abc]
can be prefixed with ^
like [^abc]
to negate it. This is often used to match everything except a given character, like when matching a quote-delimited string with:
While the .
cannot match newlines by default, this set can. The following string will match:
Many regular expressions use these characters to sanitize user input and to try to if the whole string follows a particular pattern. But in Ruby, if you can get a newline into the string, any of the lines just need to match. So one line can follow the pattern, but another line can be some arbitrary payload that gets past the check.
\n
on $
This one speaks for itself, the /i
flag specifies that a RegEx is case insensitive. By default, it is sensitive to casing, which may allow you to get through simple blocklists like this:
This is easy to bypass using something like JaVaScRiPt:
, because it won't recognize the uppercase letters. This can help in all kinds of word blocklists as many parsers don't care about casing.
Note: this example can also be bypassed by another trick, the browser ignores whitespace in protocols which allows java\tscript:
to bypass just as well.
Regular Expression character sets can contain ranges like [a-z0-9]
which matches all lowercase alphabet characters and all digits. Keep in mind that this is simply a range of ASCII characters, and no smart algorithm to try and understand what you mean. A range like [A-z]
looks like it should case insensitively match alphabet characters, but if you look what's between Z
and a
, you find the extra characters of [\]^_`
. To find exactly what's within a given range, use the following easy snippet:
A variation of this is accidental character ranges when trying to allow the literal -
character without escaping it. For example, [a-z0-9.-_]
doesn't allow just the _
, -
and .
characters, but the whole range of ./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_
.
JavaScript specifically has some more quirks that you may be able to abuse involving RegExes:
To check if any regex has such a vulnerable pattern, you could use this tool to quickly find out:
It gives you a working example as well. For the (x+x+)+y
RegEx, for example, an input like xxxxxxxxxxxxxxxxxxxxxxxxxx
already takes a few seconds to compute, while adding some more x's will make it run almost forever. With one request, a faulty application may hang when such an input is given and never recover, crashing the application.
While DoS is a possibility, in some specific cases you can gain more from this vulnerability. The timing information can also leak something about the string being matched because some strings will parse faster than others.
If you have control over the Regular Expression, and some secret string is being matched by your RegEx, you could use this to create a RegEx that will be very slow if the first character is an "A", but very fast if the first character is not an "A". Then you can slowly brute-force the secret string character by character.
Such a pattern would be:
A smart regex parser would first look if the string starts with <text>
, and if it does not, it stops instantly because it knows it will never match. Then if it does start with <text>
, it will evaluate the rest of the (((((.*)*)*)*)*)!
which is the computationally expensive part. That way we know that the string being matched starts with <text>
if the application takes long to respond.
Now we can try every possible letter in the place of <text> until the application hangs. Then we save the newly found character and brute-force the next character, etc. See an example implementation I made in Python below:
Another useful tool to visualize RegExes is . Just put a RegEx in there, and you'll get a nice image that explains the patterns, groups, etc.
The have some detailed explanations of all the special characters RegEx uses, so check those out to fully understand it from the ground up. If you're already a bit familiar with how RegEx works, here's a list of all the special characters and what they do:
Some implementations of RegEx have a little different syntax for these replacements, Python's for example, uses the \1
backslash instead of the $1
dollar sign.
The match will contain the \n
newlines which can cause problems if they are expected to be a trusted character that shouldn't be injected. This is useful in for Nginx trying to extract a URL part from such a pattern.
Multi-line mode often recognizable by /m
or sometimes enabled by default (such as in ) makes ^
and $
act differently. Normally, these would represent the start and end of the whole string. But in multi-line mode, these are the start and end of the line. This means that if there are newline characters in the string, the ^
for example could match against a line earlier or later in the string.
One strange quirk is that even in non-multiline RegExes, the $
can match a newline (without any other characters after it). This quirk was to handle terminal inputs readline()
functions often ending with newlines, but ended up being ported to many implementations in other languages.
This can be useful to cause errors, one example is in PHP's function which will silently ignore setting the header if it contains a newline.
Replacing only one or all occurrences:
With input into the replacement, inject placeholders:
Mutate the index of re-used global regexes:
ReDoS stands for "Regular Expression Denial of Service". It is when you have such a computationally expensive search pattern, that the system takes a bit of time before returning the result. This can be used to slow down a system, causing Denial of Service. There are easy pitfalls to make when writing Regular Expressions that make it possible for the computation required for some inputs to go up exponentially, called .
See for an example involving NoSQL Injection
For finding bypasses and edge cases or true values for a Regular Expression, check out the solver.