Regular Expressions (RegEx)

Regular Expressions are a syntax for writing patterns to match for. Lot of symbols mean something allowing you to write complex rules in a very short string

Description

Regular Expressions (RegEx) are a way of writing patterns that many languages understand. Almost every language has some library or way to work with regular expressions, and they are really useful for quickly finding something.

The syntax for RegEx may be hard to read at first. There is not really a way to make comments, and they are supposed to be very compact. But after working with them for a bit and understanding all the rules, you can quickly understand what a RegEx does. One great site that I always use for testing and creating Regular Expressions is RegExr:

Where can I use them?

Many languages have some library or native way to interpret Regular Expressions. Here are two examples:

Python

import re  # Import the Regular Expression library

regex = r'ab+c'  # Regex with r'' (raw) string to ignore \ escaping
s = "some text containing abbbbc to match"

print(re.findall(regex, s))  # ['abbbbc']

JavaScript

const regex = /ab+c/;  // Define with / around pattern
s = "some text containing abbbbc to match";

console.log(regex.exec(s));  // ['abbbbc', index: 21, input: 'some text containing abbbbc to match', groups: undefined]

PHP

<?php
$regex = '/ab+c/';  // Define in string with / around pattern
$s = "some text containing abbbbc to match";
preg_match($regex, $s, $matches);
print_r($matches);  // array(1) {[0] => string(6) "abbbbc"}

To search through files or command output with these regular expressions, you can use Grep which supports advanced regular expressions using the -P option (and use ' single quotes to avoid escaping issues).

$ grep -P 'ab+c' /etc/passwd
abbbbc:1000:1000:,,,:/home/abbbbc:/bin/bash

Lots of code editors (IDEs) also allow you to search through your code using regular expressions. This can be really powerful in combination with the Replacing feature to transform a pattern in your code without doing everything by hand. You can often enable this feature by clicking a .* button.

Find:    'a(b+)c'
Replace: 'd$1e'

Result

some text containing abbbbc to match  # Before
some text containing dbbbbe to match  # After

Rules/Cheatsheet

Character classes

.

Any character except newline

\w \d \s

Word, digit, whitespace

\W \D \S

Not word, digit, whitespace

[abc]

Any of a, b, or c

[^abc]

Not a, b, or c

[a-g]

Character between a & g

Anchors

^abc$

Start / end of the string

\b \B

Word, not-word boundary

Escaped characters

\. \* \\

Escaped special characters

\t \n \r

Tab, linefeed, carriage return

Groups & Lookaround

(abc)

Capture group

\1

Backreference to group #1

(?:abc)

Non-capturing group

(?=abc)

Positive lookahead

(?!abc)

Negative lookahead

Quantifiers & Alternation

a*a+a?

0 or more, 1 or more, 0 or 1

a{5}a{2,}

Exactly five, two or more

a{1,3}

Between one & three

a+?a{2,}?

Match as few as possible

ab|cd

Match ab or cd

Examples

32 hex characters:      [0-9a-f]{32}
Website URL:            https?:\/\/(www\.)?[-a-zA-Z0-9@:%._\+~#=]{2,256}\.[a-z]{2,4}\b([-a-zA-Z0-9@:%_\+.~#?&//=]*)
Valid IPv4 Address:     (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)
Simple Email address:   [A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}
RFC Email address:      (?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])
Floating-point numbers: ^[-+]?[0-9]*.?[0-9]+([eE][-+]?[0-9]+)?$

Replacing

Regular Expressions can also be used to replace matches with something. Using groups with () around parts of the pattern, you can even include groups back in the replacement. This is really useful for changing specific things around or in your pattern, without doing it manually. Here are the variables you can use in the replacement string:

$&: Full match
$1: First group
$2: Second group
etc.

You can use the $& anywhere in your replacement string to insert the full match. This is useful if you want to add some characters around the match, instead of changing it. You can also get any groups with $n, where n is the number of the group in your search pattern.

Code examples

Python

import re

search = r'a(b+)c'  # Find a series of b's in between a and c
replace = r'd\1e'  # Replace match with b's in between d and e
string = "some text containing abbbbc to match"

print(re.sub(search, replace, string))  # "some text containing dbbbbe to match"

# You can also define a function to transform the match
def uppercase(s):
    return s.group(0).upper()  # Group 0 is whole match

print(re.sub(search, uppercase, string))  # "some text containing ABBBBC to match"

JavaScript

const search = /a(b+)c/;
const replace = 'd$1e';
const string = "some text containing abbbbc to match";

console.log(string.replace(search, replace));  // "some text containing dbbbbe to match"

// You can also define a function to transform the match
function uppercase(s) {
    return s.toUpperCase();
}

console.log(string.replace(search, uppercase));  // "some text containing ABBBBC to match"

PHP

<?php
$search = '/a(b+)c/';
$replace = 'd$1e';
$string = "some text containing abbbbc to match";

var_dump(preg_replace($search, $replace, $string));  // string(36) "some text containing dbbbbe to match"

Common Bypasses

Partial match

Regular Expressions are often used in two possible ways: finding a string embedded in some text, or matching the whole text against some pattern. It can be easy to mistakenly match anywhere in the text instead of the whole text, which is what this common bypass is about. The ^ and $ characters "anchor" the pattern to the start or end of the string, see some examples below:

// Must start with "foo"
"a foo".match(/^foo/) // false
"foo b".match(/^foo/) // true
// Must end with "bar"
"bar b".match(/bar$/) // false
"a bar".match(/bar$/) // true
// Must exactly be 3 characters
"abcd".match(/^...$/) // false
"abc" .match(/^...$/) // true

As you can imagine, it's easy to forget the ^ and $ characters. If you do, any string will partially match. That causes the center of the string to be validated, but the surrounding text can still contain any special characters. Below is an example that attempts to validate a URL:

Vulnerable

const url = new URLSearchParams(location.search).get("url");
// Match "https://" or "http://" and anything after
if (url.match(/https?:\/\/.+/)) {
  location = url;
}

The above can be exploited with a javascript: URL, while it may seem the https: or http: protocol is required, it may be anywhere in the string:

Exploit

javascript:alert(origin)//https://example.com

This will trigger Cross-Site Scripting (XSS) and passes the regex using the valid URL in the comment after it.

Even with correctly adding ^ and $ around the whole RegEx, some specific syntax using | may still cause it to be vulnerable. This "OR" operator creates multiple possible patterns that the string may match, but it should always use () parentheses to define where the options start and end. Otherwise, it defaults to the whole RegEx including things like ^ in the first pattern and $ in the last pattern. See the examples below:

/* ^a|b|c$ is evaluated as (^a)|(b)|(c$) */
"a"        .match(/^a|b|c$/) // true
"d"        .match(/^a|b|c$/) // false
"a foo"    .match(/^a|b|c$/) // true
"foo b foo".match(/^a|b|c$/) // true
"foo c"    .match(/^a|b|c$/) // true

In programming languages you'll often see a distinction between match and full match as two different functions. The full match function will make sure the entire input string is the match that comes out of the given RegEx.

Sometimes, partial matches can be intentional, but without a clear delimiter between the validated and unvalidated input. Take the following check, for example:

Vulnerable domain validation

"trusted.com"    .match(/trusted\.com$/) // true
"sub.trusted.com".match(/trusted\.com$/) // true
"attacker.com"   .match(/trusted\.com$/) // false

While it looks to be correctly validating the domain, this "ends with" check can be bypassed by simply adding characters before trusted.com to form another registerable domain:

"attackertrusted.com".match(/trusted\.com$/) // true

The same can sometimes happen with "starts with" checks by creating a subdomain that matches:

"https://trusted.com.attacker.com".match(/^https?:\/\/trusted\.com/) // true

Dot matches all

If you've read the rules carefully, you'll know that . means any character, but sometimes, developers forget this when trying to match a string that should contain a literal dot. In domains, for example:

Vulnerable domain validation

"sub.trusted.com".match(/^sub.trusted.com$/) // true
"sub.trusted.com".match(/^subXtrusted.com$/) // true

While trusted.com may not be under your control, subXtrusted.com or with any other character in place of the X could be registerable to hijack a domain that passes this RegEx.

Note that this can match any character except newlines, unless the explicit "dotall" (/s) flag is set.

Negated set can match newline

The character set syntax of [abc] can be prefixed with ^ like [^abc] to negate it. This is often used to match everything except a given character, like when matching a quote-delimited string with:

RegEx

"[^"]*"

While the . cannot match newlines by default, this set can. The following string will match:

some "string
containing
newlines"

Multi-line matching

Many regular expressions use these characters to sanitize user input and to try to if the whole string follows a particular pattern. But in Ruby, if you can get a newline into the string, any of the lines just need to match. So one line can follow the pattern, but another line can be some arbitrary payload that gets past the check.

Example

Boolean("foo\nbar".match(/^foo$/))  // false
Boolean("foo\nbar".match(/^foo$/m)) // true
Boolean("bar\nfoo\nbar".match(/^foo$/m)) // true

Non-multiline still matches `\n` on `$`

Python

re.match(r'^foo$', "foo")    # True
re.match(r'^foo$', "foo\n")  # True
re.match(r'^foo$', "foo\na") # False

Case sensitivity

This one speaks for itself, the /i flag specifies that a RegEx is case insensitive. By default, it is sensitive to casing, which may allow you to get through simple blocklists like this:

const url = new URLSearchParams(location.search).get("url");
// Security measure
if (url.match(/javascript/)) {
  throw new Error("Dangerous URL blocked");
}
location = url;

This is easy to bypass using something like JaVaScRiPt:, because it won't recognize the uppercase letters. This can help in all kinds of word blocklists as many parsers don't care about casing.

Note: this example can also be bypassed by another trick, the browser ignores whitespace in protocols which allows java\tscript: to bypass just as well.

Overly large range

Regular Expression character sets can contain ranges like [a-z0-9] which matches all lowercase alphabet characters and all digits. Keep in mind that this is simply a range of ASCII characters, and no smart algorithm to try and understand what you mean. A range like [A-z] looks like it should case insensitively match alphabet characters, but if you look what's between Z and a, you find the extra characters of [\]^_`. To find exactly what's within a given range, use the following easy snippet:

Python

''.join([chr(c) for c in range(ord("A"), ord("z")+1)])

A variation of this is accidental character ranges when trying to allow the literal - character without escaping it. For example, [a-z0-9.-_] doesn't allow just the _, - and . characters, but the whole range of ./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_.

JavaScript specifically has some more quirks that you may be able to abuse involving RegExes:

ReDoS (Catastrophic Backtracking)

To check if any regex has such a vulnerable pattern, you could use this tool to quickly find out:

It gives you a working example as well. For the (x+x+)+y RegEx, for example, an input like xxxxxxxxxxxxxxxxxxxxxxxxxx already takes a few seconds to compute, while adding some more x's will make it run almost forever. With one request, a faulty application may hang when such an input is given and never recover, crashing the application.

Input Exfiltration via Timing Attack

While DoS is a possibility, in some specific cases you can gain more from this vulnerability. The timing information can also leak something about the string being matched because some strings will parse faster than others.

If you have control over the Regular Expression, and some secret string is being matched by your RegEx, you could use this to create a RegEx that will be very slow if the first character is an "A", but very fast if the first character is not an "A". Then you can slowly brute-force the secret string character by character.

Such a pattern would be:

<text>(((((.*)*)*)*)*)!
or
(?=<text>).*.*.*.*.*.*.*.*!!!!

A smart regex parser would first look if the string starts with <text>, and if it does not, it stops instantly because it knows it will never match. Then if it does start with <text>, it will evaluate the rest of the (((((.*)*)*)*)*)! which is the computationally expensive part. That way we know that the string being matched starts with <text> if the application takes long to respond.

Now we can try every possible letter in the place of <text> until the application hangs. Then we save the newly found character and brute-force the next character, etc. See an example implementation I made in Python below:

from functools import wraps
import errno
import os
import signal
import re

ALPHABET = list(b" {}_Ee3Aa@4RrIi1Oo0Tt7NnSs25$LlCcUuDdPpMmHhGg6BbFfYyWwKkVvXxZzJjQq89-,.!?'\"\n\r#%&()*+/\\:;<=>[]^`|~")  # Most common

# https://stackoverflow.com/a/2282656/10508498
class TimeoutError(Exception):
    pass

def timeout(seconds=10, error_message=os.strerror(errno.ETIME)):
    def decorator(func):
        def _handle_timeout(signum, frame):
            raise TimeoutError(error_message)

        def wrapper(*args, **kwargs):
            signal.signal(signal.SIGALRM, _handle_timeout)
            signal.setitimer(signal.ITIMER_REAL,seconds) #used timer instead of alarm
            try:
                result = func(*args, **kwargs)
            finally:
                signal.alarm(0)
            return result
        return wraps(func)(wrapper)
    return decorator


@timeout(0.1)
def match(regex):  # Target function
    re.match(regex, "CTF{f4k3_fl4g_f0r_t3st1ng}")


def attack():
    leaked = b""
    
    while True:
        for letter in ALPHABET:
            prefix = re.escape(leaked + bytes([letter])).decode()
            # regex = rf"{prefix}(((((((((((((.*)*)*)*)*)*)*)*)*)*)*)*)*)!"  # Doesn't get last byte
            regex = rf"(?={prefix}).*.*.*.*.*.*.*.*.*.*.*.*.*.*.*!!!!"  # Also gets last byte
            try:
                match(regex)
            except TimeoutError:
                leaked += bytes([letter])
                print(leaked)
                break
        else:
            break
        
    return leaked


print(attack())  # b'CTF{f4k3_fl4g_f0r_t3st1ng}'

Because RegEx is so flexible, it is possible to achieve Binary Search performance with your leaks. By providing a range of characters like [a-m], the true/false response can tell a lot more than if one certain character is in there. False means that the whole range can be discarded, and True means the correct character should be somewhere in there. You can provide a smaller range like [a-g] now to get to your goal in log2() of the time it would have taken by pure brute-force.

Solving & Finding Bypasses

PreviousNASL (Nessus Plugins)NextModbus - TCP/502

Last updated 9 days ago

Description

Where can I use them?

Rules/Cheatsheet

Examples

Replacing

Code examples

Common Bypasses

Partial match

Dot matches all

Negated set can match newline

Multi-line matching

Non-multiline still matches \n on $

Case sensitivity

Overly large range

ReDoS (Catastrophic Backtracking)

Input Exfiltration via Timing Attack

Solving & Finding Bypasses

Description

Where can I use them?

Rules/Cheatsheet

Examples

Replacing

Code examples

Common Bypasses

Partial match

Dot matches all

Negated set can match newline

Multi-line matching

Non-multiline still matches \n on $

Case sensitivity

Overly large range

ReDoS (Catastrophic Backtracking)

Input Exfiltration via Timing Attack

Solving & Finding Bypasses

Non-multiline still matches `\n` on `$`

Non-multiline still matches `\n` on `$`