🚩
Practical CTF
BlogContact
  • 🚩Home - Practical CTF
  • 🌐Web
    • Enumeration
      • Finding Hosts & Domains
      • Masscan
      • Nmap
      • OSINT
    • Client-Side
      • Cross-Site Scripting (XSS)
        • HTML Injection
        • Content-Security-Policy (CSP)
      • CSS Injection
      • Cross-Site Request Forgery (CSRF)
      • XS-Leaks
      • Window Popup Tricks
      • CRLF / Header Injection
      • WebSockets
      • Caching
    • Server-Side
      • SQL Injection
      • NoSQL Injection
      • GraphQL
      • XML External Entities (XXE)
      • HTTP Request Smuggling
      • Local File Disclosure
      • Arbitrary File Write
      • Reverse Proxies
    • Frameworks
      • Flask
      • Ruby on Rails
      • NodeJS
      • Bun
      • WordPress
      • Angular
    • Chrome Remote DevTools
    • ImageMagick
  • 🔣Cryptography
    • Encodings
    • Ciphers
    • Custom Ciphers
      • Z3 Solver
    • XOR
    • Asymmetric Encryption
      • RSA
      • Diffie-Hellman
      • PGP / GPG
    • AES
    • Hashing
      • Cracking Hashes
      • Cracking Signatures
    • Pseudo-Random Number Generators (PRNG)
    • Timing Attacks
    • Blockchain
      • Smart Contracts
      • Bitcoin addresses
  • 🔎Forensics
    • Wireshark
    • File Formats
    • Archives
    • Memory Dumps (Volatility)
    • VBA Macros
    • Grep
    • Git
    • File Recovery
  • ⚙️Reverse Engineering
    • Ghidra
    • Angr Solver
    • Reversing C# - .NET / Unity
    • PowerShell
  • 📟Binary Exploitation
    • ir0nstone's Binary Exploitation Notes
    • Reverse Engineering for Pwn
    • PwnTools
    • ret2win
    • ret2libc
    • Shellcode
    • Stack Canaries
    • Return-Oriented Programming (ROP)
      • SigReturn-Oriented Programming (SROP)
      • ret2dlresolve
    • Sandboxes (chroot, seccomp & namespaces)
    • Race Conditions
  • 📲Mobile
    • Setup
    • Reversing APKs
    • Patching APKs
    • HTTP(S) Proxy for Android
    • Android Backup
    • Compiling C for Android
    • iOS
  • 🌎Languages
    • PHP
    • Python
    • JavaScript
      • Prototype Pollution
      • postMessage Exploitation
    • Java
    • C#
    • Assembly
    • Markdown
    • LaTeX
    • JSON
    • YAML
    • CodeQL
    • NASL (Nessus Plugins)
    • Regular Expressions (RegEx)
  • 🤖Networking
    • Modbus - TCP/502
    • Redis/Valkey - TCP/6379
  • 🐧Linux
    • Shells
    • Bash
    • Linux Privilege Escalation
      • Enumeration
      • Networking
      • Command Triggers
      • Command Exploitation
      • Outdated Versions
      • Network File Sharing (NFS)
      • Docker
      • Filesystem Permissions
    • Analyzing Processes
  • 🪟Windows
    • The Hacker Recipes - AD
    • Scanning/Spraying
    • Exploitation
    • Local Enumeration
    • Local Privilege Escalation
    • Windows Authentication
      • Kerberos
      • NTLM
    • Lateral Movement
    • Active Directory Privilege Escalation
    • Persistence
    • Antivirus Evasion
    • Metasploit
    • Alternate Data Streams (ADS)
  • ☁️Cloud
    • Kubernetes
    • Microsoft Azure
  • ❔Other
    • Business Logic Errors
    • Password Managers
    • ANSI Escape Codes
    • WSL Tips
Powered by GitBook
On this page
  • Description
  • Where can I use them?
  • Rules/Cheatsheet
  • Examples
  • Replacing
  • Code examples
  • Common Bypasses
  • Partial match
  • Dot matches all
  • Multi-line matching
  • Case sensitivity
  • Overly large range
  • ReDoS (Catastrophic Backtracking)
  • Input Exfiltration via Timing Attack
  • Solving & Finding Bypasses
  1. Languages

Regular Expressions (RegEx)

Regular Expressions are a syntax for writing patterns to match for. Lot of symbols mean something allowing you to write complex rules in a very short string

PreviousNASL (Nessus Plugins)NextModbus - TCP/502

Last updated 9 days ago

Description

Regular Expressions (RegEx) are a way of writing patterns that many languages understand. Almost every language has some library or way to work with regular expressions, and they are really useful for quickly finding something.

The syntax for RegEx may be hard to read at first. There is not really a way to make comments, and they are supposed to be very compact. But after working with them for a bit and understanding all the rules, you can quickly understand what a RegEx does. One great site that I always use for testing and creating Regular Expressions is RegExr:

Where can I use them?

Many languages have some library or native way to interpret Regular Expressions. Here are two examples:

Python
import re  # Import the Regular Expression library

regex = r'ab+c'  # Regex with r'' (raw) string to ignore \ escaping
s = "some text containing abbbbc to match"

print(re.findall(regex, s))  # ['abbbbc']
JavaScript
const regex = /ab+c/;  // Define with / around pattern
s = "some text containing abbbbc to match";

console.log(regex.exec(s));  // ['abbbbc', index: 21, input: 'some text containing abbbbc to match', groups: undefined]
PHP
<?php
$regex = '/ab+c/';  // Define in string with / around pattern
$s = "some text containing abbbbc to match";
preg_match($regex, $s, $matches);
print_r($matches);  // array(1) {[0] => string(6) "abbbbc"}

To search through files or command output with these regular expressions, you can use Grep which supports advanced regular expressions using the -P option (and use ' single quotes to avoid escaping issues).

$ grep -P 'ab+c' /etc/passwd
abbbbc:1000:1000:,,,:/home/abbbbc:/bin/bash

Lots of code editors (IDEs) also allow you to search through your code using regular expressions. This can be really powerful in combination with the Replacing feature to transform a pattern in your code without doing everything by hand. You can often enable this feature by clicking a .* button.

Find:    'a(b+)c'
Replace: 'd$1e'
Result
some text containing abbbbc to match  # Before
some text containing dbbbbe to match  # After

Rules/Cheatsheet

Character classes

.

Any character except newline

\w \d \s

Word, digit, whitespace

\W \D \S

Not word, digit, whitespace

[abc]

Any of a, b, or c

[^abc]

Not a, b, or c

[a-g]

Character between a & g

Anchors

^abc$

Start / end of the string

\b \B

Word, not-word boundary

Escaped characters

\. \* \\

Escaped special characters

\t \n \r

Tab, linefeed, carriage return

Groups & Lookaround

(abc)

Capture group

\1

Backreference to group #1

(?:abc)

Non-capturing group

(?=abc)

Positive lookahead

(?!abc)

Negative lookahead

Quantifiers & Alternation

a*a+a?

0 or more, 1 or more, 0 or 1

a{5}a{2,}

Exactly five, two or more

a{1,3}

Between one & three

a+?a{2,}?

Match as few as possible

ab|cd

Match ab or cd

Examples

32 hex characters:      [0-9a-f]{32}
Website URL:            https?:\/\/(www\.)?[-a-zA-Z0-9@:%._\+~#=]{2,256}\.[a-z]{2,4}\b([-a-zA-Z0-9@:%_\+.~#?&//=]*)
Valid IPv4 Address:     (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)
Simple Email address:   [A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}
RFC Email address:      (?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])
Floating-point numbers: ^[-+]?[0-9]*.?[0-9]+([eE][-+]?[0-9]+)?$

Replacing

Regular Expressions can also be used to replace matches with something. Using groups with () around parts of the pattern, you can even include groups back in the replacement. This is really useful for changing specific things around or in your pattern, without doing it manually. Here are the variables you can use in the replacement string:

  • $&: Full match

  • $1: First group

  • $2: Second group

  • etc.

You can use the $& anywhere in your replacement string to insert the full match. This is useful if you want to add some characters around the match, instead of changing it. You can also get any groups with $n, where n is the number of the group in your search pattern.

Code examples

Python
import re

search = r'a(b+)c'  # Find a series of b's in between a and c
replace = r'd\1e'  # Replace match with b's in between d and e
string = "some text containing abbbbc to match"

print(re.sub(search, replace, string))  # "some text containing dbbbbe to match"

# You can also define a function to transform the match
def uppercase(s):
    return s.group(0).upper()  # Group 0 is whole match

print(re.sub(search, uppercase, string))  # "some text containing ABBBBC to match"
JavaScript
const search = /a(b+)c/;
const replace = 'd$1e';
const string = "some text containing abbbbc to match";

console.log(string.replace(search, replace));  // "some text containing dbbbbe to match"

// You can also define a function to transform the match
function uppercase(s) {
    return s.toUpperCase();
}

console.log(string.replace(search, uppercase));  // "some text containing ABBBBC to match"
PHP
<?php
$search = '/a(b+)c/';
$replace = 'd$1e';
$string = "some text containing abbbbc to match";

var_dump(preg_replace($search, $replace, $string));  // string(36) "some text containing dbbbbe to match"

Common Bypasses

Partial match

Regular Expressions are often used in two possible ways: finding a string embedded in some text, or matching the whole text against some pattern. It can be easy to mistakenly match anywhere in the text instead of the whole text, which is what this common bypass is about. The ^ and $ characters "anchor" the pattern to the start or end of the string, see some examples below:

// Must start with "foo"
"a foo".match(/^foo/) // false
"foo b".match(/^foo/) // true
// Must end with "bar"
"bar b".match(/bar$/) // false
"a bar".match(/bar$/) // true
// Must exactly be 3 characters
"abcd".match(/^...$/) // false
"abc" .match(/^...$/) // true

As you can imagine, it's easy to forget the ^ and $ characters. If you do, any string will partially match. That causes the center of the string to be validated, but the surrounding text can still contain any special characters. Below is an example that attempts to validate a URL:

Vulnerable
const url = new URLSearchParams(location.search).get("url");
// Match "https://" or "http://" and anything after
if (url.match(/https?:\/\/.+/)) {
  location = url;
}

The above can be exploited with a javascript: URL, while it may seem the https: or http: protocol is required, it may be anywhere in the string:

Exploit
javascript:alert(origin)//https://example.com

This will trigger Cross-Site Scripting (XSS) and passes the regex using the valid URL in the comment after it.

Even with correctly adding ^ and $ around the whole RegEx, some specific syntax using | may still cause it to be vulnerable. This "OR" operator creates multiple possible patterns that the string may match, but it should always use () parentheses to define where the options start and end. Otherwise, it defaults to the whole RegEx including things like ^ in the first pattern and $ in the last pattern. See the examples below:

/* ^a|b|c$ is evaluated as (^a)|(b)|(c$) */
"a"        .match(/^a|b|c$/) // true
"d"        .match(/^a|b|c$/) // false
"a foo"    .match(/^a|b|c$/) // true
"foo b foo".match(/^a|b|c$/) // true
"foo c"    .match(/^a|b|c$/) // true

In programming languages you'll often see a distinction between match and full match as two different functions. The full match function will make sure the entire input string is the match that comes out of the given RegEx.

Sometimes, partial matches can be intentional, but without a clear delimiter between the validated and unvalidated input. Take the following check, for example:

Vulnerable domain validation
"trusted.com"    .match(/trusted\.com$/) // true
"sub.trusted.com".match(/trusted\.com$/) // true
"attacker.com"   .match(/trusted\.com$/) // false

While it looks to be correctly validating the domain, this "ends with" check can be bypassed by simply adding characters before trusted.com to form another registerable domain:

"attackertrusted.com".match(/trusted\.com$/) // true

The same can sometimes happen with "starts with" checks by creating a subdomain that matches:

"https://trusted.com.attacker.com".match(/^https?:\/\/trusted\.com/) // true

Dot matches all

If you've read the rules carefully, you'll know that . means any character, but sometimes, developers forget this when trying to match a string that should contain a literal dot. In domains, for example:

Vulnerable domain validation
"sub.trusted.com".match(/^sub.trusted.com$/) // true
"sub.trusted.com".match(/^subXtrusted.com$/) // true

While trusted.com may not be under your control, subXtrusted.com or with any other character in place of the X could be registerable to hijack a domain that passes this RegEx.

Note that this can match any character except newlines, unless the explicit "dotall" (/s) flag is set.

Negated set can match newline

The character set syntax of [abc] can be prefixed with ^ like [^abc] to negate it. This is often used to match everything except a given character, like when matching a quote-delimited string with:

RegEx
"[^"]*"

While the . cannot match newlines by default, this set can. The following string will match:

some "string
containing
newlines"

Multi-line matching

Many regular expressions use these characters to sanitize user input and to try to if the whole string follows a particular pattern. But in Ruby, if you can get a newline into the string, any of the lines just need to match. So one line can follow the pattern, but another line can be some arbitrary payload that gets past the check.

Example
Boolean("foo\nbar".match(/^foo$/))  // false
Boolean("foo\nbar".match(/^foo$/m)) // true
Boolean("bar\nfoo\nbar".match(/^foo$/m)) // true

Non-multiline still matches \n on $

Python
re.match(r'^foo$', "foo")    # True
re.match(r'^foo$', "foo\n")  # True
re.match(r'^foo$', "foo\na") # False

Case sensitivity

This one speaks for itself, the /i flag specifies that a RegEx is case insensitive. By default, it is sensitive to casing, which may allow you to get through simple blocklists like this:

const url = new URLSearchParams(location.search).get("url");
// Security measure
if (url.match(/javascript/)) {
  throw new Error("Dangerous URL blocked");
}
location = url;

This is easy to bypass using something like JaVaScRiPt:, because it won't recognize the uppercase letters. This can help in all kinds of word blocklists as many parsers don't care about casing.

Note: this example can also be bypassed by another trick, the browser ignores whitespace in protocols which allows java\tscript: to bypass just as well.

Overly large range

Regular Expression character sets can contain ranges like [a-z0-9] which matches all lowercase alphabet characters and all digits. Keep in mind that this is simply a range of ASCII characters, and no smart algorithm to try and understand what you mean. A range like [A-z] looks like it should case insensitively match alphabet characters, but if you look what's between Z and a, you find the extra characters of [\]^_`. To find exactly what's within a given range, use the following easy snippet:

Python
''.join([chr(c) for c in range(ord("A"), ord("z")+1)])

A variation of this is accidental character ranges when trying to allow the literal - character without escaping it. For example, [a-z0-9.-_] doesn't allow just the _, - and . characters, but the whole range of ./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_.


JavaScript specifically has some more quirks that you may be able to abuse involving RegExes:

ReDoS (Catastrophic Backtracking)

To check if any regex has such a vulnerable pattern, you could use this tool to quickly find out:

It gives you a working example as well. For the (x+x+)+y RegEx, for example, an input like xxxxxxxxxxxxxxxxxxxxxxxxxx already takes a few seconds to compute, while adding some more x's will make it run almost forever. With one request, a faulty application may hang when such an input is given and never recover, crashing the application.

Input Exfiltration via Timing Attack

While DoS is a possibility, in some specific cases you can gain more from this vulnerability. The timing information can also leak something about the string being matched because some strings will parse faster than others.

If you have control over the Regular Expression, and some secret string is being matched by your RegEx, you could use this to create a RegEx that will be very slow if the first character is an "A", but very fast if the first character is not an "A". Then you can slowly brute-force the secret string character by character.

Such a pattern would be:

<text>(((((.*)*)*)*)*)!
or
(?=<text>).*.*.*.*.*.*.*.*!!!!

A smart regex parser would first look if the string starts with <text>, and if it does not, it stops instantly because it knows it will never match. Then if it does start with <text>, it will evaluate the rest of the (((((.*)*)*)*)*)! which is the computationally expensive part. That way we know that the string being matched starts with <text> if the application takes long to respond.

Now we can try every possible letter in the place of <text> until the application hangs. Then we save the newly found character and brute-force the next character, etc. See an example implementation I made in Python below:

from functools import wraps
import errno
import os
import signal
import re

ALPHABET = list(b" {}_Ee3Aa@4RrIi1Oo0Tt7NnSs25$LlCcUuDdPpMmHhGg6BbFfYyWwKkVvXxZzJjQq89-,.!?'\"\n\r#%&()*+/\\:;<=>[]^`|~")  # Most common

# https://stackoverflow.com/a/2282656/10508498
class TimeoutError(Exception):
    pass

def timeout(seconds=10, error_message=os.strerror(errno.ETIME)):
    def decorator(func):
        def _handle_timeout(signum, frame):
            raise TimeoutError(error_message)

        def wrapper(*args, **kwargs):
            signal.signal(signal.SIGALRM, _handle_timeout)
            signal.setitimer(signal.ITIMER_REAL,seconds) #used timer instead of alarm
            try:
                result = func(*args, **kwargs)
            finally:
                signal.alarm(0)
            return result
        return wraps(func)(wrapper)
    return decorator


@timeout(0.1)
def match(regex):  # Target function
    re.match(regex, "CTF{f4k3_fl4g_f0r_t3st1ng}")


def attack():
    leaked = b""
    
    while True:
        for letter in ALPHABET:
            prefix = re.escape(leaked + bytes([letter])).decode()
            # regex = rf"{prefix}(((((((((((((.*)*)*)*)*)*)*)*)*)*)*)*)*)!"  # Doesn't get last byte
            regex = rf"(?={prefix}).*.*.*.*.*.*.*.*.*.*.*.*.*.*.*!!!!"  # Also gets last byte
            try:
                match(regex)
            except TimeoutError:
                leaked += bytes([letter])
                print(leaked)
                break
        else:
            break
        
    return leaked


print(attack())  # b'CTF{f4k3_fl4g_f0r_t3st1ng}'

Because RegEx is so flexible, it is possible to achieve Binary Search performance with your leaks. By providing a range of characters like [a-m], the true/false response can tell a lot more than if one certain character is in there. False means that the whole range can be discarded, and True means the correct character should be somewhere in there. You can provide a smaller range like [a-g] now to get to your goal in log2() of the time it would have taken by pure brute-force.

Solving & Finding Bypasses

Another useful tool to visualize RegExes is . Just put a RegEx in there, and you'll get a nice image that explains the patterns, groups, etc.

The have some detailed explanations of all the special characters RegEx uses, so check those out to fully understand it from the ground up. If you're already a bit familiar with how RegEx works, here's a list of all the special characters and what they do:

Some implementations of RegEx have a little different syntax for these replacements, Python's for example, uses the \1 backslash instead of the $1 dollar sign.

The match will contain the \n newlines which can cause problems if they are expected to be a trusted character that shouldn't be injected. This is useful in for Nginx trying to extract a URL part from such a pattern.

Multi-line mode often recognizable by /m or sometimes enabled by default (such as in ) makes ^ and $ act differently. Normally, these would represent the start and end of the whole string. But in multi-line mode, these are the start and end of the line. This means that if there are newline characters in the string, the ^ for example could match against a line earlier or later in the string.

One strange quirk is that even in non-multiline RegExes, the $ can match a newline (without any other characters after it). This quirk was to handle terminal inputs readline() functions often ending with newlines, but ended up being ported to many implementations in other languages.

This can be useful to cause errors, one example is in PHP's function which will silently ignore setting the header if it contains a newline.

Replacing only one or all occurrences:

With input into the replacement, inject placeholders:

Mutate the index of re-used global regexes:

ReDoS stands for "Regular Expression Denial of Service". It is when you have such a computationally expensive search pattern, that the system takes a bit of time before returning the result. This can be used to slow down a system, causing Denial of Service. There are easy pitfalls to make when writing Regular Expressions that make it possible for the computation required for some inputs to go up exponentially, called .

See for an example involving NoSQL Injection

For finding bypasses and edge cases or true values for a Regular Expression, check out the solver.

🌎
Regexper
MDN web docs
re.sub
leftover from Perl
header()
"Catastrophic Backtracking"
RegExr: Learn, Build, & Test RegExRegExr
A site to create, explain and test Regular Expressions
ReDoS checker | Devina.io
Input a regex, and solve/fuzz to check if it is vulnerable, and give working payload examples
CRLF Injection
CrossHair: RegEx and more
Ruby
Logo
replace vs replaceAll
Replacement String Templates
Global Regexes
Logo
RegEx Binary Search