Blip has a dedicated Security Testing team that performs penetration testing, continuous testing, and red teaming. By being part of S-SDLC (Secure Software Development Life Cycle) process, the team handles development and infrastructure security vulnerabilities. During the development phase, penetration tests are carried out to detect and correct vulnerabilities and ensure that new products are launched into production with a high-level of security.
Introduction
In the dynamic realm of cybersecurity, where the battle between defenders and adversaries intensifies daily, the discovery of vulnerabilities is both a triumph and a challenge.
In this blogpost, we rev our engines and delve into the world of race conditions in security testing, where the unexpected often lies hidden beneath layers of code. Our spotlight is on a challenge encountered during a penetration test – the discovery of a race condition. We'll explore the practical aspects of identifying a race condition, and the key takeaways it brings to the forefront in our ongoing battle against threats, emphasising the importance of proactive testing in maintaining a robust digital infrastructure.
What Is a Race Condition?
Developers and testers often encounter the challenge of race conditions. These vulnerabilities, similar to a high-speed race in the digital domain, can introduce security loopholes that may compromise the integrity of systems.
In the context of penetration testing, a race condition is a security flaw that arises when a system's behaviour is affected by the order or timing of particular operations taken during the test. A race condition occurs when two or more threads can access shared data and they try to change it at the same time. Think about two threads running the same program and working with the same variables - the order in which the operations happen will lead to unexpected behaviours.
An attacker may use this kind of vulnerability to change the way things are supposed to happen, which could result in data breaches, illegal access, or other security lapses. To evaluate a system's resistance to simulated attacks, penetration testers frequently look for and take advantage of race circumstances. By doing this, they assist organisations in identifying and fixing vulnerabilities before hostile actors may use them in actual attacks.
Application Logic
In an application that stores an integer X value, a new Y value can be set, but there are a few things to consider:
Assuming there is a default starting X value:
- By choosing to set a higher Y value, then the value will be kept on hold during a period before being set.
- By choosing to set a lower Y value, the value is immediately set.
- The application continuously tracks the ACTIVE value and the PENDING value (the latter can be null).
After observing this behaviour, a simplified version of the code would be close to this:
def function(int y):
if (y > x):
wait_set(y)
else:
set(y)
Two things can happen here depending on the conditional statement:
If Y is larger than X –> ACTIVE = x and PENDING = y If Y is not larger than X –> ACTIVE = y and PENDING = null (this also applies if the variables are equal, but that won’t matter too much)
The Problem
Suppose we launch two threads of the program at the same time and the variables are shared. So, if thread 1 sets a value for Y, it is also set for thread 2. That being said, the odds of something weird happening seem null, since it requires super precise timing, right? Not quite. We will go over how James Kettle made this so much easier to exploit later. But first, let’s go back to our problem - what if we could fail the if statement (Y value lower than X) and then immediately change Y to a value higher than X mid execution, so when the line set(y) is ran, a value higher than X will become active without a cooldown?
Looking at the diagram below, on the left we have thread 1, on the right thread 2. The initial values are shown above the code, which is there for quick reference.
⚪ Starting with ACTIVE = 50 and PENDING = null 🔴 T1 sets y = 49 🔴 T1 fails the if condition since 49 > 50 ? False 🟢 T2 sets y = 51 🔴 T1 calls set(y), which translates to set(51) since T2 just set the Y variable to 51 –> ACTIVE = 51 and PENDING = null 🟢 T2 passes the if condition since 51 > 50 ? True 🟢 T2 calls wait_set(51) –> ACTIVE = 51 (same as the latest value) and PENDING = 51
Burp Suite’s repeater tab was used to send these requests by putting both tabs in a folder and selecting Send group in parallel (single-packet attack). The example below is sending the three requests in the limit-overrun folder using this technique.
It took several attempts of sending two requests in a single packet (one with a higher value than ACTIVE, other with a lower value) to make this actually work. The server would act as if only one of the values was sent. No pattern or consistent behaviour was noted, but one of the values would indeed be set. It would either overwrite ACTIVE (lower value) or set a new PENDING (higher value), depending on which one the server “chooses”. To not reuse the same values, after every attempt the higher value was incremented and the lower value decremented - for example, send 51 and 49, then 52 and 48, 53 and 47 and so on.
After many attempts, the response size suddenly shot up. By taking a more careful look, two ACTIVE variables were observed. One set to the higher value and another one set to the lower one. Even if a request that fetched the current values was sent, the response would be something like this:
ACTIVE = 51 ACTIVE = 49 PENDING = null
Understanding What Happened
After getting 2 ACTIVE variables, setting up a new value was attempted, but the server response was a 500 Internal Server Error. The same happened when performing operations that required these values. Even after resetting the environment and repeating the attack, the behaviour was consistent.
Timing Is The Key
A race condition is difficult to exploit because there are a lot of timing variables involved, and one has to be lucky enough to have each single line executed in a specific order. Taken from James Kettle:
Even if the requests are coded to be sent at the exact same time, there’s the network latency, the jitter (time between transmission and reception of data) and the server’s latency.
After some iterations, James’ final solution is to send several requests in a single TCP packet. Which means the jitter and latencies involved will all be the same. There’s still some trial and error due to the processing times on the server, but the effectiveness of this attack goes up exponentially.
Other techniques were attempted, such as using Burp Suite’s intruder and last-byte sync, but the only time interesting results were obtained was when the requests were sent in a single TCP packet.
Conclusion
Applications that involve concurrent processing can easily incur this type of vulnerability. One way to avoid them is to design the application with the concept of transactions, in which parallel transactions that share resources can include several operations each, but one transaction is only executed when no other transaction is taking place (atomic execution). Another way to reduce the risk of this vulnerability and other types of attacks is to implement a "rate limit" in which the user must wait a few seconds before sending a second operation.
Our experience with penetration testing emphasises the importance of diligent security checks. By fixing flaws like race conditions, we prepare our systems to handle both simulated and real-world threats effectively. That being said, encountering situations like this is never a setback but always chance to improve.
Are you a Security Testing Engineer looking to join our team? Check out the opportunity we have available.
Bruno Caseiro, Soraia Goncalves
