Join us as we explore our use case of exhaustive testing during the refactoring process. Hear directly from the team about the challenges faced, the solutions implemented, and how this approach restored clarity and functionality while shaping the development process.
Tackling Complexity in Software
In the realm of complex software systems, managing "complexity hotspots" is a critical challenge for development teams. These areas are marked by high complexity and essential functionality, often becoming increasingly difficult to maintain over time. In this article, we will explore the concept of exhaustive testing as a vital strategy for ensuring software reliability, using the refactoring of our QuoteEngine—a subsystem responsible for calculating bet prices for our Cashout product—as a key example.
As our system evolved, the QuoteEngine transformed into a complexity hotspot, leading to prolonged implementation times for small changes and an unexpectedly high defect rate. We observed alarming signs of distress in the codebase, including lengthy classes, convoluted methods, and the notorious "shotgun surgery" effect, where minor updates required adjustments across multiple classes.
Meet the Team: The Minds Behind This Strategy
Behind every successful testing strategy are the people who envision, implement, and refine it. Our journey into exhaustive testing began with Joana Ribeiro (QA Engineer) and João Costa (Senior Software Engineer), both key contributors to the Cashout product. While enhancing the QuoteEngine, they identified that implementing exhaustive tests could significantly improve validation processes. This realization came amidst the absence of documentation or prior guidance on the subject, prompting them to innovate and share their learnings. Their collaborative effort not only optimized our system but also led to the creation of reusable insights aimed at helping others facing similar challenges.
When Testing Every Input Isn’t Feasible
Exhaustive testing, or complete testing, aims to evaluate a system by running tests on all possible input combinations. In the context of the refactored QuoteEngine, the goal was to ensure it produced consistent results with the original system across every input. This method is crucial when dealing with numerous parameters, as manual testing becomes impractical.
While exhaustive testing theoretically helps detect bugs by evaluating all potential input scenarios, it often proves impractical in real-world applications due to the sheer number of test cases, especially in complex systems. For example, a simple application with a 4-digit numeric input has 10,000 combinations. As more variables are involved, the number of possible test cases grows exponentially, requiring significant time and resources, often exceeding typical development cycles.
To manage this complexity, we focused on key inputs and variables influencing the system's behavior, allowing us to benchmark numerical, boolean, and string values (e.g., "inplay/preplay" events) without testing every single possibility. While exhaustive testing is an ideal concept, its limitations underscore the need for effective strategies. Techniques like boundary value analysis, equivalence partitioning, and risk-based testing help testers concentrate on critical areas, ensuring thorough evaluation without the impracticalities of testing every scenario.
Refactoring the QuoteEngine: Challenges and Risks
Given the centrality of the QuoteEngine to our service—where even a small bug could lead to substantial revenue losses and reputational damage—it became clear that we needed to address these issues. We initially attempted several approaches, including surgical code changes, extensive documentation, and additional unit tests, but none fully resolved the root problems. This led to the decision to perform a large-scale refactor of the subsystem to simplify the code structure and enhance maintainability.
However, large-scale refactoring presents challenges, particularly in validating that no regressions have been introduced. The lack of unit test coverage and the system's complexity made thorough reviews difficult. Although thousands of component tests existed, they were too coarse-grained to guarantee proper handling of all edge cases. This prompted us to explore exhaustive testing as a solution to ensure that the refactor didn’t introduce unintended side effects.
Why Exhaustive Testing Was the Right Choice
The decision to implement exhaustive testing was driven by the complexity of the QuoteEngine. The subsystem had 37 different parameters, each with a variety of potential values, creating an enormous list of possible input combinations. Manually testing all these combinations was impossible due to time constraints and the potential for human error. Exhaustive testing enabled us to run these tests automatically, ensuring that the output from the refactored QuoteEngine matched that of the original system for every input permutation.
How We Designed and Executed Exhaustive Testing
1. Identifying and Categorizing Input Parameters
We began the exhaustive testing process by identifying the input parameters for the PriceCalculator within the QuoteEngine. This involved a detailed analysis of all variables influencing the calculation. Boolean inputs (e.g., inplay and HorseRacing) were defined with their two possible states: YES/NO. For numerical inputs, we applied techniques such as Boundary Value Analysis (BVA) to ensure comprehensive coverage by focusing on edge cases where transitions between boundaries occur—common areas for defects. For instance, with a threshold field set to 0.02, where X ≤ 0.02triggers different logic, BVA helped us define scenarios like 0.019, 0.02, and 0.021 to validate the boundaries effectively. Similarly, string inputs were evaluated to include only those values that significantly impacted the calculation, while redundant ones were excluded. Ultimately, this meticulous process resulted in a table comprising 37 inputs and their respective values for exhaustive validation.
2. Automating the Test Execution
Next, we set up a testing framework to conduct the exhaustive tests. The test framework loads the csv with the variable and respective values (table above), inflates the test objects with the specific combination of variable values and for each compares the output of the original and refactored QuoteEngine.
public void runExhaustiveTestSuite() {
List<List<String>> records = new ArrayList<>();
try (BufferedReader br = new BufferedReader(
new FileReader(
new ClassPathResource("Exhaustive_Testing.csv").getFile()))) {
String line;
while ((line = br.readLine()) != null) {
String[] values = line.split(COMMA_DELIMITER);
List<String> params = Arrays.asList(values).subList(1, values.length);
if (!params.isEmpty()) {
records.add(params);
}
}
} catch (IOException e) {
e.printStackTrace();
}
List<String> params = Arrays.asList(new String[records.size()]);
LOGGER.info("Running for {} inputs", records.size());
generateAndExecuteScenarios(records, params);
LOGGER.info("End of Exhaustive test run");
}
The system processes input data from the table and runs it through both the original and the refactored code. It then compares the results and logs any inconsistencies between the two. Initially, the focus was on executing the tests, but now the system is verifying the output and logging any differences it detects.
public void runScenario(Bet bet, List<String> params) {
Result newResult = newEngine.calculateTradeoutPrice(bet, params);
Result oldResult = oldEngine.calculateTradeoutPrice(bet, params);
if (!newResult.equals(oldResult)) {
LOGGER.error("Inconsistent scenario, old={}, new={}, bet={}", oldResult, newResult, bet);
}
long count = counter.incrementAndGet();
if (count % 1_000_000 == 0) {
LOGGER.info("Executed scenario so far: {}M, with params: {}", count / 1_000_000, params);
}
Proving the Approach: Results and Validation
In total, 290 237 644 800 tests were executed, one for each permutation of the input parameters. This extensive test set ran for 10h hours, after which no discrepancies were found between the original and refactored versions of the QuoteEngine. This validation gave us the confidence to deploy the refactored system into production, knowing it would maintain the same functionality as before.
It’s important to note that we did not use this exhaustive test suite as part of our ongoing regression testing strategy. Instead, it served as a one-time validation tool during the large-scale refactor. After successful deployment, we returned our focus to unit and component tests to ensure ongoing code quality.
Lessons from Exhaustive Testing
Exhaustive testing proved to be an essential tool in validating the refactor of our QuoteEngine subsystem. By systematically comparing the outputs of the original and refactored systems across all possible inputs, we were able to confidently deploy a cleaner, more maintainable version of the subsystem. This not only reduced the technical debt associated with the QuoteEngine but also provided the foundation for future improvements and bug fixes.
In a world where complex hotspots and shotgun surgery can slow down development and increase the likelihood of defects, exhaustive testing offers a powerful approach to ensuring that large-scale refactors do not introduce regressions. Though resource-intensive, this method allowed us to confidently address a critical component of our system and ensure that it continues to function reliably, now and in the future.
Joana Ribeiro, João Costa, Soraia Gonçalves
