Timing of Penetration Tests

Reading time is 4 minutes

TL-DR: Performing a security test for an application in development is like shooting at a moving target. What is the best approach and timing for this tricky activity?

The usual course of squishing software bugs

The typical fixing process for a newly discovered security flaw consists of the following steps:

• Developers analyze the discovered vulnerability and develop a fix,
• The fixed code gets tested then deployed,
• If all goes well, the documentation may have to be updated accordingly.

However, some of the security vulnerabilities like broken authorization mechanisms or application-wide Insecure Direct Object Reference (IDOR) may have a wide-ranging impact on the application and even may require significant design changes. Similarly, it may be very laborious to make major version upgrades to vulnerable client-side JavaScript components. Sounds like considerable rework, right?

Such cases are more frequent than one would like to think of, and yes, they usually have a significant impact on the cost, as well as the project delivery dates.

The problem of timing

Thinking about serious security flaws, performing the security tests at the early stages of the development seems like the right choice. Early testing would theoretically save time, since being aware of securıty issues early on would minimize rework.

However, there are 2 main drawbacks to this approach:

1. The entire function set is not available for testing at the early stage. Therefore, the test scope will be limited. This may lead to inadequate coverage. Even a single function not being tested can result in the compromise of the system and the data. This would mean inadequate testing.
2. The tested portions of the app will likely go through changes as the development continues. User Acceptance Test (UAT) phase, which takes place after the development, typically triggers many changes, both because of discovering functional bugs, and for improving the user experience. Add the regression effect into the mix and the version used in security testing would no longer be a proper representation of the final version. Resultantly, the security test is very likely to fail identifying the vulnerabilities.

If we prioritize test coverage, the ideal time to start would be after the User Acceptance Test (UAT) completes. This way, test would be on the final product. The obvious downside of this would be that identifying a critical vulnerability would mean rework, increased costs, and slippages in project delivery.

Determining the correct approach for timing & scoping of security tests is not a trivial problem. It requires a more refined approach.

The refined approach

To find an improved solution, we must take a closer look at the related processes, namely penetration testing, and software development lifecycle (SDLC). 

By examining, categorizing, and prioritizing the items in our security testing checklist, we identified the checks that can we can perform at earlier stages, also assessing whether or not doing so would provide benefits to our clients by avoiding reworks and redesigns.

The relevant sections are listed as follows:

• session management mechanism checks,
• input validation controls,
• back-end controls against Insecure Direct Object Reference (IDOR) issues,
• authorization controls for various user roles in the app,
• potential harmful/fraudulent activities (E.g., malicious file uploads, rate-limiting controls against data scraping or mass querying of user data.)

These aspects are likely to have significant rework impact if we discover them in later stages of the development. 

Both our experiences on SDLC and common sense tell us that most of the critical bugs get discovered until the first half of the UATs. This concentration means that in most projects, a low number of bugs remain for the second half of the UATs. Therefore there may be room kicking off the pentest at an earlier time. 

As a result of this simple analysis, our suggestion is to implement a two-phased penetration test approach:

Phase 1: is a limited testing round. It aims to check only some key security aspects at an early stage, to identify the issues that would be more costly to repair later on.

An example is the authorization mechanism. Having to redesign it after project completion would be a significant setback. However, if developers could be alerted early, they would save precious time & effort and prevent slippages in the delivery schedule.

A similar case is with the IDOR vulnerabilities. They are usually high-risk findings that can allow an attacker to use the existing application functionality for gaining unauthorized access to data, or to perform unauthorized activities. Usually, all that an attacker needs is altering a few parameters in a simple request. Instead of having to go back to build back-end controls for every attack scenario after project completion, the developer can save enormous time and effort by just designing a robust set of controls at the early development stages.

The first phase would also be likely to decrease the total security testing efforts and contribute to savings.

Phase 2: is a full test that starts at 40% UAT progression. This phase aims to perform a full coverage of all the application functionalities, using a late version that is representative of the final version. If the first round has identified some critical vulnerabilities, then the issues identified in round 2 are likely easier to fix.

Conclusion

The two-phase approach that we recommend for projects in development provides adequate coverage while minimizing reworks and the probability of budget and calendar overruns.