For over 15 years, I have worked as a consultant in the field of information security. Primarily focusing on penetration tests, I have worked with many clients, from small businesses to large enterprises, from industries such as retail, product manufacturing, government, and education.
I’ve seen industry trends come and go, but one thing I think will dramatically improve the value of cybersecurity assessments is the adoption of continuous penetration testing.
In this article, I’ll present three personal cases where conventional fixed-time penetration tests failed to provide the desired outcomes. For each case, I hope to show how continuous penetration testing would have improved the engagement’s outcome and provided significantly more value to the customer.
Case 1: The Smart Toy
My team worked with a toy manufacturer developing a smart toy for children. The toy was a doll, designed to integrate with a mobile app used by parents to read stories and adopt multiple personas and dialog patterns. Parents would use the mobile app to download new stories and dialog, and the doll would dynamically interact with the child.
I scoped the project for multiple phases to best leverage the expertise of the testing team and the scheduling needs of the customer:
- Mobile App Security Assessment (iOS and Android)
- Cloud Services Assessment (APIs and Web Services)
- Smart Toy Security Assessment (Firmware and Hardware)
- Wireless Services Assessment (Bluetooth and Wi-Fi)
The assessment was scheduled to be completed in four weeks, two of which would be on-site at the client’s location for access to prototype hardware boards and the development team. We planned to start in late January and finish the report by the end of February.
Delays Begin
At the start of the remote assessment element, we reviewed the security of the mobile app and the cloud services. This was straightforward. We identified several vulnerabilities, including weak storage of credentials in the mobile app:
However, several days before the on-site phase, the client informed us that the smart toy’s firmware was not ready for testing. The development team was still integrating the mobile app with the toy, and the firmware was not yet stable enough for testing. We delayed the on-site phase by several weeks to give the team enough time to finish the integration elements.
Weeks later, the customer informed us that there was a problem sourcing some of the smart toy’s hardware components. The development team was working on switching the firmware to a different System on Chip (SoC), setting the project back several months. We cancelled travel and waited for the customer to provide us with a new testing date.
After months of sporadic updates, the client contacted us in August:
“We’re ready for the on-site phase now. Can you come next week?”
Unfortunately, we could not cancel other scheduled work, and the soonest we could come on-site was late September.
Critical Findings, Too Late to Fix
When we arrived on-site, we found that the development team significantly changed the smart toy’s firmware, and we were now testing the actual doll, not the prototype boards. With access to the wireless services, we found that the Bluetooth Low Energy (BLE) services were not properly secured. We connected to the doll without authorization and sent commands to it, including getting the doll to render arbitrary text as speech.
This was a significant finding as it represented a potential child safety concern. An attacker could send commands to the doll to interact with the child, disguised through the text-to-speech rendering in the doll’s voice.
We reported the findings, spoke with the customer about remediation steps, and provided a detailed report of the assessment results. We billed the customer for the time, and the project was closed.
A Year Later…
A year later, I found the same toy in a store, bought it as a memento of the project, and took it home to test it again.
The BLE services were still unsecured, and I was able to connect and send commands to the doll without authorization.
The Case for Continuous Penetration Testing, #1
In this case, the fixed-time assessment failed to provide the desired outcomes. The project was delayed multiple times, which is common in new product development projects. Since we waited for the product to be nearly finished before conducting the pentest assessment, we lost the opportunity for engineering to address any findings the assessment turned up before product readiness.
When we identified high-risk vulnerabilities, the customer had already committed to a production schedule. Changing course meant delaying product delivery until next year’s holiday shopping schedule. So, the customer decided to ship the product anyway, despite the vulnerabilities we identified.
A continuous penetration test would have been a better fit for this project. By choosing four concrete assessment phases and scheduling them to be completed in a fixed time frame with half the project completed off-site and the remainder on-site, we forced the customer to continue delaying until almost the very end of development.
A continuous penetration test would have allowed us to test individual components as they became available. While we might not have seen the complete product until the end of the development effort, firmware testing on prototype boards could have been tested much earlier. This would have allowed us to identify the BLE services vulnerability much earlier in the project and provide feedback earlier so the customer could have fixed the vulnerabilities in time for product readiness.
Case 2: The Assumed Breach Assessment
An assumed breach assessment is a common service offering where we assume that a sophisticated threat adversary has already gained an initial foothold on the network. The goal is to identify whether the organization’s compensating controls limit the attack to just the initial access.
For the consultant, an assumed breach assessment involves extensive testing to bypass endpoint control systems (EDR/XDR, application allowlists, etc.) while trying not to get kicked off the network by the security operations team. The consultant will use various techniques to avoid detection, limiting the number of alerts generated by his or her actions.
One customer approached us for an assumed breach assessment with the following goals:
- Bypass EDR systems
- Evaluate internal network threat detection capabilities
- Measure SOC efficacy for detection
- Move laterally to other internal target systems
- Disable EDR capabilities for reporting and analysis
- Escalate privileges following initial access to administrator-level accounts
- Exercise the incident response team response actions
They even provided us a scatter plot to better visualize the priorities of the engagement.
A Frustrating Engagement
We scheduled the assessment to start after several weeks.
Before starting the assessment, we researched the customer’s EDR solution, installed it locally in our lab, and began to test bypass techniques. We found a few opportunities to bypass detection, including the ability to terminate the endpoint agent from the command line by using a Bring Your Own Vulnerable Driver (BYOVD) attack. Using a signed Microsoft Windows Hardware Compatibility Program (HWCP) driver, we terminated the endpoint agent on our lab systems by targeting the process ID:
Things went well at the beginning of the engagement. We terminated the endpoint agent quickly on initial foothold system we’d been granted access to and started to examine the configuration of the Windows host for privilege escalation opportunities. That is, until we lost access shortly thereafter.
We weren’t sure what happened, so we reconnected and reestablished our foothold on the system. We accessed the system for a short time, then lost it again. This time, we could not reconnect to the remote access system. After changing our IP address through a cloud virtual machine (VM), we reestablished access and attempted to continue our assessment, only to lose access a few minutes later.
The rest of the engagement went similarly. Every time we’d gain access to a target system or attempt to bypass endpoint controls, we’d lose it a short time later. We’d have to re-establish our foothold on the system, only to lose it again.
After the engagement, we met with the customer to discuss our findings.
We were dejected that we didn’t get further in the engagement, but were pleased that the customer’s defense in depth controls were so effective at denying our access.
Then I noticed the director of the security operations team was smiling.
“We made sure we had all hands on deck for the assessment. We cancelled vacations. We authorized overtime. We had the entire team working 24/7 to monitor your actions. We were ready for you,” he said.
The Case for Continuous Penetration Testing, #2
For this customer, the assumed breach assessment was a fixed-time engagement. We had a set number of days to complete the assessment, and we had to work within that time frame to complete the assessment. The customer knew this and engaged every possible resource within the company to stay on top of any actions we took.
This was not a practical representation of what their security operations team could do.
In a continuous penetration test, we could have tested the customer’s defenses over a longer period. This is particularly well-suited to an assumed breach assessment and the goals identified by the customer (see the earlier scatter plot of prioritized goals). By testing the customer’s defenses over a longer period, we can identify the gaps in their defenses that allow us to bypass their controls. We could have also provided the customer with a more realistic view of their security posture, not at a time when they artificially leveraged every possible resource into monitoring our actions, but at the time when they were most likely to be caught off guard. This would have been a much more realistic representation of their security posture, and one that would have been more valuable in helping them identify and mitigate any gaps in their controls.
Case 3: The Project Manager
Scheduling a penetration test for a customer can be challenging.
A financial technology customer hired us to assess their online analytics platform. They had a clear set of testing goals wanted to complete the project before the fall release.
For scheduling, they deferred us to the project manager (PM) overseeing the project activities and milestones.
The Scheduling Nightmare
We agreed to the assessment and scheduled a meeting with the PM. During the call, he showed us the project plan.
The PM asked us where in the project plan we’d like to schedule the assessment. I had a lot of difficulty answering this question since much of the scheduling depended on the goals of the organization. We needed answers to some key questions:
- What platform features were most important to the organization?
- When were those features be ready for testing?
- How much time would the development team need to address any findings we identified?
- Were there regulatory or compliance requirements affecting the project schedule?
- Were there any third-party dependencies that need to be assessed?
- Could we use a dedicated test environment, or would testing impacting development? And is the platform reliable enough for us to use for testing?
The PM was not able to answer these questions and deferred to the development team. The development team didn’t know, so they referred us to the security team. And they didn’t know, so they referred us back to the PM.
We never completed the security assessment.
The Case for Continuous Penetration Testing, #3
In this case, the fixed-time assessment failed to provide the desired outcomes.
While the customer had a clear understanding of the project schedule and milestones, the question of when was daunting. Some wanted us to wait until the end, just before launch, but this would set back development efforts if we identified significant vulnerabilities. Others wanted us to test at a single significant milestone for core project features, but that would not provide the comprehensive assessment results the security team needed.
In a continuous penetration test scenario, we would have worked with the customer to integrate testing into the development cycle, instead of trying to figure out a single point in time. This would have minimized the impact of any findings on the project schedule. With a continuous pentest, we could have provided the customer with a more comprehensive assessment of the project and identified and remediated any findings early in the project development cycle.
The Move to Continuous Penetration Testing
Continuous penetration testing doesn’t work for every scenario, and many organizations will prefer the fixed-time assessment model. However, the move to continuous penetration testing will provide organizations a more comprehensive assessment of their security posture and allow them to identify and remediate findings earlier in the project development cycle.
There are, of course, costs associated with a transition to continuous penetration testing. For penetration testers, scheduling and resource management become more challenging. For customers, the cost of the assessment will likely increase, as it will be more comprehensive and require more time to complete.
These costs must be evaluated against the benefits of continuous penetration testing. However, many organizations find that early integration and continuous penetration testing provide a more valuable outcome, helping them meet regularly requirements, mitigate vulnerabilities earlier, and provide a more comprehensive assessment of the organization’s security posture.
Until that time, I’m going to keep working on my Gantt chart skills.
The shift to continuous penetration testing can dramatically improve your organization's security outcomes, allowing you and your security team to detect and remediate vulnerabilities earlier in the development cycle. If you’re looking to sharpen your offensive and defensive techniques and strengthen your incident response capabilities, SEC504TM: Hacker Tools, Techniques, and Incident HandlingTM delivers the hands-on skills you need.
Register now for SEC504 or sign up for a free demo to see how proactive security testing can strengthen your defenses.
Joshua Wright is the senior technical director for Counter Hack Innovations and a SANS Institute faculty fellow. He has identified and disclosed critical security vulnerabilities for Fortune 500 companies, critical national infrastructure providers, casinos, banks, cloud service providers, smart toys, and more. He is the author of the best-selling SANS course SEC504: Hacker Tools, Techniques, and Incident Handling, and is the dean of students for the SANS Technology Institute. He hopes that someday he’ll be able to write a bio in the third person and not feel like a complete tool while doing so.
