Fuzz Testing ROI Framework
A review of software security investments reveals that a majority of spending is in application testing solutions, such as static analysis, software composition analysis, and scanners. These conventional testing approaches, however, test known or common attack patterns, only addressing CVEs or CWEs. But what about the unknown vulnerabilities -- the weaknesses attackers often exploit?
Fuzz testing is a technique where malformed inputs are sent to an application in hopes of triggering anomalous behavior. Anomalous behavior is usually a sign of an underlying vulnerability -- typically a zero-day. Fuzzing is a proven technique that maximizes defect detection with the least amount of time and resources. As a result, it not only buys organizations time and money, it also frees scarce technical resources from manual, mundane tasks and allows them to focus on strategic initiatives that require true expertise.
This framework is a model for framing the way you evaluate the economic return of investing in fuzz testing or other comparable solutions. Organizations can also use this framework to help predict which fuzz testing solutions will offer the most value based on organizational needs.
Fuzz Testing is a Proven Technique for Accurately Uncovering Defects.
Find out how ForAllSecure delivers advance fuzz testing to development pipelines.
We’ll start our analysis by addressing a contentious and multilayered topic: product licensing. Product licensing is an obvious cost, but it is a common misconception that it is the largest cost. Below is a detailed walk through of product cost for each solution.
Manual Penetration Testing Operation Costs
Penetration testing has no direct product license or operational cost. However, we urge readers to consider how service costs can impact your organization’s budget.
Recurring service costs are considered an operational expense (OpEx), while annual product licenses are considered a capital expense (CapEx). Depending on your organization, acquiring OpEx budget may be more challenging than acquiring CapEx budget. The availability of OpEx budget is unpredictable, hinging on company performance or quarterly financial reporting timelines. As a result, you will have to reflect on whether security testing is something you would consider a luxury or necessity.
Protocol Fuzzing Operation Costs
Protocol fuzzers charge on a per protocol basis. Our market research revealed that vendors offer roughly 32 protocols and files in a “standard” offering for decent, mid-level fuzzing.
A critical consideration for those evaluating protocol fuzzers is whether your tool of choice supports your desired protocol or file format. Remember: Engineers from these vendors must manually build the library of test suites based on RFCs. Therefore, test suites for newer or uncommon protocols, such as 5G or Zigbee, are either unavailable or immature.
Organizations that choose to build their own test suite, may find it more costly and even impossible due to lack of technical expertise in the talent market.
Bootstrapped Continuous Fuzzing
Bootstrapping fuzzing is an alluring alternative, because open-source fuzzers, such as AFL, are available free of charge. However, free is never free. Security engineers with ClusterFuzz and OSS-Fuzz have disclosed that while it is possible to bootstrap and operate these high-performance fuzzers in production, people often underestimate the complexity of upstanding such solutions. Their comment echoes what we’ve observed in the market as well. Customers have cited to us that one of the biggest oversights they made was not thinking ahead to the ongoing maintenance cost of such a complicated product.
Several brave ForAllSecure customers have attempted to bootstrapped their own continuous fuzzing solutions. Some were successful in developing a minimum viable product (MVP) that was deployed into their organization. It even gained internal buy-in and traction. Ultimately, they eventually transitioned to ForAllSecure Mayhem because they realized that they had become a development organization for their bootstrapped fuzzing solution -- deploying bug fixes and building new features on an ongoing basis. Eventually, maintenance became a distraction from the larger application security vision for the department.
ForAllSecure Mayhem is priced based on two factors: tier and number of cores. The appropriate tier for you will be determined by the features that you seek. The number of cores you use will be determined by the scale and speed you’d like out of your analysis engine. In short, the more computing power you place behind the fuzzing engine, the more effective your analysis runs will be.
Want to learn more about this? Download the Fuzz Testing ROI Framework white paper.