ForAllSecure Blog

How Does Mayhem Work?

Once the runtime environment is provided, Mayhem will run your application and automatically perform the following tasks:

1. Exploitability Factors: Checks to sees if the application is hardened or protected against exploitation.
2. Fuzzing and Symbolic Execution: Tests the target application with different input permutations and monitors for any abnormal behaviors/defects.
3. Triage and Diagnosis: Validate discovered defects and provide coverage analysis.
4. Automated Regression Testing: Validate current results against previous results with saved test suites.

Exploitability Factors

When Mayhem tests a target application, it checks to see if your compiled application is protected or "hardened". A hardened application is one in which its vulnerabilities–-both known and unknown–-are harder to exploit. The developer can harden an application by choosing whether to enable certain compile-time flags.

The current best practices or exploitability factors for hardening an application include:

• ASLR: Address Space Layout Randomization (ASLR) makes it hard for an attacker to predict what memory will look like at runtime, making it harder to reliably exploit a system. ASLR is an operating system feature that is only available if the application was compiled as a “Position Independent Executable” (PIE).
• FORTIFY_SOURCE: The FORTIFY_SOURCE macro provides lightweight support for detecting buffer overflows in various functions that perform operations on memory and strings.
• StackGuard: StackGuard inserts a stack canary into the stack during function entry and checks their value prior to function exit, terminating execution if the value has been corrupted. StackGuard is used to detect stack-based buffer overflow exploits.
• DEP: Data Execution Prevention (DEP) is a set of technologies that perform additional checks on memory to help prevent malicious code from running on your system.

An application will be significantly more difficult to exploit by an attacker if all four hardening techniques are enabled.

Fuzzing and Symbolic Execution

Traditionally, fuzzing was just about feeding random input into an application to test for abnormal behavior—similar to a million monkeys randomly typing on a keyboard. In fact, back in 1998, a professor at the University of Wisconsin named Bart Miller noticed that 33% of UNIX applications would crash if given random input—just like what a monkey would type!

Traditional fuzzing is like a monkey typing on a keyboard.

Mayhem is smart, efficient, and automatically tests your apps with the latest fuzzing and symbolic execution technology.

Mayhem comes with both instrumentation-guided fuzzing and symbolic execution. The two techniques have different performance curves, but are complementary and provide the most advanced form of Behavior Testing. Behavior testing is testing that runs an application with different inputs and observes how program flow, code coverage, and application termination vary as the input changes. Abnormal observations are triaged to determine if a behavior is a vulnerability.

Instrumentation-guided fuzzing is much more intelligent – and efficient – than randomly generating inputs. The key idea is to add instrumentation that analyzes a program as it executes, and use that knowledge to intelligently pick a new input.

Symbolic execution is even more advanced. Symbolic execution is like program verification, but with a twist. Instead of verifying a program is secure, we verify it is insecure by producing a Proof of Defect. Symbolic execution models each program execution in formal logic, and then uses a mathematical solver to explore each behavior of the program.

Triage and Diagnosis

If a program crashes, you know something is wrong, but likely don’t know why. Mayhem helps provide you more insight by automatically performing triage and diagnosis to give you more information about each defect.

For each defect, Mayhem will provide you:

• A backtrace. The backtrace identifies which functions were called, helping security practitioners and developers more quickly identify the root cause of a problem.
• The register and memory state at time of crash. This information can be used to provide context regarding the crash to see if the conditions are the same as that of another crash.

Automated Regression Testing

When Mayhem tests a target application, Mayhem generates test cases where each test case represents unique code coverage for the underlying target. Therefore when a defect is found, a unique test case represents the particular defect and can be re-run to reproduce the error.

As a user, you may want to know when defects are fixed. Thus, Mayhem saves these test cases to then automatically replay the stored test suite and let you know the number of defects fixed compared to previous runs. This is automated regression testing.