Mayhem: Software Testing Made Easy

Mayhem automates software testing of binary programs. Mayhem outputs test cases, including those that trigger bugs. Mayhem even checks if bugs can easily be exploited. The result is a set of test cases, and a list of bug reports prioritized by exploitability. Mayhem has found over 13,869 unique bugs in more than 37,000 programs in Debian Linux. Just point, click, and test.

Software Testing Picture

ForAllSecure's mission is to weed out mayhem in software by checking the world's software for bugs using Mayhem. Mayhem addresses several challenges:

  • Reproducible test cases. Software evolves, and new versions need to be periodically re-tested. Mayhem outputs 1000's of test cases per run, including those that exploit the program and trigger bugs, so that you can check automatically if new software versions behave the same and fix previously reported bugs.
  • Actionable result. Many program analysis tools tell you where potential problems are, but also suffer from high false positive rates. Mayhem is different. Mayhem gives you a reproducible, working input that triggers the bug. Every bug and exploit reported is guaranteed real. Developers can use the input to debug the problem more quickly than with other reports, and end-users can use the input to prove to a vendor there is a problem.
  • Prioritize bugs by exploitability. Not all bugs are equal. Bugs that can be exploited by an attacker should be fixed first. Mayhem automatically searches a program for a variety of well-known bugs, such as memory leaks, information leaks, and crashes. For each bug discovered, Mayhem also determines whether the bug can be exploited. The end result is a prioritized list of bugs by exploitability.
  • Binary-only code analysis. Mayhem does not require source code, thus end users can now automatically check for bugs. Mayhem provides high-fidelity results based on the code that actually runs, not just what was compiled. But Mayhem isn't just for end-users. Developers can also use Mayhem, and without changing the build process like with other tools.

Case Study: Testing Debian

In June 2013, we downloaded 22,678 programs available through Debian Linux stable's package management system and ran Mayhem for only 5 minutes per program. Mayhem generated over 5.5 million testcases automatically and found 4,801 bugs. Invariably longer runs would have found more bugs. Mayhem had no knowledge of the applications, such as the proper command line, file inputs, or other options. We simply pointed Mayhem at the programs, and Mayhem found the new bugs missed by previous human and automatic analysis. The result was actionable information to Linux developers to identify and help fix problems.

At the heart of Mayhem is a symbolic execution engine. Symbolic execution automatically explores code paths and generates test cases. Mayhem supplements symbolic execution with path mining to determine if the path can trigger an error condition, and if it can be exploited. Mayhem checks for buffer overflows and command injection attacks, which are among the most devastating in security. Mayhem supplements symbolic execution with fuzzing to find additional problems.

Automated Testing

Mayhem automatically generates test cases for your Linux programs. Already have test cases? Mayhem amplifies them by exploring alternate paths.

Find Bugs

Mayhem find bugs. Lots of bugs. With just 5 minutes run time per application, Mayhem found more than 5,000 unique bugs in Debian stable.

Actionable Results

Mayhem provides real test cases, making every run and every reported bug reproducible. Typical bug-finding tools don't, wasting your time chasing false positives.

Easy Setup

Just point Mayhem at your application and run. It is that easy. Want more? Mayhem can also be fine-tuned to your application.

It's clear that running Mayhem on the 23,000 or so binaries found in the Debian "Wheezy" repository has found real bugs

Jake Edge on LWN.net

Thanks for your extensive feedback, it's a pleasure to work with such detailed material (and easy to pin the bug, BTW).

Debian Developer

At ForAllSecure, we love open source software. As a way to give back to the community, we decided to use Mayhem to improve the robustness and security of open source programs. Mayhem is constantly testing open source software on a 100-node cluster. So far, we have found 13869 distinct crashes on 4943 programs.

In July 2013, after discussing with Debian developers, we submitted a first batch of reports comprising more than a thousand bugs. You can view our reports on the Debian bug tracking system. Developers and package maintainers welcomed our reports. Many of them particularly appreciated our reproducible testcases, mentioning that it made the bug easy to pin-point and fix. We are proud to say that, as of June 17th, 204 bugs were already fixed on Debian Sid, and not a single bug was marked as unreproducible on the Debian bug tracker.

This was the first of many batches of reports. We plan to keep testing open source software, free of charge, to help developers create robust and secure software.

Find out about using Mayhem on your applications today!