Mayhem: Software Testing Made Easy
Mayhem automates software testing of binary programs. Mayhem outputs test cases, including those that trigger bugs. Mayhem even checks if bugs can easily be exploited. The result is a set of test cases, and a list of bug reports prioritized by exploitability. Mayhem has found over 13,869 unique bugs in more than 37,000 programs in Debian Linux. Just point, click, and test.
ForAllSecure's mission is to weed out mayhem in software by checking the world's software for bugs using Mayhem. Mayhem addresses several challenges:
- Reproducible test cases. Software evolves, and new versions need to be periodically re-tested. Mayhem outputs 1000's of test cases per run, including those that exploit the program and trigger bugs, so that you can check automatically if new software versions behave the same and fix previously reported bugs.
- Actionable result. Many program analysis tools tell you where potential problems are, but also suffer from high false positive rates. Mayhem is different. Mayhem gives you a reproducible, working input that triggers the bug. Every bug and exploit reported is guaranteed real. Developers can use the input to debug the problem more quickly than with other reports, and end-users can use the input to prove to a vendor there is a problem.
- Prioritize bugs by exploitability. Not all bugs are equal. Bugs that can be exploited by an attacker should be fixed first. Mayhem automatically searches a program for a variety of well-known bugs, such as memory leaks, information leaks, and crashes. For each bug discovered, Mayhem also determines whether the bug can be exploited. The end result is a prioritized list of bugs by exploitability.
- Binary-only code analysis. Mayhem does not require source code, thus end users can now automatically check for bugs. Mayhem provides high-fidelity results based on the code that actually runs, not just what was compiled. But Mayhem isn't just for end-users. Developers can also use Mayhem, and without changing the build process like with other tools.
Automatic Test Case Generation
At the heart of Mayhem is a symbolic execution engine. Symbolic execution automatically explores code paths and generates test cases. Mayhem supplements symbolic execution with path mining to determine if the path can trigger an error condition, and if it can be exploited. Mayhem checks for buffer overflows and command injection attacks, which are among the most devastating in security. Mayhem supplements symbolic execution with fuzzing to find additional problems.
Mayhem automatically generates test cases for your Linux programs. Already have test cases? Mayhem amplifies them by exploring alternate paths.
Mayhem find bugs. Lots of bugs. With just 5 minutes run time per application, Mayhem found more than 5,000 unique bugs in Debian stable.
Mayhem provides real test cases, making every run and every reported bug reproducible. Typical bug-finding tools don't, wasting your time chasing false positives.
It's clear that running Mayhem on the 23,000 or so binaries found in the Debian "Wheezy" repository has found real bugsJake Edge on LWN.net
Thanks for your extensive feedback, it's a pleasure to work with such detailed material (and easy to pin the bug, BTW).Debian Developer
Mayhem on Open Source
At ForAllSecure, we love open source software. As a way to give back to the community, we decided to use Mayhem to improve the robustness and security of open source programs. Mayhem is constantly testing open source software on a 100-node cluster. So far, we have found 13869 distinct crashes on 4943 programs.
In July 2013, after discussing with Debian developers, we submitted a first batch of reports comprising more than a thousand bugs. You can view our reports on the Debian bug tracking system. Developers and package maintainers welcomed our reports. Many of them particularly appreciated our reproducible testcases, mentioning that it made the bug easy to pin-point and fix. We are proud to say that, as of June 17th, 204 bugs were already fixed on Debian Sid, and not a single bug was marked as unreproducible on the Debian bug tracker.
This was the first of many batches of reports. We plan to keep testing open source software, free of charge, to help developers create robust and secure software.
Find out about using Mayhem on your applications today!