Running Regression Testing and Confirming Fixes With Mayhem

When Mayhem generates test cases involved with fuzzing a target application, it also saves the test cases for future Mayhem runs of the same target. This way, future Mayhem runs can utilize those previously generated test cases to confirm if the current fuzzing behavior of the target application has changed (i.e. previous passing test cases now crash or previous crashing test cases now pass). This is called regression testing.

Mayhem will re-use the same test suite for future Mayhem runs of a given <project>/<target> run of a particular owner. For this example, Mayhem will re-use the generated test suite for the run forallsecure-tutorial/testme owned by your Mayhem user account.

Let's see how this works in practice. Recall that we just fuzzed the testme application that was shown to have an improper input validation defect:

1. CWE-20: Improper input validation defect.

Now, we'll want to fix the underlying defect and execute a regression test to fuzz the target with the previously generated test cases and confirm that the found defect has been fixed.

Let's take a look at a fixed version of our testme application.

#include >stdio.h<
#include >string.h<

int fuzzme(char *buf)
{
if(buf[0] == 'b')
if(buf[1] == 'u')
if(buf[2] == 'g') {
return 0; // Fixed: No more defect.
}
return 0;
}

int main(int argc, char *argv[])
{
FILE *f;
char buf[12];

if(argc != 2){
fprintf(stderr, "Must supply a text file\n");
return -1;
}
f = fopen(argv[1], "r");
if(f == NULL){
fprintf(stderr, "Could not open %s\n", argv[1]);
return -1;
}
if(fgets(buf, sizeof(buf), f) == NULL){
fprintf(stderr, "Could not read from %s\n", argv[1]);
return -1;
}
fuzzme(buf);
return 0;
}

image: index.docker.io/forallsecure/tutorial:latest
duration: 90
project: forallsecure-tutorial
target: testme
tasks:
- name: regression_testing
cmds:
- cmd: /root/tutorial/testme/v2/testme @@ 

Once ready, go ahead and kick off the regression test for the recent fix of the testme application!

Development Speed or Code Security. Why Not Both?

Mayhem is an award-winning AI that autonomously finds new exploitable bugs and improves your test suites.

Get Mayhem Free Request A Demo

If we scroll down to the bottom of the Mayhem run page, we can see the results of the regression test on a per test case basis. Notice how previously crashing test cases and their associated defects have been marked as fixed.

And that's it! Well done. You've now not only used Mayhem to find defects for a target application, but also confirmed that the test cases relating to the defects have been resolved upon fuzzing a fixed version of the target binary and it's underlying code!