ForAllSecure Blog

Reproducing Generated Test Case Crashes in Mayhem

Whenever Mayhem finds a crash, click on the test case ID to see more detail about the crashing test cases. The Mayhem UI should reveal a screen like so:

Here we can gain further insight into the behavior of the testme binary as a result of the particular input test case via the following tabs:

1. Defects: Shows a full list of defects exhibited by the particular test case.
2. Triage: Reveals metadata generated from triage for the particular test case (console output, backtrace, disassembly, register slate, signal number, memory maps).
3. Advanced Triage: Lists runtime errors as a result of performing advanced triage for the particular test case.

Mayhem also provides the necessary commands to re-execute the testme binary with the given input test case to reproduce the logged behavior.

 ./root/tutorial/testme/v1/testme <TESTCASE_ID>

 

Therefore, if we were to download and extract the testme binary as well as the crashing test case, and then execute the commands to reproduce the defect, we should see something similar to the following:

 ~/tutorial/testme/v1# ./testme b86204ef1dc8cfc10654e1fe7b1d1c241ef7a0b1da67430b9865c328499d13e7 
Aborted

 

Info. Here we provide the testme binary for download, which requires a Linux OS to execute. If you have Docker installed, you can fuzz the containerized testme target within the Docker container itself. We will show you how to do this in the next lesson on CLI Fuzzing.

And there it is! We can see that the defect has been reproduced for the given input test case. Now, let's fix the underlying code, re-compile the testme binary, and execute another Mayhem run to confirm that the defect (and its associated test cases) are fixed!

Running Regression Testing and Confirming Fixes

When Mayhem generates test cases involved with fuzzing a target application, it also saves the test cases for future Mayhem runs of the same target. This way, future Mayhem runs can utilize those previously generated test cases to confirm if the current fuzzing behavior of the target application has changed (i.e. previous passing test cases now crash or previous crashing test cases now pass). This is called regression testing.

Info

Mayhem will re-use the same test suite for future Mayhem runs of a given <project>/<target> run of a particular owner. For this example, Mayhem will re-use the generated test suite for the run forallsecure-tutorial/testme owned by your Mayhem user account.

Let's see how this works in practice. Recall that we just fuzzed the testme application that was shown to have an improper input validation defect:

1. CWE-20: Improper input validation defect.

Now, we'll want to fix the underlying defect and execute a regression test to fuzz the target with the previously generated test cases and confirm that the found defect has been fixed.

Let's take a look at a fixed version of our testme application.

#include <stdio.h>
#include <string.h>

int fuzzme(char *buf)
{
  if(buf[0] == 'b')
    if(buf[1] == 'u')
      if(buf[2] == 'g') {
        return 0; // Fixed: No more defect.
      }
  return 0;
}

int main(int argc, char *argv[])
{
  FILE *f;
  char buf[12];

  if(argc != 2){
    fprintf(stderr, "Must supply a text file\n");
    return -1;
  }
  f = fopen(argv[1], "r");
  if(f == NULL){
    fprintf(stderr, "Could not open %s\n", argv[1]);
    return -1;
  }
  if(fgets(buf, sizeof(buf), f) == NULL){
    fprintf(stderr, "Could not read from %s\n", argv[1]);
    return -1;
  }
  fuzzme(buf);
  return 0;
} 

Simply changing the abort(); to return 0; on line 9 should be enough to fix the improper input validation defect.

Within the create new run flow, click the Show Mayhemfile link at the bottom of the page to confirm that your Mayhemfile looks similar to the following:

 image: index.docker.io/forallsecure/tutorial:latest
duration: 90
project: forallsecure-tutorial
target: testme
tasks:
  - name: regression_testing
cmds:
  - cmd: /root/tutorial/testme/v2/testme @@

Once ready, go ahead and kick off the regression test for the recent fix of the testme application!

If we scroll down to the bottom of the Mayhem run page, we can see the results of the regression test on a per test case basis. Notice how previously crashing test cases and their associated defects have been marked as fixed.

And that's it! Well done. You've now not only used Mayhem to find defects for a target application, but also confirmed that the test cases relating to the defects have been resolved upon fuzzing a fixed version of the target binary and it's underlying code!