In late March 2014, two teams of security researchers independently started fuzz testing OpenSSL, an open source utility that encrypts traffic from a web browser to a server and forms the basis of trusted transactions online. On April 1, Neel Mehta of Google disclosed (privately) an exploitable vulnerability to OpenSSL that would be independently discovered and confirmed on April 3 by a Finnish company, Codenomicon (now Synopsys). On April 8, 2014, the Heartbleed vulnerability was announced publicly for the first time with a unique name, a logo, and website, and more importantly a patched version of OpenSSL.
When more than one researcher finds the same vulnerability at approximately the same time it’s known as a “collision.” To test their fuzzing engines, security vendors will randomly choose from open source tools in use today. Given that OpenSSL is open source and widely used, it’s not too surprising that two independent fuzz testing organizations happened to choose it in March of 2014.
Both teams of researchers reported the details to the vendor first. In the case of Codenomicon, they contacted the Finnish CERT which helped them get in touch with OpenSSL. Neel Mehta of Google contacted OpenSSL directly. "I was doing laborious auditing of OpenSSL, going through the [Secure Sockets Layer] stack line by line," Mehta told the podcast Risky Biz, adding that he hadn't spoken about it until now because it made him "nervous".
Metha told Sydney Morning Herald that the "main reason" he went looking through the SSL stack was because of some of the other encryption bugs found earlier that year, including the GoToFail security bug, found in February, and the GnuTLS bug, found in March. "You sort of sense that the discovery of flaws in SSL stacks is accelerating,” he said, “and so I was curious about [what] the current state of security was with regards to SSL stacks and I went and took a look..."
On the other side of the world, Rauli Kaksonen, now Senior Information Security Specialist at University of Oulu, said Codenomicon in Finland had been designing a new test suite for its protocol fuzzer when they discovered the flaw. He describes the events leading up to it in this 2017 podcast.
The whole idea of responsible disclosure is to inform the vendor of the problem and give that vendor a chance to fix the vulnerability before it was disclosed publicly. In this case, OpenSSL was used on nearly 600,000 servers worldwide, including financial and retail sites. Had either team gone public without a fix in place, that might have been a disaster.
The Heartbeat Problem
When an OpenSSL session is initiated, there is also a heartbeat established between the browser and the server. This heartbeat is to let each other know that they’re still connected. The idea is for the browser to request some simple data to be exchanged.
The simplest explanation is that the client asks the server for four characters, bird. The server then sends back four characters, bird. The spec for OpenSSL says this can be up to 56 bytes in length. However, the spec didn’t put in an upper bound, and since, according to RFC 6520, a Heartbeat response needs to contain the exact copy of the payload from the request a bad actor could request, say, 500 bytes in which case the server would scramble to find 500 available bytes in response.
Where would that information come from? It might already be in memory, so now the server might be returning to the requesting client passwords, keys, even personally identifiable information (PII) that have been recently processed. That was the danger of Heartbleed, that there would severe data leakage as a result.
Researchers were able to trace the leak back to an update two years earlier. In 2012, a change was made to OpenSSL-- that change allowed for Heartbleed to exist. The vulnerability is only present in OpenSSL versions 1.0.1 through 1.0.1f and only if -DOPENSSL_NO_HEARTBEATS option is not used. In those two years in which OpenSSL was vulnerable, it is unknown whether anyone knew it existed. If someone did, they could request a large amount of private data to be returned via the heartbeat feature of OpenSSL.
How Heartbleed was Found
That’s where fuzz testing becomes valuable in exposing hidden vulnerabilities. By providing invalid input continuously for several hours, flaws in the software can be revealed. Usually, it results in a crash. However the teams of researchers from Google and Codenomicon each began to notice not a crash, but anomalous behavior. A crash is a crash and usually with a fuzz test, a crash might be indicative of a vulnerability and therefore bears more research. In this case, the anomalous did not crash the application but its response nonetheless caught the attention of the two teams -- more data than expected was being returned.
The disclosure of Heartbleed by Google and Codenomicon brought new attention to the sophisticated security testing technique. Since the discovery of Heartbleed, fuzz testing has increasingly become the trusted technique for time- and cost-effectively discovering unknown vulnerabilities, allowing organizations to build security into software before deployment.
Saying CVE-2014-0160 over and over is not very sticky, so Codenomicon realized it needed to get the information out fast. One of the researchers in Finland dubbed it Heartbleed (because it leaked private information via the heartbeat feature), and someone in the design department came up with a cute logo. The company also registered the domain.
Metha told the the Sydney Morning Herald that the use of the name and logo was a good idea. "I was a little surprised by the reaction," he said. "I think a lot of that had to do with a lot of the marketing around the bug that [security firm] Codenomicon did. A logo and a great name goes a long way; I mean I didn't have any of those."
Devices Still Infected Today
In 2017, three years after disclosure, independent security researcher Billy Rios downgraded the number of vulnerable systems to 250 thousand. While that marked an improvement, that number has been very slow to decline further. Why?
Rios concluded that these remaining system were most likely embedded devices out in the field that were either physically hard to reach or otherwise impossible to update and would simply have to be replaced over time. This list includes routers and gateways as well as ICS and SADA systems.
Rios told Bank InfoSecurity, “Given the fact that Heartbleed is probably one of the most well-known vulnerabilities ever ... I'm actually a little surprised that folks who own this infrastructure do not realize that they're running something on the internet that's vulnerable to such a bug,"