Automated detection of vulnerabilities in software

Applying technology developed in-house, IT security experts from Bochum have identified hundreds of bugs in critical software, including operating systems such as Windows and Linux. Clouds, too, are affected.

Programming errors in software occur fairly frequently; attackers routinely use such vulnerabilities to control the program in question. In recent years, the team at the Chair for Systems Security at RUB has been developing tools for automated detection of vulnerabilities in program code. “We are using so-called fuzzing for this purpose – a technique that has been known since the 1980s and has been immensely improved in the recent years,” explains Professor Thorsten Holz, who heads the Chair for Systems Security. “We and other research groups, as well as the industry, have optimised it significantly.”

Crashes indicate programming errors

Fuzzers are algorithms that feed the tested software with random inputs and monitor if the application crashes as a result. “The fuzzer provides a specific input to the program and tracks which code elements are executed,” describes Holz. Subsequently, it varies the input slightly and monitors if any new regions of the code are uncovered. In the process, the purpose of the fuzzer is to execute as much of the program code as possible and, ultimately, to test all parts of the program in order to check if a specific input causes them to crash. “Such crashes indicate programming errors,” says Thorsten Holz. “After detecting vulnerabilities with this method, we verify in the next step if they are security-sensitive.”

Together with his PhD students Sergej Schumilo and Cornelius Aschermann, the researcher has developed a number of fuzzers in the recent years. The tested programs include programming languages, operating systems and even virtualisation solutions that are indispensable for cloud computing. The researchers detected errors in almost all tested applications and classified 60 of the hundreds of identified bugs as security-critical. “The fuzzer has proved so successful when it comes to finding bugs that we had to employ student assistants to report all errors to the developers,” elaborates Holz.

At present, the Bochum-based team is developing fuzzers for so-called hypervisors, i.e. programs that generate virtual environments on a hardware, in which several operating systems can be simultaneously installed and which are essential for the security of modern cloud systems. “We have found various security gaps that attackers may potentially use to take over the cloud provider’s entire infrastructure,” concludes Thorsten Holz. “They would, for example, enable an attacker to seize control not only of the software, but also of the hardware for which the virtual environments are generated.”

Start-up planned

“Fuzzing has been extensively used by software developers for a while now, and it will definitely constitute the foundation of software quality management in future,” points out Thorsten Holz. Accordingly, enterprises are very much interested in new, efficient fuzzers like the ones developed by Sergej Schumilo and Cornelius Aschermann in their PhD projects. Both are now planning to launch a start-up company specialised in security solutions for critical software systems, supported by the Cube 5 start-up incubator at RUB.

Protection mechanisms against fuzzing

At the same time, research at the Chair for Systems Security continues. The team is currently developing protection mechanisms to prevent software fuzzing. “Enterprises often use fuzzing to test their own software in-house with regard to vulnerabilities and to eliminate them,” explains Holz. “But they want to stop attackers from using the same method to detect and exploit their vulnerabilities.” A tool supplied by the researchers might prevent unauthorised fuzzing in future.