Encarsia: Evaluating CPU Fuzzers via Automatic Bug Injection

By Matej Bölcskei, Flavien Solt, Katharina Ceesay-Seitz and Kaveh Razavi
ETH Zurich

Abstract

Hardware fuzzing has recently gained momentum with many discovered bugs in open-source RISC-V CPU designs. Comparing the effectiveness of different hardware fuzzers, however, remains a challenge: each fuzzer optimizes for a different metric and is demonstrated on different CPU designs. Furthermore, the number of newly-discovered bugs is not an appropriate metric since finding new bugs becomes increasingly more difficult as designs mature. We argue that a corpus of automatically injectable bugs will help compare hardware fuzzers to better understand their strengths and weaknesses. Through a large-scale study of 177 software-observable bugs in open-source RISC-V CPUs, we discover that CPU bugs can be modelled by manipulating conditional statements or signal drivers. Based on this observation, we design ENCARSIA, a framework that automatically transforms the intermediate representation of a given CPU design to inject bugs that are equivalent to incorrect conditions or assignments at the HDL level. To ensure that an injected bug has an observable architectural effect, we leverage formal methods to prove the existence of an architectural deviation due to the bug-specific transformation. We evaluate ENCARSIA by injecting bugs into three open-source RISC-V CPUs, fuzzing these CPUs with recently-proposed CPU fuzzers, and comparing their bugfinding performance. Our experiments reveal key insights into the limitations of existing hardware fuzzers, including their inability to cover large sections of the designs under test, ineffective coverage metrics, and bug detection mechanisms that often miss bugs or produce false positives, highlighting the urgent need to reassess current approaches.

To read the full article, click here