Regression testing with random tests cannot identify regressions
Daniel Hansson, CEO, Verifyter
EETimes (10/15/2012 10:47 AM EDT)
Abstract
Most ASIC companies use random tests not only to verify new designs but also for regression testing. Using random tests for regression testing is a great idea for coverage as the randomness over time will ensure that the total coverage will improve. Instead of running the same tests every night, each night’s regression test suite is slightly different with different seeds. However improving coverage is not what the specific topic of regression testing is about. The purpose of regression testing is to quickly identify dips in quality, i.e. regressions, in order to address them and keep the quality high. And here random tests have one downside – they cannot identify regressions. But there are ways to address this issue.
Is it better or worse?
To be more precise, random tests cannot distinguish between a dip in quality and increased coverage. A random test that fails may do so because it hit a new and never before tested corner case which reveals a bug in a module that was designed by professors and PhD’s long time ago in a completely different project. It is great news to stumble upon such a corner case in order to iron it out, hopefully before the customer will notice it. Alternatively the random test may fail because John accidentally sat on his keyboard while checking in his code update (he is very agile). This caused some unexpected behavior in same functions he was not even working on (sitting on keyboards often do). This is a classic case of a regression. In the first case you have great news to report, an old corner case has been identified, you are a hero. In the second case, you have to hit the panic button and hold the release. Distinguishing between good and bad news is always welcome, not only in the world of regression testing, but alas random tests cannot help you with this. The random test just tells you that something failed, but cannot say whether it is a new or old problem.
To read the full article, click here
Related Semiconductor IP
- nQrux® Root of Trust IP
- AXI to UCIe Bridge IP
- UCIe 2.x Controller IP
- SWI3S (SoundWire I3S Interface) Peripheral Controller Core IP
- OpenTitan-based RISC-V Secure Element
Related Articles
- Methodology Independent Exhaustive Constraint Solver for Random Verification and Regression Generation
- Creating SoC Integration Tests with Portable Stimulus and UVM Register Models
- Embedded Software Unit Testing with Ceedling
- QA Automation Testing with Container and Jenkins CICD
Latest Articles
- A Secure dToF LiDAR SoC with Dual-Domain Fingerprinting and Event-Driven AFE Circuit Achieving Sensor-Level Attack Resilience
- ZTA-Q: an Open-source RISC-V Platform for Accurate Quantized CNN Inference
- Automated Pre-Silicon Verification of High-Speed DDR5 and LPDDR5/6 Memory Controllers: Closed-Loop Timing, Mode Register, and PHY Synchronization in UVM
- U-Sonic: An Open-Source 8-Channel Ultrasound Transmit IP in a 130 nm RISC-V SoC
- S-ALSA: Co-Design of Adiabatic Logic-based Sensing and Balanced Bit-Cells for Secure and Energy-Efficient MRAM