Casting a wide safety net through post processing Checking

By Purushottham R, Nigitha A, Jayashree R, Sharan Basappa (HCL Technologies)

Introduction

Post processing stage is a critical part of any verification project. The post processing step for catching errors is done for multiple reasons some of which are highlighted below,

• Look for standard error signatures (e.g. UVM_ERROR) in the log files
• Look for error signatures that cannot be logged through UVM reporting (e.g. third party components that may not strictly follow UVM reporting mechanism etc.)
• Ignore error signatures globally (e.g. known errors that need temporary or permanent bypass)
• Ignore error signatures specific to a file (or a specific test case that is expected to create error scenarios)

If you notice the above cases, the post processing is done mainly to parse for error patterns. However, there is another aspect to the post processing, which is generally not understood well, but it can play a critical role in the verification cycle. That is, making sure that, signatures that are expected to be present in the log files, are actually present. While the former case results in error when a signature is found, in the latter case, it is treated as an error if an expected signature is not found.

A few examples of this type of post processing checks that are needed are listed below,

• The tests have gone through the expected stages/events so that we don’t end up with false positive
• Scoreboard is disabled when they are not expected to
• Assertions are disabled when they are not expected to
• Certain testbench checks are disabled when they are not expected to
• Events like interrupts have occurred in the expected tests

A typical DV project involves a lot of engineers making simultaneous code updates. Such a collaborative work can result in a lot of inadvertent errors. To handle such common errors, the project team will do review checks at the end of every simulation or a regression run to ensure that any oversight errors have not been introduced. However, this process of review checks itself is manual and hence, is not only effort intensive but is prone to errors.

In this paper, we explain how we adopted the post processing method to make sure we check for signatures that are expected are present in the log files. This acted as a safety net when engineers made inadvertent mistakes and introduced issues in the code base.

Related work

While we are aware of post processing approach to catch error signatures in the log files, we haven’t come across using the post processing methods that look for expected signatures are present in the log files. This is normally done manually by reviewing log files and if needed, additionally reviewing the waveforms.

One of the shortcomings of the prevailing manual method to check that the expected signatures are indeed present. While this approach does work, it has few shortcomings,

• High effort
• The more the number of regression sessions, the more the effort
• Effort increases with increased log files count and hence scaling is an issue

Implementation Approach

The post processing was deployed using Python script. The script accepted JSON file as an input. The JSON file contains a list of rules.

The rules are nothing but signature patterns that need to be parsed in the log files. Some of the key features of the checker parser are captured below,

Sample rule file is shown in the snapshot below.

The key python modules used are,

• Re – regular expression module
• Os – operating system module
• Json – json related operations
• Logging – logger module

Results

For the project where we deployed this method, we had a total of 100+ rules for signatures. The 100 odd rules themselves can be broadly classified into the following categories:

There were several issues that were found. A sample of issues are given below,

• Many tests did not clear the interrupts as a part of ISR servicing
• Tests did not generate the interrupt as they were masked at the IP
• Due to DUT change and testbench being out of sync, the DUT initialization was not happening properly

Final thoughts

Traditionally, we use post processing to check for error signatures in the log files. We don’t use post processing to check for signatures that we expect to be present. We also use, in certain cases, manual method to review the log files.

Our proposed approach brought a lot of value due to the following,

• Check certain signatures that could not be checked directly in the testbench environment
• Make sure that there are no oversight issues and hence get better confidence in terms of quality of regression runs
• Act as an additional checker to ensure that the expected test intention was met
• Reduce a significant effort by reducing manual review