OpenTitan System Tests
System tests are end-to-end tests for the whole OpenTitan system. They operate
on build outputs (typically in
build-bin) and can be used to check that the
full system, consisting of software, hardware, and tooling, works as expected.
Tests can be executed against different target platforms (such as FPGAs or simulations), even though not all OpenTitan-supported targets are supported at the moment.
earlgrey/test_sim_verilator.py: Run various software tests against a Verilator-built simulation of the Earl Grey design.
earlgrey/test_fpga_nexysvideo.py: Run various software tests against the Earl Grey design running on a Nexys Video FPGA board.
Run the tests
Before the tests can be run, the various components of OpenTitan need to be
built and present in the
BIN_DIR directory (typically
Alternatively, the tests can be executed against any pre-built OpenTitan snapshot available from GitHub releases, or from CI.
To test build artifacts from CI follow these steps:
Go to an individual build job in the “CI” pipelines of Azure Pipelines. (The page where the individual steps for a single change are listed, not the job overview page.)
In the information bar on the top click on a link labeled “5 published” (or something similar). It will take you to a page titeled “Artifacts”.
Expand the “opentitan-dist” artifact, and download the file
Untar this file and set the
BIN_DIRenvironment variable to point to the extracted directory, for example:
tar -xf opentitan-snapshot-20191101-1-2462-g3ad4fd1b.tar.xz export BIN_DIR=$PWD/opentitan-snapshot-20191101-1-2462-g3ad4fd1b
Tests running on FPGA boards need some configuration to help with the discovery
of things like the UART port. To configure these settings, copy the
configuration file example from
to one of the supported configuration file locations:
The pytest documentation gives a good overview of the available options to run tests in the Usage and Invocations chapter. Here’s a quick reference.
# Run all tests pytest test/systemtest # Run tests in a specific file pytest test/systemtest/earlgrey/test_sim_verilator.py # Run a specific test in a specific parametrization pytest test/systemtest/earlgrey/test_sim_verilator.py -k test_apps_selfchecking[usbdev_test] # Run tests with more verbose output pytest test/systemtest -sv --log-cli-level=DEBUG