You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
150 lines
4.2 KiB
150 lines
4.2 KiB
# Running device tests
|
|
|
|
## 1. Running the full test suite
|
|
|
|
_Note: You need Poetry, as mentioned in the core's [documentation](https://docs.trezor.io/trezor-firmware/core/) section._
|
|
|
|
In the `trezor-firmware` checkout, in the root of the monorepo, install the environment:
|
|
|
|
```sh
|
|
poetry install
|
|
```
|
|
|
|
And run the automated tests:
|
|
|
|
```sh
|
|
poetry run make -C core test_emu
|
|
```
|
|
|
|
## 2. Running tests manually
|
|
|
|
Install the poetry environment as outlined above. Then switch to a shell inside the
|
|
environment:
|
|
|
|
```sh
|
|
poetry shell
|
|
```
|
|
|
|
If you want to test against the emulator, run it in a separate terminal:
|
|
```sh
|
|
./core/emu.py
|
|
```
|
|
|
|
Now you can run the test suite with `pytest` from the root directory:
|
|
```sh
|
|
pytest tests/device_tests
|
|
```
|
|
|
|
### Useful Tips
|
|
|
|
The tests are randomized using the [pytest-random-order] plugin. The random seed is printed in the header of the tests output, in case you need to run the tests in the same order.
|
|
|
|
If you only want to run a particular test, pick it with `-k <keyword>` or `-m <marker>`:
|
|
|
|
```sh
|
|
pytest -k nem # only runs tests that have "nem" in the name
|
|
pytest -k "nem or stellar" # only runs tests that have "nem" or "stellar" in the name
|
|
pytest -m stellar # only runs tests marked with @pytest.mark.stellar
|
|
```
|
|
|
|
If you want to see debugging information and protocol dumps, run with `-v`.
|
|
|
|
Print statements from testing files are not shown by default. To enable them, use `-s` flag.
|
|
|
|
If you would like to interact with the device (i.e. press the buttons yourself), just prefix pytest with `INTERACT=1`:
|
|
|
|
```sh
|
|
INTERACT=1 pytest tests/device_tests
|
|
```
|
|
|
|
When testing transaction signing, there is an option to check transaction hashes on-chain using Blockbook. It is chosen by setting `CHECK_ON_CHAIN=1` environment variable before running the tests.
|
|
|
|
```sh
|
|
CHECK_ON_CHAIN=1 pytest tests/device_tests
|
|
```
|
|
|
|
To run the tests quicker, spawn the emulator with disabled animations using `-a` flag.
|
|
|
|
```sh
|
|
./core/emu.py -a
|
|
```
|
|
|
|
To run the tests even quicker, the emulator should come from a frozen build. (However, then changes to python code files are not reflected in emulator, one needs to build it again each time.)
|
|
|
|
```sh
|
|
PYOPT=0 make build_unix_frozen
|
|
```
|
|
|
|
It is possible to specify the timeout for each test in seconds, using `PYTEST_TIMEOUT` env variable.
|
|
```sh
|
|
PYTEST_TIMEOUT=15 pytest tests/device_tests
|
|
```
|
|
|
|
When running tests from Makefile target, it is possible to specify `TESTOPTS` env variable with testing options, as if pytest would be called normally.
|
|
|
|
```sh
|
|
TESTOPTS="-x -v -k test_msg_backup_device.py" make test_emu
|
|
```
|
|
|
|
When troubleshooting an unstable test that is failing occasionally, following runs it until it fails (so failure is visible on screen):
|
|
|
|
```sh
|
|
export TESTOPTS="-x -v -k test_msg_backup_device.py"
|
|
while make test_emu; do sleep 1; done
|
|
```
|
|
|
|
## 3. Using markers
|
|
|
|
When you're developing a new currency, you should mark all tests that belong to that
|
|
currency. For example, if your currency is called NewCoin, your device tests should have
|
|
the following marker:
|
|
|
|
```python
|
|
@pytest.mark.newcoin
|
|
```
|
|
|
|
This marker must be registered in `REGISTERED_MARKERS` file in `tests` folder.
|
|
|
|
Tests can be run only for specific models - it is done by disallowing the tests for the other models.
|
|
`@pytest.mark.skip_t1`
|
|
`@pytest.mark.skip_t2`
|
|
`@pytest.mark.skip_tr`
|
|
`@pytest.mark.skip_t3t1`
|
|
are valid markers to skip current test for T1, TT and TR respectively.
|
|
|
|
[pytest-random-order]: https://pypi.org/project/pytest-random-order/
|
|
|
|
## Extended testing and debugging
|
|
|
|
### Building for debugging (Emulator only)
|
|
|
|
Build the debuggable unix binary so you can attach the gdb or lldb.
|
|
This removes optimizations and reduces address space randomizaiton.
|
|
|
|
```sh
|
|
make build_unix_debug
|
|
```
|
|
|
|
The final executable is significantly slower due to ASAN(Address Sanitizer) integration.
|
|
If you want to catch some memory errors use this.
|
|
|
|
```sh
|
|
time ASAN_OPTIONS=verbosity=1:detect_invalid_pointer_pairs=1:strict_init_order=true:strict_string_checks=true TREZOR_PROFILE="" poetry run make test_emu
|
|
```
|
|
|
|
### Coverage (Emulator only)
|
|
|
|
Get the Python code coverage report.
|
|
|
|
If you want to get HTML/console summary output you need to install the __coverage.py__ tool.
|
|
|
|
```sh
|
|
pip3 install coverage
|
|
```
|
|
|
|
Run the tests with coverage output.
|
|
|
|
```sh
|
|
make build_unix && make coverage
|
|
```
|