# Running device tests ## 1. Running the full test suite _Note: You need Poetry, as mentioned in the core's [documentation](https://docs.trezor.io/trezor-firmware/core/) section._ In the `trezor-firmware` checkout, in the root of the monorepo, install the environment: ```sh poetry install ``` And run the automated tests: ```sh poetry run make -C core test_emu ``` ## 2. Running tests manually Install the poetry environment as outlined above. Then switch to a shell inside the environment: ```sh poetry shell ``` If you want to test against the emulator, run it in a separate terminal: ```sh ./core/emu.py ``` Now you can run the test suite with `pytest` from the root directory: ```sh pytest tests/device_tests ``` ### Useful Tips The tests are randomized using the [pytest-random-order] plugin. The random seed is printed in the header of the tests output, in case you need to run the tests in the same order. If you only want to run a particular test, pick it with `-k <keyword>` or `-m <marker>`: ```sh pytest -k nem # only runs tests that have "nem" in the name pytest -k "nem or stellar" # only runs tests that have "nem" or "stellar" in the name pytest -m stellar # only runs tests marked with @pytest.mark.stellar ``` If you want to see debugging information and protocol dumps, run with `-v`. Print statements from testing files are not shown by default. To enable them, use `-s` flag. If you would like to interact with the device (i.e. press the buttons yourself), just prefix pytest with `INTERACT=1`: ```sh INTERACT=1 pytest tests/device_tests ``` When testing transaction signing, there is an option to check transaction hashes on-chain using Blockbook. It is chosen by setting `CHECK_ON_CHAIN=1` environment variable before running the tests. ```sh CHECK_ON_CHAIN=1 pytest tests/device_tests ``` To run the tests quicker, spawn the emulator with disabled animations using `-a` flag. ```sh ./core/emu.py -a ``` To run the tests even quicker, the emulator should come from a frozen build. (However, then changes to python code files are not reflected in emulator, one needs to build it again each time.) ```sh PYOPT=0 make build_unix_frozen ``` It is possible to specify the timeout for each test in seconds, using `PYTEST_TIMEOUT` env variable. ```sh PYTEST_TIMEOUT=15 pytest tests/device_tests ``` When running tests from Makefile target, it is possible to specify `TESTOPTS` env variable with testing options, as if pytest would be called normally. ```sh TESTOPTS="-x -v -k test_msg_backup_device.py" make test_emu ``` When troubleshooting an unstable test that is failing occasionally, following runs it until it fails (so failure is visible on screen): ```sh export TESTOPTS="-x -v -k test_msg_backup_device.py" while make test_emu; do sleep 1; done ``` ## 3. Using markers When you're developing a new currency, you should mark all tests that belong to that currency. For example, if your currency is called NewCoin, your device tests should have the following marker: ```python @pytest.mark.newcoin ``` This marker must be registered in [REGISTERED_MARKERS] file. If you wish to run a test only on TT, mark it with `@pytest.mark.skip_t1`. If the test should only run on T1, mark it with `@pytest.mark.skip_t2`. You must not use both on the same test. [pytest-random-order]: https://pypi.org/project/pytest-random-order/ [REGISTERED_MARKERS]: ../../tests/REGISTERED_MARKERS ## Extended testing and debugging ### Building for debugging (Emulator only) Build the debuggable unix binary so you can attach the gdb or lldb. This removes optimizations and reduces address space randomizaiton. ```sh make build_unix_debug ``` The final executable is significantly slower due to ASAN(Address Sanitizer) integration. If you want to catch some memory errors use this. ```sh time ASAN_OPTIONS=verbosity=1:detect_invalid_pointer_pairs=1:strict_init_order=true:strict_string_checks=true TREZOR_PROFILE="" poetry run make test_emu ``` ### Coverage (Emulator only) Get the Python code coverage report. If you want to get HTML/console summary output you need to install the __coverage.py__ tool. ```sh pip3 install coverage ``` Run the tests with coverage output. ```sh make build_unix && make coverage ```