trezor-firmware

Commit Graph

Author	SHA1	Message	Date
matejcik	af98a4071a	device_tests: allow custom runxfail We can now selectively runxfail certain tests. This is useful for accepting PRs into trezor-core: 1. trezor-core is going to get a pytest.ini that sets xfail_strict. That means that if an `xfail`ed test actually passes, that will break the test suite. So it will be visible when we implement a feature for which tests exist. 2. To allow PRs to pass the test suite without touching python-trezor directly, we add a new pytest.ini option: run_xfail. This adds a list of markers which will ignore `xfail`. So: 2.1 First, the python-trezor PR marks the tests with the name of the feature. This commit already does that: Lisk tests are marked `@pytest.mark.lisk`, NEMs are `@pytest.mark.nem`, etc. The tests will be also marked with `xfail`, because the feature is not in core yet. 2.2 Then, the trezor-core PR implements the feature, which makes the `xfail`ed tests pass. That breaks the test suite. 2.3 To fix the test suite, the core PR also adds a `run_xfail` to `pytest.ini`: `run_xfail = lisk`. (it can take a list: `run_xfail = lisk nem stellar`) That will make the test suite behave as if the tests are not `xfail`ed. If the feature is implemented correctly, the tests will pass. 2.4 When the PR is accepted to core, the next step should be a PR to python-trezor that removes the `xfail`s. After that, we should also remove the `run_xfail` option, just to be tidy.	6 years ago
Tomas Susanka	edbcd2012d	tests/device/nem: multisig tests	7 years ago
Tomas Susanka	891865f948	tests/device/nem: tests fixes due to layout changes	7 years ago

Author

SHA1

Message

Date

matejcik

af98a4071a

device_tests: allow custom runxfail

We can now selectively runxfail certain tests. This is useful for
accepting PRs into trezor-core:

1. trezor-core is going to get a pytest.ini that sets xfail_strict.
   That means that if an `xfail`ed test actually passes, that will
   break the test suite. So it will be visible when we implement
   a feature for which tests exist.
2. To allow PRs to pass the test suite without touching python-trezor
   directly, we add a new pytest.ini option: run_xfail.
   This adds a list of markers which will ignore `xfail`.
   So:
   2.1 First, the python-trezor PR marks the tests with the name
       of the feature. This commit already does that: Lisk tests
       are marked `@pytest.mark.lisk`, NEMs are `@pytest.mark.nem`,
       etc.
       The tests will be also marked with `xfail`, because the
       feature is not in core yet.
   2.2 Then, the trezor-core PR implements the feature, which makes
       the `xfail`ed tests pass. That breaks the test suite.
   2.3 To fix the test suite, the core PR also adds a `run_xfail`
       to `pytest.ini`: `run_xfail = lisk`.
       (it can take a list: `run_xfail = lisk nem stellar`)
       That will make the test suite behave as if the tests are not
       `xfail`ed. If the feature is implemented correctly, the tests
       will pass.
   2.4 When the PR is accepted to core, the next step should be
       a PR to python-trezor that removes the `xfail`s. After that,
       we should also remove the `run_xfail` option, just to be tidy.

Tomas Susanka

edbcd2012d

tests/device/nem: multisig tests

Tomas Susanka

891865f948

tests/device/nem: tests fixes due to layout changes

3 Commits (f273bbb736945d169dde496379aea713fe5abc6c)