mirror of
https://github.com/trezor/trezor-firmware.git
synced 2025-01-03 12:00:59 +00:00
new README for tools, improved README for defs
This commit is contained in:
matejcik
parent
00fcdb5605
commit
d54978bd28
@ -38,7 +38,7 @@ Supported coins that are not derived from Bitcoin, Ethereum or NEM are currently
|
||||
and listed in separate file [`misc/misc.json`](misc/misc.json). Each coin must also have
|
||||
an icon in `misc/<short>.png`, where `short` is lowercased `shortcut` field from the JSON.
|
||||
|
||||
## Keys and Duplicate Detection
|
||||
## Keys
|
||||
|
||||
Throughout the system, coins are identified by a _key_ - a colon-separated string
|
||||
generated from the coin's type and shortcut:
|
||||
@ -53,14 +53,18 @@ generated from the coin's type and shortcut:
|
||||
If a token shortcut has a suffix, such as `CAT (BlockCat)`, the whole thing is part
|
||||
of the key (so the key is `erc20:eth:CAT (BlockCat)`).
|
||||
|
||||
Sometimes coins end up with duplicate symbols. Especially in the ERC20 world this is
|
||||
common occurence. In such cases, keys are deduplicated by adding a counter at end,
|
||||
e.g.: `erc20:eth:SMT:0`, `erc20:eth:SMT:1`. Note that the suffix _is not stable_, so
|
||||
these coins can't be reliably uniquely identified.
|
||||
Sometimes coins end up with duplicate symbols, which in case of ERC20 tokens leads to
|
||||
key collisions. We do not allow duplicate symbols in the data, so this doesn't affect
|
||||
everyday use (see below). However, for validation purposes, it is sometimes useful
|
||||
to work with unfiltered data that includes the duplicates. In such cases, keys are
|
||||
deduplicated by adding a counter at end, e.g.: `erc20:eth:SMT:0`, `erc20:eth:SMT:1`.
|
||||
Note that the suffix _is not stable_, so these coins can't be reliably uniquely identified.
|
||||
|
||||
This doesn't matter, because **duplicate symbols are not allowed** in our data. Tokens
|
||||
that have symbol collisions are removed from the data set before processing. The duplicate
|
||||
status is mentioned in `support.json` (see below), but it is impossible to override.
|
||||
## Duplicate Detection
|
||||
|
||||
**Duplicate symbols are not allowed** in our data. Tokens that have symbol collisions
|
||||
are removed from the data set before processing. The duplicate status is mentioned
|
||||
in `support.json` (see below), but it is impossible to override from there.
|
||||
|
||||
Duplicate detection works as follows:
|
||||
|
||||
|
||||
207
tools/README.md
207
tools/README.md
@ -1,61 +1,178 @@
|
||||
# Tools directory
|
||||
# Common Tools
|
||||
|
||||
### `coin_info.py`
|
||||
This directory contains mostly tools that can manipulate definitions in [defs/](../defs).
|
||||
|
||||
Central module that extracts information from jsons in `defs/` directory.
|
||||
Its most important function is `get_all()`.
|
||||
Tools are written with [Click](http://click.pocoo.org/6/), so you always get a help text
|
||||
if you use the `--help` option.
|
||||
|
||||
All tools require Python 3.6 or higher and a bunch of dependencies, listed in `requirements.txt`.
|
||||
You can install them all with `pip3 install -r requirements.txt`.
|
||||
|
||||
## Tools overview
|
||||
|
||||
### `cointool.py`
|
||||
|
||||
Code and data generator. Has the following functions:
|
||||
This is a general-purpose tool to examine coin definitions. Currently it implements
|
||||
the following commands:
|
||||
|
||||
* __`check`__: runs validations. Currently, that is:
|
||||
* schema validity in `defs/coins/*.json`
|
||||
* availability of bitcore/blockbook backends
|
||||
* icon format
|
||||
* **`render`**: generate code based on a [Mako](http://docs.makotemplates.org/en/latest/index.html)
|
||||
template. By default, `cointool.py render foo.bar.mako` will put its result into
|
||||
file `foo.bar` in the same directory. See [usage in `trezor-core`](https://github.com/trezor/trezor-core/commit/348b99b8dc5bcfc4ab85e1e7faad3fb4ef3e8763).
|
||||
* **`check`**: check validity of json definitions and associated data. Used in CI.
|
||||
* **`coins_json`**: generate `coins.json` file for use in Connect, webwallet and python-trezor.
|
||||
This file contains data on Bitcoin-like coins and their support status.
|
||||
* **`coindefs`**: generate signed protobuf descriptions of coins. This is for future use
|
||||
and could allow us to not need to store coin data in Trezor itself.
|
||||
|
||||
* __`coins_json`__: generates `coins.json` for use in python-trezor, connect
|
||||
and wallet. By default outputs to current directory.
|
||||
|
||||
* __`coindefs`__: generates `coindefs.json`, intended future format for sending
|
||||
coin definitions to Trezor.
|
||||
|
||||
* __`render`__: for every `filename.ext.mako` passed (or for all found in directory),
|
||||
renders the Mako template with coin definitions and stores as corresponding
|
||||
`filename.ext`. This is used to generate code in trezor-mcu and trezor-core.
|
||||
|
||||
### `coins_details.py`
|
||||
|
||||
Regenerates `defs/coins_details.json`, which is a catalog of coins for https://trezor.io/coins.
|
||||
|
||||
All information is generated from coin definitions in `defs/`, support info is
|
||||
taken either from `support.json`, or assumed (see `coin_info.support_info()`).
|
||||
|
||||
If needed, any value can be overriden in `coins_details.override.json`.
|
||||
Use `cointool.py command --help` to get more information on each command.
|
||||
|
||||
### `support.py`
|
||||
|
||||
Support info management. Ensures `support.json` is in the proper format. Has the
|
||||
following subcommands:
|
||||
Used to query and manage info in `support.json`. This mainly supports the release flow.
|
||||
|
||||
* __`check`__: checks validity of support data. Ensures that data is valid,
|
||||
there are no orphaned keys in `support.json`, no known coins have missing
|
||||
support info, etc.
|
||||
The following commands are available:
|
||||
|
||||
* __`show <keyword>`__: searches coin database, matching key (`coin:BTC`),
|
||||
name ("Bitcoin") or shortcut / ticker symbol ("BTC"). Displays all coins that match
|
||||
and their support info, if found.
|
||||
* **`check`**: check validity of json data. Used in CI.
|
||||
* **`fix`**: fix expected problems: prune keys without associated coins and ensure
|
||||
that ERC20 tokens are correctly entered as duplicate.
|
||||
* **`show`**: keyword-search for a coin and show its support status for each device.
|
||||
* **`set`**: set support data.
|
||||
* **`release`**: perform the [release workflow](#release-workflow).
|
||||
|
||||
* __`set <key> [symbol=value]...`__: updates support info for coin key (`coin:BTC`,
|
||||
can be found with `support.py show`). Basic `symbol`s are: `trezor1 trezor2
|
||||
connect webwallet`. Anything else is considered a link name:
|
||||
`"Electrum=https://electrum.org"`
|
||||
Allowed `value`s are `yes`, `no`, `soon`, `planned`, URLs and firmware version
|
||||
numbers. Empty value (`trezor1=`) clears the respective symbol.
|
||||
Use `support.py command --help` to get more information on each command.
|
||||
|
||||
### `requirements.txt`
|
||||
### `coins_details.py`
|
||||
|
||||
List of Python requiremens for all tools in `pip` format. Set up your environment with
|
||||
`pip3 install -r requirements.txt`, or `pipenv install -r requirements.txt`.
|
||||
Generates `coins_details.json`, source file for https://trezor.io/coins.
|
||||
Collects data on coins, downloads market caps and puts everything into a single file.
|
||||
Caches market cap data so you don't have to download it every time.
|
||||
|
||||
Python 3.6 or higher is required.
|
||||
### `diffize_coins_details.py`
|
||||
|
||||
Compares generated `coins_details.json` to the released version currently served
|
||||
on https://trezor.io/coins, in a format that is nicely readable to humans and
|
||||
hard(er) to mess up by diff.
|
||||
|
||||
### `coin_info.py`
|
||||
|
||||
In case where code generation with `cointool.py render` is impractical or not sufficient,
|
||||
you can query the data directly through Python. Short usage example:
|
||||
|
||||
```python
|
||||
import coin_info
|
||||
|
||||
defs = coin_info.coin_info()
|
||||
list_of_all_coins = defs.as_list()
|
||||
dict_by_coin_key = defs.as_dict()
|
||||
|
||||
for token in defs.erc20:
|
||||
print(token["key"], token["name"], token["shortcut"])
|
||||
|
||||
support_info = coin_info.support_info(defs.misc)
|
||||
for key, support in support_info.values():
|
||||
t2_support = support["trezor2"]
|
||||
coin_name = dict_by_coin_key[key]
|
||||
if t2_support == "soon":
|
||||
print(coin_name, "will be supported soon!")
|
||||
elif t2_support:
|
||||
print(coin_name, "is supported since version", t2_support)
|
||||
else:
|
||||
print(coin_name, "is not supported")
|
||||
```
|
||||
|
||||
See docstrings for the most important functions: `coin_info()` and `support_info()`.
|
||||
|
||||
The file `coindef.py` is a protobuf definition for passing coin data to Trezor
|
||||
from the outside.
|
||||
|
||||
|
||||
# Release Workflow
|
||||
|
||||
This entails collecting information on coins whose support status is unknown,
|
||||
marking coins whose support status is `soon`, and including new ERC20 tokens.
|
||||
|
||||
## Maintaining Support Status
|
||||
|
||||
When a new coin definition is added, its support status is _unknown_. It is excluded
|
||||
from code generation by default. If you want to include a coin in a firmware build,
|
||||
you need to switch it to _soon_ first. You can set multiple support statuses at the
|
||||
same time:
|
||||
|
||||
```
|
||||
$ ./support.py show Ontology
|
||||
misc:ONT - Ontology (ONT)
|
||||
* connect : NO
|
||||
* trezor1 : support info missing
|
||||
* trezor2 : support info missing
|
||||
* webwallet : NO
|
||||
|
||||
$ ./support.py set misc:ONT trezor1=no -r "not planned on T1" trezor2=soon
|
||||
misc:ONT - Ontology (ONT)
|
||||
* connect : NO
|
||||
* trezor1 : NO (reason: not planned on T1)
|
||||
* trezor2 : SOON
|
||||
* webwallet : NO
|
||||
```
|
||||
|
||||
Afterwards, review and commit changes to `defs/support.json`, and update the `trezor-common`
|
||||
submodule in your target firmware.
|
||||
|
||||
If you're adding multiple coins at once, you can use `support.py release 1 --soon` to automatically
|
||||
add all currently-unknown coins to _soon_. (The `1` indicates that this is for Trezor One)
|
||||
|
||||
ERC20 tokens in _unknown_ state are considered _soon_ as well, unless their symbols
|
||||
are duplicates. Use `support.py fix` to synchronize duplicate status in `support.json` file.
|
||||
Or mark them as unsupported explicitly.
|
||||
|
||||
## Releasing a new firmware
|
||||
|
||||
#### **Step 1:** update the tokens repo
|
||||
|
||||
```sh
|
||||
pushd defs/ethereum/tokens
|
||||
git checkout master
|
||||
git pull
|
||||
popd
|
||||
git add defs/ethereum/tokens
|
||||
```
|
||||
|
||||
#### **Step 2:** run the release flow
|
||||
|
||||
```sh
|
||||
./tools/support.py release 2
|
||||
```
|
||||
|
||||
The number `2` indicates that you are releasing Trezor 2. The version will be
|
||||
automatically determined, based on currently released firmwares. Or you can explicitly
|
||||
specify the version with `-r 2.1.0`.
|
||||
|
||||
All currently known unreleased ERC20 tokens are automatically set to the given version.
|
||||
|
||||
All coins marked _soon_ are set to the current version. This is automatic - coins that
|
||||
were marked _soon_ were used in code generation and so should be released. If you want
|
||||
to avoid this, you will have to manually revert each coin to _soon_ status, either with
|
||||
`support.py set`, or by manually editing `support.json`.
|
||||
|
||||
Coins in state _unknown_, i.e., coins that are known in the definitions but not listed
|
||||
in support files, will be also added. But you will be interactively asked to confirm
|
||||
each one. Use `-y` or `--add-all` to auto-add all of them.
|
||||
|
||||
Use `-n` or `--dry-run` to see changes without writing them to `support.json`. Use
|
||||
`-v` or `--verbose` to also show ERC20 tokens which are processed silently by default.
|
||||
|
||||
Use `-g` or `--git-tag` to automatically tag the current `HEAD` with a version, e.g.,
|
||||
`trezor2-2.1.0`. This might become default in the future.
|
||||
|
||||
XXX this should also commit the changes though, otherwise the tag will apply to the wrong
|
||||
commit.
|
||||
|
||||
#### **Step 3:** review and commit your changes
|
||||
|
||||
Use `git diff` to review changes made, commit and push. If you tagged the commit in the
|
||||
previous step, don't forget to `git push --tags` too.
|
||||
|
||||
#### **Step 4:** update submodule in your target repository
|
||||
|
||||
Go to `trezor-core` or `trezor-mcu` checkout and update the submodule. Checkout the
|
||||
appropriate tag if you created it. If you're in `trezor-core`, run `make templates`
|
||||
to update source files.
|
||||
|
||||
Loading…
Reference in New Issue
Block a user