arno/trezor-firmware
1
0
Fork 0
mirror of https://github.com/trezor/trezor-firmware.git synced 2025-07-17 20:18:10 +00:00
Code Issues Releases Wiki Activity

chore(common, docs): ethereum definitions

Browse Source
This commit is contained in:
Martin Novak 2022-10-18 15:30:57 +02:00
parent 9f36248520
commit 5186ca1c99
18 changed files with 243 additions and 39638 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

58
common/defs/README.md
View File

@ -35,20 +35,19 @@ We will not support coins that have `address_type` 0, i.e., same as Bitcoin.
#### `eth` and `erc20` #### `eth` and `erc20`
Definitions for Ethereum chains(networks) and tokens(erc20) are splitted in two parts: Definitions for Ethereum chains(networks) and tokens(erc20) are split in two parts:
1. built-in definitions - some of the chain and token definitions are built into the firmware 1. built-in definitions - some of the chain and token definitions are built into the firmware
image. List of built-in chains is stored in [`ethereum/eth_builtin_networks.json`](ethereum/eth_builtin_networks.json) image. List of built-in chains is stored in [`ethereum/networks.json`](ethereum/networks.json)
and tokens in [`ethereum/eth_builtin_tokens.json`](ethereum/eth_builtin_tokens.json). and tokens in [`ethereum/tokens.json`](ethereum/tokens.json).
2. external definitions - external definitions are dynamically generated from multiple 2. external definitions - external definitions are dynamically generated from multiple
sources - [`coingecko.com`](https://www.coingecko.com/), [defillama](https://defillama.com/), sources. Whole process is described in separate [document](Ethereum_definitions.md).
[Ethereum Lists - chains](https://github.com/ethereum-lists/chains)
and [Ethereum Lists - tokens](https://github.com/ethereum-lists/tokens).
They are re-generated and signed on every release of the firmware. Signed
definitions are available at #TODO: add link!!
If you want to add or update a token definition in Trezor, you need to get your change If you want to add or update a token definition in Trezor, you need to get your change
to the [Ethereum Lists - tokens](https://github.com/ethereum-lists/tokens) repository first. to the [Ethereum Lists - tokens](https://github.com/ethereum-lists/tokens) repository first.
For more details see [document](https://docs.trezor.io/trezor-firmware/common/communication/ethereum-definitions-binary-format.html)
about Ethereum definitions.
#### `nem` #### `nem`
The file [`nem/nem_mosaics.json`](nem/nem_mosaics.json) describes NEM mosaics. The file [`nem/nem_mosaics.json`](nem/nem_mosaics.json) describes NEM mosaics.
@ -64,21 +63,23 @@ an icon in `misc/<short>.png`, where `short` is lowercased `shortcut` field from
Throughout the system, coins are identified by a _key_ - a colon-separated string Throughout the system, coins are identified by a _key_ - a colon-separated string
generated from the coin's type and shortcut: generated from the coin's type and shortcut:
* for Bitcoin-likes, key is `bitcoin:XYZ` * for Bitcoin-likes, key is `bitcoin:<shortcut>`
* for Ethereum networks, key is `eth:XYZ` * for Ethereum networks, key is `eth:<shortcut>`
* for ERC20 tokens, key is `erc20:<chain>:XYZ` * for ERC20 tokens, key is `erc20:<chain_symbol>:<token_shortcut>`
* for NEM mosaic, key is `nem:XYZ` * for NEM mosaic, key is `nem:<shortcut>`
* for others, key is `misc:XYZ` * for others, key is `misc:<shortcut>`
If a token shortcut has a suffix, such as `CAT (BlockCat)`, the whole thing is part If a token shortcut has a suffix, such as `CAT (BlockCat)`, the whole thing is part
of the key (so the key is `erc20:eth:CAT (BlockCat)`). of the key (so the key is `erc20:eth:CAT (BlockCat)`).
Sometimes coins end up with duplicate symbols (keys), which in case of ERC20 tokens leads to Sometimes coins end up with duplicate keys. We do not allow duplicate symbols
key collisions. We do not allow duplicate symbols in the built-in data, so this doesn't affect in the built-in data. In such cases, keys are deduplicated by adding:
everyday use (see below). * first 4 characters of a token address in case of ERC20 tokens, e.g. `erc20:eth:BID:1da0`, or
* a counter at end, e.g.: `erc20:eth:SMT:0`, `erc20:eth:SMT:1`.
Note that the suffix _is not stable_, so these coins can't be reliably uniquely identified.
External definitions are generated based on data from external APIs on which we rely to have For Ethereum networks and ERC20 tokens we have a small and stable set of definitions, so key
the collisions solved. collisions should not happen.
## Duplicate Detection ## Duplicate Detection
@ -87,7 +88,8 @@ are removed from the data set before processing. The duplicate status is mention
in `support.json` (see below), but it is impossible to override from there. in `support.json` (see below), but it is impossible to override from there.
We try to minimize the occurence of duplicates in built-in tokens, but sometimes its We try to minimize the occurence of duplicates in built-in tokens, but sometimes its
unavoidable. unavoidable. For Ethereum networks and ERC20 tokens we have a small and stable set
of definitions, so symbol collisions should not happen.
Duplicate detection works as follows: Duplicate detection works as follows:
@ -115,26 +117,10 @@ asked to.
You can use `./tools/cointool.py check -d all` to inspect duplicate detection in detail. You can use `./tools/cointool.py check -d all` to inspect duplicate detection in detail.
# Coins Details # Wallet URLs
The file [`coins_details.json`](coins_details.json) is a list of all known coins
with support status, market cap information and relevant links. This is the source
file for https://trezor.io/coins.
You should never make changes to `coins_details.json` directly. Use `./tools/coins_details.py`
to regenerate it from known data.
If you need to change information in this file, modify the source information instead -
one of the JSON files in the groups listed above, support info in `support.json`, or
make a pull request to the tokens repository.
If you want to add a **wallet link**, modify the file [`wallets.json`](wallets.json). If you want to add a **wallet link**, modify the file [`wallets.json`](wallets.json).
If this is not viable for some reason, or if there is no source information ,
you can also edit [`coins_details.override.json`](coins_details.override.json).
External contributors should not touch this file unless asked to.
# Support Information # Support Information
We keep track of support status of each built-in coin over our devices. That is We keep track of support status of each built-in coin over our devices. That is

39100
common/defs/coins_details.json
View File

File diff suppressed because it is too large Load Diff

10
common/defs/coins_details.override.json
View File

@ -1,10 +0,0 @@
{
"nem:DIMTOK": {
"coinmarketcap_alias": "dimcoin"
},
"misc:LSK": {
"hidden": true,
"ignore_cmc_rank": true,
"reason": "delisted incompatible hardfork"
}
}

0
common/defs/ethereum/eth_builtin_networks.json → common/defs/ethereum/networks.json
View File

0
common/defs/ethereum/eth_builtin_tokens.json → common/defs/ethereum/tokens.json
View File

28
common/protob/protocol.md
View File

@ -20,26 +20,6 @@ Following packets has the following structure:
| 0 | 1 | char[1] | '?' magic constant | | 0 | 1 | char[1] | '?' magic constant |
| 1 | 63 | uint8_t[63] | following bytes of message encoded in Protocol Buffers (padded with zeroes if shorter) | | 1 | 63 | uint8_t[63] | following bytes of message encoded in Protocol Buffers (padded with zeroes if shorter) |
## Ethereum network and token definitions
Ethereum network and token definitions could be sent from host to device. External definitions are generated from time to time
and provided on publicly accessible website - `data.trezor.io` # TODO: update url. For more info look
at the [definitions README.md](../defs/README.md#eth-and-erc20).
To ensure that the definitions send to device are genuine we use Merkle tree with a combination of signed Merkle tree root hash.
Every definition is then checked and verified at FW side on receiving.
Definitions (network/token) are binary encoded and have a special format:
1. prefix:
1. format version of the definition (UTF-8 string `trzd` + version number, padded with zeroes if shorter, 8 bytes)
2. type of data (unsigned integer, 1 byte)
3. data version of the definition (unsigned integer, 4 bytes)
4. length of the encoded protobuf message - payload length in bytes (unsigned integer, 2 bytes)
2. payload: serialized form of protobuf message EthereumNetworkInfo or EthereumTokenInfo (N bytes)
3. suffix:
1. length of the Merkle tree proof - number of hashes in the proof (unsigned integer, 1 byte)
2. proof - individual hashes used to compute Merkle tree root hash (plain bits, N*32 bytes)
3. signed Merkle tree root hash (plain bits, 64 bytes)
## Adding new message ## Adding new message
To add new message to Trezor protocol follow these steps: To add new message to Trezor protocol follow these steps:
@ -75,11 +55,13 @@ where:
- in case that the message is embedded in another message we don't state the message type - in case that the message is embedded in another message we don't state the message type
- `MSG_FLOW_TAG` denotes the flow of the messages in a session (use case). Possible tags are: - `MSG_FLOW_TAG` denotes the flow of the messages in a session (use case). Possible tags are:
- `@start` means that this message is first - `@start` means that this message is first
- `@end`means that this message is last - `@end` means that this message is last
- `@next` denotes messages that could follow after this message - `@next` denotes messages that could follow after this message
- `@embed` denotes messages that are embedded into other message(s) - `@embed` denotes messages that are embedded into other message(s)
- `@auxstart` - ?? diamond start - `@auxstart` - denotes message for sub-flow start; this message might appear at any time
- `@auxend` - ?? diamond end during any other flow. When the corresponding @auxend message is processed, the original
flow resumes.
- `@auxend` - see `@auxstart`
Messages flow is checked at the compile time. Messages flow is checked at the compile time.

30
common/tools/README.md
View File

@ -39,31 +39,21 @@ The following commands are available:
Use `support.py command --help` to get more information on each command. Use `support.py command --help` to get more information on each command.
### `coins_details.py`
Generates `coins_details.json`, source file for https://trezor.io/coins.
Collects data on coins, downloads market caps and puts everything into a single file.
Caches market cap data so you don't have to download it every time.
### `diffize_coins_details.py`
Compares generated `coins_details.json` to the released version currently served
on https://trezor.io/coins, in a format that is nicely readable to humans and
hard(er) to mess up by diff.
### `ethereum_definitions.py` ### `ethereum_definitions.py`
Script used to work with Ethereum definitions. Script used to work with Ethereum definitions.
Definitions for Ethereum chains(networks) and tokens(erc20) are dynamically generated Dynamically generate and/or sign definitions for Ethereum chains(networks) and tokens(erc20).
from multiple sources - [`coingecko.com`](https://www.coingecko.com/), [defillama](https://defillama.com/), For more info see the
[Ethereum Lists - chains](https://github.com/ethereum-lists/chains) [document](https://docs.trezor.io/trezor-firmware/common/communication/ethereum-definitions-binary-format.html)
and [Ethereum Lists - tokens](https://github.com/ethereum-lists/tokens). describing Ethereum definitions.
The following commands are available: The following commands are available:
* **`prepare-definitions`**: collect and process definitions for Ethereum networks (chains) and tokens. * **`prepare-definitions`**: collect and process definitions for Ethereum networks (chains) and tokens.
* **`sign-definitions`**: generate signed protobuf definitions for Ethereum networks (chains) and tokens. * **`sign-definitions`**: generate signed protobuf definitions for Ethereum networks (chains) and tokens.
Use `ethereum_definitions.py command --help` to get more information on each command.
### `coin_info.py` ### `coin_info.py`
In case where code generation with `cointool.py render` is impractical or not sufficient, In case where code generation with `cointool.py render` is impractical or not sufficient,
@ -96,7 +86,7 @@ from the outside.
### `marketcap.py` ### `marketcap.py`
Module for obtaining market cap and price data used by `coins_details.py` and `maxfee.py`. Module for obtaining market cap and price data used by `maxfee.py`.
### `maxfee.py` ### `maxfee.py`
@ -149,10 +139,12 @@ Or mark them as unsupported explicitly.
All currently known unreleased ERC20 tokens are automatically set to the given version. All currently known unreleased ERC20 tokens are automatically set to the given version.
All coins marked _soon_ are set to the current version. This is automatic - coins that **_Note that "soon" feature was already removed and following paragraph is deprecated._**
_All coins marked _soon_ are set to the current version. This is automatic - coins that
were marked _soon_ were used in code generation and so should be released. If you want were marked _soon_ were used in code generation and so should be released. If you want
to avoid this, you will have to manually revert each coin to _soon_ status, either with to avoid this, you will have to manually revert each coin to _soon_ status, either with
`support.py set`, or by manually editing `support.json`. `support.py set`, or by manually editing `support.json`._
Coins in state _unknown_, i.e., coins that are known in the definitions but not listed Coins in state _unknown_, i.e., coins that are known in the definitions but not listed
in support files, will be also added. But you will be interactively asked to confirm in support files, will be also added. But you will be interactively asked to confirm

61
common/tools/coin_info.py
View File

@ -365,8 +365,8 @@ def _load_btc_coins() -> Coins:
def _load_builtin_ethereum_networks() -> Coins: def _load_builtin_ethereum_networks() -> Coins:
"""Load ethereum networks from `ethereum/eth_builtin_networks.json`""" """Load ethereum networks from `ethereum/networks.json`"""
chains_data = load_json("ethereum", "eth_builtin_networks.json") chains_data = load_json("ethereum", "networks.json")
networks: Coins = [] networks: Coins = []
for chain_data in chains_data: for chain_data in chains_data:
chain_data.update( chain_data.update(
@ -379,8 +379,8 @@ def _load_builtin_ethereum_networks() -> Coins:
def _load_builtin_erc20_tokens() -> Coins: def _load_builtin_erc20_tokens() -> Coins:
"""Load ERC20 tokens from `ethereum/eth_builtin_tokens.json`.""" """Load ERC20 tokens from `ethereum/tokens.json`."""
tokens_data = load_json("ethereum", "eth_builtin_tokens.json") tokens_data = load_json("ethereum", "tokens.json")
all_tokens: Coins = [] all_tokens: Coins = []
for chain_id_and_chain, tokens in tokens_data.items(): for chain_id_and_chain, tokens in tokens_data.items():
@ -662,55 +662,6 @@ def apply_duplicity_overrides(coins: Coins) -> Coins:
return override_bucket return override_bucket
def deduplicate_erc20(buckets: CoinBuckets, networks: Coins) -> None:
"""Apply further processing to ERC20 duplicate buckets.
This function works on results of `mark_duplicate_shortcuts`.
Buckets that contain at least one non-token are ignored - symbol collisions
with non-tokens always apply.
Otherwise the following rules are applied:
1. If _all tokens_ in the bucket have shortcuts with distinct suffixes, e.g.,
`CAT (BitClave)` and `CAT (Blockcat)`, the bucket is cleared - all are considered
non-duplicate.
(If even one token in the bucket _does not_ have a distinct suffix, e.g.,
`MIT` and `MIT (Mychatcoin)`, this rule does not apply and ALL tokens in the bucket
are still considered duplicate.)
2. If there is only one "main" token in the bucket, the bucket is cleared.
That means that all other tokens must be on testnets.
"""
testnet_networks = {n["chain"] for n in networks if n["slip44"] == 1}
def clear_bucket(bucket: Coins) -> None:
# allow all coins, except those that are explicitly marked through overrides
for coin in bucket:
coin["duplicate"] = False
for bucket in buckets.values():
# Only check buckets that contain purely ERC20 tokens. Collision with
# a non-token is always forbidden.
if not all(is_token(c) for c in bucket):
continue
splits = (symbol_from_shortcut(coin["shortcut"]) for coin in bucket)
suffixes = {suffix for _, suffix in splits}
# if 1. all suffixes are distinct and 2. none of them are empty
if len(suffixes) == len(bucket) and all(suffixes):
clear_bucket(bucket)
continue
# protected categories:
testnets = [coin for coin in bucket if coin["chain"] in testnet_networks]
remaining = [coin for coin in bucket if coin not in testnets]
if len(remaining) <= 1:
clear_bucket(bucket)
def deduplicate_keys(all_coins: Coins) -> None: def deduplicate_keys(all_coins: Coins) -> None:
dups: CoinBuckets = defaultdict(list) dups: CoinBuckets = defaultdict(list)
for coin in all_coins: for coin in all_coins:
@ -795,9 +746,7 @@ def coin_info_with_duplicates() -> tuple[CoinsInfo, CoinBuckets]:
coin_list = all_coins.as_list() coin_list = all_coins.as_list()
# generate duplicity buckets based on shortcuts # generate duplicity buckets based on shortcuts
buckets = mark_duplicate_shortcuts(all_coins.as_list()) buckets = mark_duplicate_shortcuts(all_coins.as_list())
# apply further processing to ERC20 tokens # ensure the whole list has unique keys
deduplicate_erc20(buckets, all_coins.eth)
# ensure the whole list has unique keys (taking into account changes from deduplicate_erc20)
deduplicate_keys(coin_list) deduplicate_keys(coin_list)
# apply duplicity overrides # apply duplicity overrides
buckets["_override"] = apply_duplicity_overrides(coin_list) buckets["_override"] = apply_duplicity_overrides(coin_list)

309
common/tools/coins_details.py
View File

@ -1,309 +0,0 @@
#!/usr/bin/env python3
"""Fetch information about coins supported by Trezor and update it in coins_details.json."""
import json
import logging
import os
import sys
import time
import click
import coin_info
import marketcap
LOG = logging.getLogger(__name__)
OPTIONAL_KEYS = ("links", "notes", "wallet")
ALLOWED_SUPPORT_STATUS = ("yes", "no")
WALLETS = coin_info.load_json("wallets.json")
OVERRIDES = coin_info.load_json("coins_details.override.json")
VERSIONS = coin_info.latest_releases()
TREZOR_KNOWN_URLS = (
"https://suite.trezor.io",
"https://wallet.trezor.io",
)
def update_marketcaps(coins):
for coin in coins.values():
coin["marketcap_usd"] = marketcap.marketcap(coin) or 0
def summary(coins, api_key):
t1_coins = 0
t2_coins = 0
supported_marketcap = 0
for coin in coins.values():
if coin.get("hidden"):
continue
t1_enabled = coin["t1_enabled"] == "yes"
t2_enabled = coin["t2_enabled"] == "yes"
if t1_enabled:
t1_coins += 1
if t2_enabled:
t2_coins += 1
if t1_enabled or t2_enabled:
supported_marketcap += coin.get("marketcap_usd", 0)
total_marketcap = None
try:
ret = marketcap.call("global-metrics/quotes/latest", api_key)
total_marketcap = int(ret["data"]["quote"]["USD"]["total_market_cap"])
except Exception:
pass
marketcap_percent = 100 * supported_marketcap / total_marketcap
return dict(
updated_at=int(time.time()),
updated_at_readable=time.asctime(),
t1_coins=t1_coins,
t2_coins=t2_coins,
marketcap_usd=supported_marketcap,
total_marketcap_usd=total_marketcap,
marketcap_supported=f"{marketcap_percent:.02f} %",
)
def _is_supported(support, trezor_version):
# True or version string means YES
# False or None means NO
return "yes" if support.get(trezor_version) else "no"
def _suite_support(coin, support):
"""Check the "suite" support property.
If set, check that at least one of the backends run on trezor.io.
If yes, assume we support the coin in our wallet.
Otherwise it's probably working with a custom backend, which means don't
link to our wallet.
"""
if not support.get("suite"):
return False
return any(".trezor.io" in url for url in coin["blockbook"])
def dict_merge(orig, new):
if isinstance(new, dict) and isinstance(orig, dict):
for k, v in new.items():
orig[k] = dict_merge(orig.get(k), v)
return orig
else:
return new
def update_simple(coins, support_info, type):
res = {}
for coin in coins:
key = coin["key"]
support = support_info[key]
details = dict(
name=coin["name"],
shortcut=coin["shortcut"],
type=type,
t1_enabled=_is_supported(support, "trezor1"),
t2_enabled=_is_supported(support, "trezor2"),
wallet={},
)
for k in OPTIONAL_KEYS:
if k in coin:
details[k] = coin[k]
details["wallet"].update(WALLETS.get(key, {}))
res[key] = details
return res
def update_bitcoin(coins, support_info):
res = update_simple(coins, support_info, "coin")
for coin in coins:
key = coin["key"]
support = support_info[key]
details = dict(
name=coin["coin_label"],
links=dict(Homepage=coin["website"], Github=coin["github"]),
wallet=coin_info.WALLET_SUITE if _suite_support(coin, support) else {},
)
dict_merge(res[key], details)
return res
def update_erc20(coins, networks, support_info):
# TODO skip disabled networks?
network_support = {n["chain"]: support_info.get(n["key"]) for n in networks}
network_testnets = {n["chain"] for n in networks if "Testnet" in n["name"]}
res = update_simple(coins, support_info, "erc20")
for coin in coins:
key = coin["key"]
chain = coin["chain"]
details = dict(
network=chain,
address=coin["address"],
shortcut=coin["shortcut"],
links={},
)
if chain in network_testnets:
details["hidden"] = True
if network_support.get(chain, {}).get("suite"):
details["wallets"] = coin_info.WALLET_SUITE
dict_merge(res[key], details)
return res
def update_ethereum_networks(coins, support_info):
res = update_simple(coins, support_info, "coin")
for coin in coins:
key = coin["key"]
details = dict(links=dict(Homepage=coin.get("url")))
if support_info[key].get("suite"):
details["wallets"] = coin_info.WALLET_SUITE
dict_merge(res[key], details)
return res
def update_nem_mosaics(coins, support_info):
res = update_simple(coins, support_info, "mosaic")
for coin in coins:
key = coin["key"]
details = dict(wallet=coin_info.WALLET_NEM)
dict_merge(res[key], details)
return res
def check_missing_data(coins):
for k, coin in coins.items():
hide = False
if "Homepage" not in coin.get("links", {}):
level = logging.WARNING
if k.startswith("erc20:"):
level = logging.INFO
LOG.log(level, f"{k}: Missing homepage")
hide = True
if coin["t1_enabled"] not in ALLOWED_SUPPORT_STATUS:
LOG.error(f"{k}: Unknown t1_enabled: {coin['t1_enabled']}")
hide = True
if coin["t2_enabled"] not in ALLOWED_SUPPORT_STATUS:
LOG.error(f"{k}: Unknown t2_enabled: {coin['t2_enabled']}")
hide = True
# check wallets
for wallet in coin["wallet"]:
name = wallet.get("name")
url = wallet.get("url")
if not name or not url:
LOG.warning(f"{k}: Bad wallet entry")
hide = True
continue
if "trezor" in name.lower() and url not in TREZOR_KNOWN_URLS:
LOG.warning(f"{k}: Strange URL for Trezor Wallet")
if coin["t1_enabled"] == "no" and coin["t2_enabled"] == "no":
LOG.info(f"{k}: Coin not enabled on either device")
hide = True
if len(coin.get("wallet", [])) == 0:
LOG.debug(f"{k}: Missing wallet")
if "Testnet" in coin["name"] or "Regtest" in coin["name"]:
LOG.debug(f"{k}: Hiding testnet")
hide = True
if not hide and coin.get("hidden"):
LOG.info(f"{k}: Details are OK, but coin is still hidden")
if hide:
data = marketcap.get_coin(coin)
if data and data["cmc_rank"] < 150 and not coin.get("ignore_cmc_rank"):
LOG.warning(f"{k}: Hiding coin ranked {data['cmc_rank']} on CMC")
coin["hidden"] = 1
# summary of hidden coins
hidden_coins = [k for k, coin in coins.items() if coin.get("hidden")]
for key in hidden_coins:
del coins[key]
def apply_overrides(coins):
for key, override in OVERRIDES.items():
if key not in coins:
LOG.warning(f"override without coin: {key}")
continue
dict_merge(coins[key], override)
def finalize_wallets(coins):
def sort_key(w):
if "trezor.io" in w["url"]:
return 0, w["name"]
else:
return 1, w["name"]
for coin in coins.values():
wallets_list = [
dict(name=name, url=url) for name, url in coin["wallet"].items() if url
]
wallets_list.sort(key=sort_key)
coin["wallet"] = wallets_list
@click.command()
# fmt: off
@click.option("-r", "--refresh", "refresh", flag_value=True, default=None, help="Force refresh market cap info")
@click.option("-R", "--no-refresh", "refresh", flag_value=False, default=None, help="Force use cached market cap info")
@click.option("-A", "--api-key", required=True, envvar="COINMARKETCAP_API_KEY", help="Coinmarketcap API key")
@click.option("-v", "--verbose", is_flag=True, help="Display more info")
# fmt: on
def main(refresh, api_key, verbose):
# setup logging
log_level = logging.DEBUG if verbose else logging.WARNING
root = logging.getLogger()
root.setLevel(log_level)
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(log_level)
root.addHandler(handler)
marketcap.init(api_key, refresh=refresh)
defs, _ = coin_info.coin_info_with_duplicates()
support_info = coin_info.support_info(defs)
coins = {}
coins.update(update_bitcoin(defs.bitcoin, support_info))
coins.update(update_erc20(defs.erc20, defs.eth, support_info))
coins.update(update_ethereum_networks(defs.eth, support_info))
coins.update(update_nem_mosaics(defs.nem, support_info))
coins.update(update_simple(defs.misc, support_info, "coin"))
apply_overrides(coins)
finalize_wallets(coins)
update_marketcaps(coins)
check_missing_data(coins)
info = summary(coins, api_key)
details = dict(coins=coins, info=info)
print(json.dumps(info, sort_keys=True, indent=4))
with open(os.path.join(coin_info.DEFS_DIR, "coins_details.json"), "w") as f:
json.dump(details, f, sort_keys=True, indent=4)
f.write("\n")
if __name__ == "__main__":
main()

59
common/tools/diffize_coins_details.py
View File

@ -1,59 +0,0 @@
#!/usr/bin/env python3
import json
import os
import subprocess
import tempfile
import click
import requests
LIVE_URL = "https://trezor.io/static/json/coins_details.json"
COINS_DETAILS = os.path.join(
os.path.dirname(__file__), "..", "defs", "coins_details.json"
)
def diffize_file(coins_details, tmp):
coins_list = list(coins_details["coins"].values())
for coin in coins_list:
coin.pop("marketcap_usd", None)
links = coin.get("links", {})
wallets = coin.get("wallet", {})
for link in links:
links[link] = links[link].rstrip("/")
for wallet in wallets:
wallet["url"] = wallet["url"].rstrip("/")
if not coin.get("wallet"):
coin.pop("wallet", None)
coins_list.sort(key=lambda c: c["name"])
for coin in coins_list:
name = coin["name"]
for key in coin:
print(name, "\t", key, ":", coin[key], file=tmp)
tmp.flush()
@click.command()
def cli():
"""Compare data from trezor.io/coins with current coins_details.json
Shows a nicely formatted diff between the live version and the trezor-common
version. Useful for catching auto-generation problems, etc.
"""
live_json = requests.get(LIVE_URL).json()
with open(COINS_DETAILS) as f:
coins_details = json.load(f)
Tmp = tempfile.NamedTemporaryFile
with Tmp("w") as tmpA, Tmp("w") as tmpB:
diffize_file(live_json, tmpA)
diffize_file(coins_details, tmpB)
subprocess.call(["diff", "-u", "--color=auto", tmpA.name, tmpB.name])
if __name__ == "__main__":
cli()

1
common/tools/ethereum_definitions.py
View File

@ -43,6 +43,7 @@ else:
DEFINITIONS_CACHE_FILEPATH = pathlib.Path("definitions-cache.json") DEFINITIONS_CACHE_FILEPATH = pathlib.Path("definitions-cache.json")
# ====== utils ====== # ====== utils ======

24
common/tools/marketcap.py
View File

@ -79,30 +79,6 @@ def init(api_key, refresh=None):
COINS_SEARCHABLE = data_searchable COINS_SEARCHABLE = data_searchable
def get_coin(coin):
if coin["type"] == "erc20":
address = coin["address"].lower()
return COINS_SEARCHABLE.get(address)
data = None
if "coinmarketcap_alias" in coin:
data = COINS_SEARCHABLE.get(coin["coinmarketcap_alias"])
if data is None:
slug = coin["name"].replace(" ", "-").lower()
data = COINS_SEARCHABLE.get(slug)
if data is None:
data = COINS_SEARCHABLE.get(coin["shortcut"].lower())
return data
def marketcap(coin):
data = get_coin(coin)
if data is None:
return None
return int(data["quote"]["USD"]["market_cap"])
def fiat_price(coin_symbol): def fiat_price(coin_symbol):
data = COINS_SEARCHABLE.get(coin_symbol) data = COINS_SEARCHABLE.get(coin_symbol)
if data is None: if data is None:

2
common/tools/release.sh
View File

@ -31,6 +31,4 @@ make -C $HERE/../.. gen
diff $CHECK_OUTPUT/pre.txt $CHECK_OUTPUT/post.txt diff $CHECK_OUTPUT/pre.txt $CHECK_OUTPUT/post.txt
$HERE/coins_details.py
$HERE/ethereum_definitions.py prepare-definitions $HERE/ethereum_definitions.py prepare-definitions

2
docs/SUMMARY.md
View File

@ -31,8 +31,10 @@
- [Passphrase](common/communication/passphrase.md) - [Passphrase](common/communication/passphrase.md)
- [Migration](common/communication/passphrase-redesign-migration.md) - [Migration](common/communication/passphrase-redesign-migration.md)
- [Bitcoin signing](common/communication/bitcoin-signing.md) - [Bitcoin signing](common/communication/bitcoin-signing.md)
- [Ethereum definitions binary format](common/communication/ethereum-definitions-binary-format.md)
- [Reproducible builds](common/reproducible-build.md) - [Reproducible builds](common/reproducible-build.md)
- [Message Workflows](common/message-workflows.md) - [Message Workflows](common/message-workflows.md)
- [Ethereum definitions](common/ethereum-definitions.md)
- [Storage](storage/index.md) - [Storage](storage/index.md)
- [Tests](tests/index.md) - [Tests](tests/index.md)
- [Device Tests](tests/device-tests.md) - [Device Tests](tests/device-tests.md)

13
docs/common/communication/ethereum-definitions-binary-format.md Normal file
View File

@ -0,0 +1,13 @@
# Ethereum definition binary format
Definitions are binary encoded and have a special format:
1. prefix:
1. format version of the definition (UTF-8 string `trzd` + version number, padded with zeroes if shorter, 8 bytes)
2. type of data (unsigned integer, 1 byte)
3. data version of the definition (unsigned integer, 4 bytes)
4. length of the encoded protobuf message - payload length in bytes (unsigned integer, 2 bytes)
2. payload: serialized form of protobuf message EthereumNetworkInfo or EthereumTokenInfo (N bytes)
3. suffix:
1. length of the Merkle tree proof - number of hashes in the proof (unsigned integer, 1 byte)
2. proof - individual hashes used to compute Merkle tree root hash (plain bits, N*32 bytes)
3. signed Merkle tree root hash (plain bits, 64 bytes)

4
docs/common/communication/index.md
View File

@ -8,6 +8,10 @@ We use [Protobuf v2](https://developers.google.com/protocol-buffers/) for host-d
Protobuf messages are defined in the [Common](https://github.com/trezor/trezor-firmware/tree/master/common) project, which is part of this monorepo. This repository is also exported to [trezor/trezor-common](https://github.com/trezor/trezor-common) to be used by third parties, which prefer not to include the whole monorepo. That copy is read-only mirror and all changes are happening in this monorepo. Protobuf messages are defined in the [Common](https://github.com/trezor/trezor-firmware/tree/master/common) project, which is part of this monorepo. This repository is also exported to [trezor/trezor-common](https://github.com/trezor/trezor-common) to be used by third parties, which prefer not to include the whole monorepo. That copy is read-only mirror and all changes are happening in this monorepo.
## Ethereum definitions binary format
Ethereum definition binary format is described in [short document](ethereum-definitions-binary-format.md).
## Notable topics ## Notable topics
- [Sessions](sessions.md) - [Sessions](sessions.md)

176
docs/common/ethereum-definitions.md Normal file
View File

@ -0,0 +1,176 @@
# Ethereum definitions
Ethereum definitions for networks (chains) and tokens are dynamically generated and
encoded into binary blobs. These blobs could be send as a part of some messages
to the device.
## Built-in definitions
In addition to generated binary blobs, small subset of the definitions is also hardcoded
in firmware in a decoded form (not as a binary blob).
Location of these definitions is for:
* networks - [`networks.json`](https://github.com/trezor/trezor-firmware/blob/master/common/defs/ethereum/networks.json)
* tokens - [`tokens.json`](https://github.com/trezor/trezor-firmware/blob/master/common/defs/ethereum/tokens.json)
Built-in definitions are written by hand and are not subject to any generation described
below.
## External definitions
Generated binary blobs are first saved locally (by the script that generates them)
and then published online. By "external" is meant definitions not hardcoded in firmware.
#### File structure
Saved definitions (binary blobs) are stored in directory (e.g. `definitions-latest/`)
with specific file structure:
````
definitions-latest/
├── by_chain_id/
│ ├── "CHAIN_ID"/
│ │ ├── network.dat
│ │ ├── token_"TOKEN_ADDRESS".dat
│ │ ├── token_"TOKEN_ADDRESS".dat
│ │ ...
│ ├── "CHAIN_ID"/
│ │ ├── network.dat
│ │ ├── token_"TOKEN_ADDRESS".dat
│ │ ├── token_"TOKEN_ADDRESS".dat
│ │ ...
│ ...
└── by_slip44/
├── "SLIP44_ID"/
│ └── network.dat
├── "SLIP44_ID"/
│ └── network.dat
...
````
where:
* `CHAIN_ID` is a corresponding chain ID of included network/tokens
* `SLIP44_ID` is a corresponding SLIP44 ID of included network
* `TOKEN_ADDRESS` is a lowercase token address (stripped of `0x` prefix)
Notice that token definitions are only accessible by `CHAIN_ID` and `TOKEN_ADDRESS`
(directory `by_chain_id`), not by `SLIP44_ID` (directory `by_slip44`).
#### Definitions online
Generated binary definitions are available online at [publicly accessible website](https://data.trezor.io/eth_definitions) # TODO: update url.
To get the desired definition (one at a time), URL can be composed in multiple ways
and the structure is the same as it is described in the [file structure](#file-structure)
section. Base URL format is `https://data.trezor.io/eth_definitions/LOOKUP_TYPE/ID/NAME`
where:
* `LOOKUP_TYPE` is one of `by_chain_id` or `by_slip44`
* `ID` is either chain ID or SLIP44 ID (depends on the chosen lookup type)
* `NAME` is either:
* `network.dat` for network definition at given chain ID or SLIP44 ID or
* `token_"TOKEN_ADDRESS".dat` for token definition at given chain ID and token address
(see [file structure](#file-structure) section on how to format the token address)
Definitions could be also downloaded by one request in a ZIP file at https://data.trezor.io/eth_definitions/definitions.zip # TODO: update url.
#### `Trezorctl`
Automatic manipulation with the definitions is implemented in `trezorctl` tool.
All Ethereum commands that do work with definitions contains contains options
to automatically download online definitions or use locally stored definitions.
For more info look at the `trezorctl` [documentation]
(https://github.com/trezor/trezor-firmware/blob/master/python/docs/README.rst).
## Process of generating the definitions
Binary definitions are generated by one script -
[`ethereum_definitions.py`](https://github.com/trezor/trezor-firmware/blob/master/common/tools/ethereum_definitions.py)
The process is composed of multiple stages described in the following sections.
### 1. Prepare the definitions to JSON file
Preparation stage starts with collection of all data from all [data sources](#data-sources)
(or from locally stored cache) and creation of connections between the data (finding
the same IDs, etc.). In case that no cache was found at the beginning the cache file
is saved (by default `definitions-cache.json`).
Every definition is checked for the size limitations and user can decide (if he choosed
the interactive mode) what should happen if the size of some field is bigger than
the allowed maximum. This is needed due to size restrictions on Model 1 (buffers with fixed
size).
Subsequently all the definition are checked for duplicates and user has to decide
which ones will be removed. No duplicates are allowed in further processing.
If there are already locally stored definitions in JSON file the comparison takes place. Simple
comparison algorithm looks for these types of changes:
* moved definition - definition was moved to other chain ID or address (in case of tokens)
* modified definition - definition has same chain ID or address but other fields changed
* deleted definition - definition was deleted
* resurrected definition - previously deleted definition is available again
* changes in symbol - symbol has changed in the definition
Results of the comparison are printed out.
After solving all the collisions, size limitations and changes we have "clean" data
saved in a local file (by default `definitions-latest.json`).
### 2. Generate and sign the definitions
Input for this stage is the JSON file from previous stage with definitions.
For the purpose of [verifying the definitions on FW side](#verification-and-validation-on-fw-side)
we use Merkle Tree data structure, which gives us the option to effectively split
this stage into three sub-stages:
1. Getting the Merkle tree root hash - Merkle tree is build from all the definitions
and the root hash is computed.
2. Signing the hash - computed Merkle tree hash is signed using private key.
3. Generating the binary definitions - Merkle tree is build from all the definitions
and the root hash is computed again. Then this hash is verified against signed hash
from previous step using public key to ensure that nothing has changed. If everything
is ok the binary encoded (see
[definition binary format](communication/ethereum-definitions-binary-format.md) section)
definitions are generated to directory with specific [file structure](#file-structure).
### 3. Publish the definitions
Binary definitions are published to our website. See section [definitions online](#definitions-online)
for more information.
## Data sources
External Ethereum definitions are generated based on data from external APIs and repositories:
* [`coingecko.com`](https://www.coingecko.com/) for most of the info about networks and tokens
* [defillama](https://defillama.com/) to pair as much networks as we can to CoinGecko ID
* [Ethereum Lists - chains](https://github.com/ethereum-lists/chains) as the only source of EVM-based networks
* [Ethereum Lists - tokens](https://github.com/ethereum-lists/tokens) as another source of tokens
## Validation and verification on FW side
To ensure that the definitions send to device are genuine we have to check them
on receiving.
#### Validation
First thing that happens when binary definition is received is validation.Validation
is a single process when FW compares hardcoded values against values found in received
definition.
Based on the [definition binary format](communication/ethereum-definitions-binary-format.md)
FW checks the following values:
* format version - format version found in definiton has to be equal or higher than
the one specified in FW
* type of data - type of data found in definiton has to be the same as FW expects
* data version - data version found in definiton has to be equal or higher than
the one specified in FW
If any of these checks fail the definition is rejected.
#### Verification
Verification of received definitions is made using Merkle tree data structure
in combination with a signed Merkle tree root hash. Every encoded definition
is packed with list of Merkle tree proofs and the signed Merkle tree root hash
(see [definition binary format](communication/ethereum-definitions-binary-format.md)
section).
When the definition is received, hash from the combination of the encoded definition
itself and the list of proofs is computed. This hash is then verified against signed
hash included in received definition using the public key hardcoded in FW. This last
step ensures that nothing has changed in the definition.

4
docs/common/index.md
View File

@ -6,6 +6,10 @@ Common contains files shared among Trezor projects.
JSON coin definitions and support tables. JSON coin definitions and support tables.
## Ethereum definitions
Description of Ethereum definitions and the process of their generation. See [Ethereum definitions](ethereum-definitions.md).
## Protobuf Definitions ## Protobuf Definitions
Common Protobuf definitions for the Trezor protocol. Also see [Communication](communication/index.md). Common Protobuf definitions for the Trezor protocol. Also see [Communication](communication/index.md).