mirror of
https://github.com/trezor/trezor-firmware.git
synced 2025-07-16 11:38:12 +00:00
chore(common, docs): ethereum definitions
This commit is contained in:
Martin Novak
parent
9f36248520
commit
5186ca1c99
@ -35,20 +35,19 @@ We will not support coins that have `address_type` 0, i.e., same as Bitcoin.
|
||||
|
||||
#### `eth` and `erc20`
|
||||
|
||||
Definitions for Ethereum chains(networks) and tokens(erc20) are splitted in two parts:
|
||||
Definitions for Ethereum chains(networks) and tokens(erc20) are split in two parts:
|
||||
1. built-in definitions - some of the chain and token definitions are built into the firmware
|
||||
image. List of built-in chains is stored in [`ethereum/eth_builtin_networks.json`](ethereum/eth_builtin_networks.json)
|
||||
and tokens in [`ethereum/eth_builtin_tokens.json`](ethereum/eth_builtin_tokens.json).
|
||||
image. List of built-in chains is stored in [`ethereum/networks.json`](ethereum/networks.json)
|
||||
and tokens in [`ethereum/tokens.json`](ethereum/tokens.json).
|
||||
2. external definitions - external definitions are dynamically generated from multiple
|
||||
sources - [`coingecko.com`](https://www.coingecko.com/), [defillama](https://defillama.com/),
|
||||
[Ethereum Lists - chains](https://github.com/ethereum-lists/chains)
|
||||
and [Ethereum Lists - tokens](https://github.com/ethereum-lists/tokens).
|
||||
They are re-generated and signed on every release of the firmware. Signed
|
||||
definitions are available at #TODO: add link!!
|
||||
sources. Whole process is described in separate [document](Ethereum_definitions.md).
|
||||
|
||||
If you want to add or update a token definition in Trezor, you need to get your change
|
||||
to the [Ethereum Lists - tokens](https://github.com/ethereum-lists/tokens) repository first.
|
||||
|
||||
For more details see [document](https://docs.trezor.io/trezor-firmware/common/communication/ethereum-definitions-binary-format.html)
|
||||
about Ethereum definitions.
|
||||
|
||||
#### `nem`
|
||||
|
||||
The file [`nem/nem_mosaics.json`](nem/nem_mosaics.json) describes NEM mosaics.
|
||||
@ -64,21 +63,23 @@ an icon in `misc/<short>.png`, where `short` is lowercased `shortcut` field from
|
||||
Throughout the system, coins are identified by a _key_ - a colon-separated string
|
||||
generated from the coin's type and shortcut:
|
||||
|
||||
* for Bitcoin-likes, key is `bitcoin:XYZ`
|
||||
* for Ethereum networks, key is `eth:XYZ`
|
||||
* for ERC20 tokens, key is `erc20:<chain>:XYZ`
|
||||
* for NEM mosaic, key is `nem:XYZ`
|
||||
* for others, key is `misc:XYZ`
|
||||
* for Bitcoin-likes, key is `bitcoin:<shortcut>`
|
||||
* for Ethereum networks, key is `eth:<shortcut>`
|
||||
* for ERC20 tokens, key is `erc20:<chain_symbol>:<token_shortcut>`
|
||||
* for NEM mosaic, key is `nem:<shortcut>`
|
||||
* for others, key is `misc:<shortcut>`
|
||||
|
||||
If a token shortcut has a suffix, such as `CAT (BlockCat)`, the whole thing is part
|
||||
of the key (so the key is `erc20:eth:CAT (BlockCat)`).
|
||||
|
||||
Sometimes coins end up with duplicate symbols (keys), which in case of ERC20 tokens leads to
|
||||
key collisions. We do not allow duplicate symbols in the built-in data, so this doesn't affect
|
||||
everyday use (see below).
|
||||
Sometimes coins end up with duplicate keys. We do not allow duplicate symbols
|
||||
in the built-in data. In such cases, keys are deduplicated by adding:
|
||||
* first 4 characters of a token address in case of ERC20 tokens, e.g. `erc20:eth:BID:1da0`, or
|
||||
* a counter at end, e.g.: `erc20:eth:SMT:0`, `erc20:eth:SMT:1`.
|
||||
Note that the suffix _is not stable_, so these coins can't be reliably uniquely identified.
|
||||
|
||||
External definitions are generated based on data from external APIs on which we rely to have
|
||||
the collisions solved.
|
||||
For Ethereum networks and ERC20 tokens we have a small and stable set of definitions, so key
|
||||
collisions should not happen.
|
||||
|
||||
## Duplicate Detection
|
||||
|
||||
@ -87,7 +88,8 @@ are removed from the data set before processing. The duplicate status is mention
|
||||
in `support.json` (see below), but it is impossible to override from there.
|
||||
|
||||
We try to minimize the occurence of duplicates in built-in tokens, but sometimes its
|
||||
unavoidable.
|
||||
unavoidable. For Ethereum networks and ERC20 tokens we have a small and stable set
|
||||
of definitions, so symbol collisions should not happen.
|
||||
|
||||
Duplicate detection works as follows:
|
||||
|
||||
@ -115,26 +117,10 @@ asked to.
|
||||
You can use `./tools/cointool.py check -d all` to inspect duplicate detection in detail.
|
||||
|
||||
|
||||
# Coins Details
|
||||
|
||||
The file [`coins_details.json`](coins_details.json) is a list of all known coins
|
||||
with support status, market cap information and relevant links. This is the source
|
||||
file for https://trezor.io/coins.
|
||||
|
||||
You should never make changes to `coins_details.json` directly. Use `./tools/coins_details.py`
|
||||
to regenerate it from known data.
|
||||
|
||||
If you need to change information in this file, modify the source information instead -
|
||||
one of the JSON files in the groups listed above, support info in `support.json`, or
|
||||
make a pull request to the tokens repository.
|
||||
# Wallet URLs
|
||||
|
||||
If you want to add a **wallet link**, modify the file [`wallets.json`](wallets.json).
|
||||
|
||||
If this is not viable for some reason, or if there is no source information ,
|
||||
you can also edit [`coins_details.override.json`](coins_details.override.json).
|
||||
External contributors should not touch this file unless asked to.
|
||||
|
||||
|
||||
# Support Information
|
||||
|
||||
We keep track of support status of each built-in coin over our devices. That is
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@ -1,10 +0,0 @@
|
||||
{
|
||||
"nem:DIMTOK": {
|
||||
"coinmarketcap_alias": "dimcoin"
|
||||
},
|
||||
"misc:LSK": {
|
||||
"hidden": true,
|
||||
"ignore_cmc_rank": true,
|
||||
"reason": "delisted incompatible hardfork"
|
||||
}
|
||||
}
|
||||
@ -20,26 +20,6 @@ Following packets has the following structure:
|
||||
| 0 | 1 | char[1] | '?' magic constant |
|
||||
| 1 | 63 | uint8_t[63] | following bytes of message encoded in Protocol Buffers (padded with zeroes if shorter) |
|
||||
|
||||
## Ethereum network and token definitions
|
||||
Ethereum network and token definitions could be sent from host to device. External definitions are generated from time to time
|
||||
and provided on publicly accessible website - `data.trezor.io` # TODO: update url. For more info look
|
||||
at the [definitions README.md](../defs/README.md#eth-and-erc20).
|
||||
To ensure that the definitions send to device are genuine we use Merkle tree with a combination of signed Merkle tree root hash.
|
||||
Every definition is then checked and verified at FW side on receiving.
|
||||
|
||||
|
||||
Definitions (network/token) are binary encoded and have a special format:
|
||||
1. prefix:
|
||||
1. format version of the definition (UTF-8 string `trzd` + version number, padded with zeroes if shorter, 8 bytes)
|
||||
2. type of data (unsigned integer, 1 byte)
|
||||
3. data version of the definition (unsigned integer, 4 bytes)
|
||||
4. length of the encoded protobuf message - payload length in bytes (unsigned integer, 2 bytes)
|
||||
2. payload: serialized form of protobuf message EthereumNetworkInfo or EthereumTokenInfo (N bytes)
|
||||
3. suffix:
|
||||
1. length of the Merkle tree proof - number of hashes in the proof (unsigned integer, 1 byte)
|
||||
2. proof - individual hashes used to compute Merkle tree root hash (plain bits, N*32 bytes)
|
||||
3. signed Merkle tree root hash (plain bits, 64 bytes)
|
||||
|
||||
## Adding new message
|
||||
|
||||
To add new message to Trezor protocol follow these steps:
|
||||
@ -75,11 +55,13 @@ where:
|
||||
- in case that the message is embedded in another message we don't state the message type
|
||||
- `MSG_FLOW_TAG` denotes the flow of the messages in a session (use case). Possible tags are:
|
||||
- `@start` means that this message is first
|
||||
- `@end`means that this message is last
|
||||
- `@end` means that this message is last
|
||||
- `@next` denotes messages that could follow after this message
|
||||
- `@embed` denotes messages that are embedded into other message(s)
|
||||
- `@auxstart` - ?? diamond start
|
||||
- `@auxend` - ?? diamond end
|
||||
- `@auxstart` - denotes message for sub-flow start; this message might appear at any time
|
||||
during any other flow. When the corresponding @auxend message is processed, the original
|
||||
flow resumes.
|
||||
- `@auxend` - see `@auxstart`
|
||||
|
||||
Messages flow is checked at the compile time.
|
||||
|
||||
|
||||
@ -39,31 +39,21 @@ The following commands are available:
|
||||
|
||||
Use `support.py command --help` to get more information on each command.
|
||||
|
||||
### `coins_details.py`
|
||||
|
||||
Generates `coins_details.json`, source file for https://trezor.io/coins.
|
||||
Collects data on coins, downloads market caps and puts everything into a single file.
|
||||
Caches market cap data so you don't have to download it every time.
|
||||
|
||||
### `diffize_coins_details.py`
|
||||
|
||||
Compares generated `coins_details.json` to the released version currently served
|
||||
on https://trezor.io/coins, in a format that is nicely readable to humans and
|
||||
hard(er) to mess up by diff.
|
||||
|
||||
### `ethereum_definitions.py`
|
||||
|
||||
Script used to work with Ethereum definitions.
|
||||
|
||||
Definitions for Ethereum chains(networks) and tokens(erc20) are dynamically generated
|
||||
from multiple sources - [`coingecko.com`](https://www.coingecko.com/), [defillama](https://defillama.com/),
|
||||
[Ethereum Lists - chains](https://github.com/ethereum-lists/chains)
|
||||
and [Ethereum Lists - tokens](https://github.com/ethereum-lists/tokens).
|
||||
Dynamically generate and/or sign definitions for Ethereum chains(networks) and tokens(erc20).
|
||||
For more info see the
|
||||
[document](https://docs.trezor.io/trezor-firmware/common/communication/ethereum-definitions-binary-format.html)
|
||||
describing Ethereum definitions.
|
||||
|
||||
The following commands are available:
|
||||
* **`prepare-definitions`**: collect and process definitions for Ethereum networks (chains) and tokens.
|
||||
* **`sign-definitions`**: generate signed protobuf definitions for Ethereum networks (chains) and tokens.
|
||||
|
||||
Use `ethereum_definitions.py command --help` to get more information on each command.
|
||||
|
||||
### `coin_info.py`
|
||||
|
||||
In case where code generation with `cointool.py render` is impractical or not sufficient,
|
||||
@ -96,7 +86,7 @@ from the outside.
|
||||
|
||||
### `marketcap.py`
|
||||
|
||||
Module for obtaining market cap and price data used by `coins_details.py` and `maxfee.py`.
|
||||
Module for obtaining market cap and price data used by `maxfee.py`.
|
||||
|
||||
### `maxfee.py`
|
||||
|
||||
@ -149,10 +139,12 @@ Or mark them as unsupported explicitly.
|
||||
|
||||
All currently known unreleased ERC20 tokens are automatically set to the given version.
|
||||
|
||||
All coins marked _soon_ are set to the current version. This is automatic - coins that
|
||||
**_Note that "soon" feature was already removed and following paragraph is deprecated._**
|
||||
|
||||
_All coins marked _soon_ are set to the current version. This is automatic - coins that
|
||||
were marked _soon_ were used in code generation and so should be released. If you want
|
||||
to avoid this, you will have to manually revert each coin to _soon_ status, either with
|
||||
`support.py set`, or by manually editing `support.json`.
|
||||
`support.py set`, or by manually editing `support.json`._
|
||||
|
||||
Coins in state _unknown_, i.e., coins that are known in the definitions but not listed
|
||||
in support files, will be also added. But you will be interactively asked to confirm
|
||||
|
||||
@ -365,8 +365,8 @@ def _load_btc_coins() -> Coins:
|
||||
|
||||
|
||||
def _load_builtin_ethereum_networks() -> Coins:
|
||||
"""Load ethereum networks from `ethereum/eth_builtin_networks.json`"""
|
||||
chains_data = load_json("ethereum", "eth_builtin_networks.json")
|
||||
"""Load ethereum networks from `ethereum/networks.json`"""
|
||||
chains_data = load_json("ethereum", "networks.json")
|
||||
networks: Coins = []
|
||||
for chain_data in chains_data:
|
||||
chain_data.update(
|
||||
@ -379,8 +379,8 @@ def _load_builtin_ethereum_networks() -> Coins:
|
||||
|
||||
|
||||
def _load_builtin_erc20_tokens() -> Coins:
|
||||
"""Load ERC20 tokens from `ethereum/eth_builtin_tokens.json`."""
|
||||
tokens_data = load_json("ethereum", "eth_builtin_tokens.json")
|
||||
"""Load ERC20 tokens from `ethereum/tokens.json`."""
|
||||
tokens_data = load_json("ethereum", "tokens.json")
|
||||
all_tokens: Coins = []
|
||||
|
||||
for chain_id_and_chain, tokens in tokens_data.items():
|
||||
@ -662,55 +662,6 @@ def apply_duplicity_overrides(coins: Coins) -> Coins:
|
||||
return override_bucket
|
||||
|
||||
|
||||
def deduplicate_erc20(buckets: CoinBuckets, networks: Coins) -> None:
|
||||
"""Apply further processing to ERC20 duplicate buckets.
|
||||
|
||||
This function works on results of `mark_duplicate_shortcuts`.
|
||||
|
||||
Buckets that contain at least one non-token are ignored - symbol collisions
|
||||
with non-tokens always apply.
|
||||
|
||||
Otherwise the following rules are applied:
|
||||
|
||||
1. If _all tokens_ in the bucket have shortcuts with distinct suffixes, e.g.,
|
||||
`CAT (BitClave)` and `CAT (Blockcat)`, the bucket is cleared - all are considered
|
||||
non-duplicate.
|
||||
|
||||
(If even one token in the bucket _does not_ have a distinct suffix, e.g.,
|
||||
`MIT` and `MIT (Mychatcoin)`, this rule does not apply and ALL tokens in the bucket
|
||||
are still considered duplicate.)
|
||||
|
||||
2. If there is only one "main" token in the bucket, the bucket is cleared.
|
||||
That means that all other tokens must be on testnets.
|
||||
"""
|
||||
|
||||
testnet_networks = {n["chain"] for n in networks if n["slip44"] == 1}
|
||||
|
||||
def clear_bucket(bucket: Coins) -> None:
|
||||
# allow all coins, except those that are explicitly marked through overrides
|
||||
for coin in bucket:
|
||||
coin["duplicate"] = False
|
||||
|
||||
for bucket in buckets.values():
|
||||
# Only check buckets that contain purely ERC20 tokens. Collision with
|
||||
# a non-token is always forbidden.
|
||||
if not all(is_token(c) for c in bucket):
|
||||
continue
|
||||
|
||||
splits = (symbol_from_shortcut(coin["shortcut"]) for coin in bucket)
|
||||
suffixes = {suffix for _, suffix in splits}
|
||||
# if 1. all suffixes are distinct and 2. none of them are empty
|
||||
if len(suffixes) == len(bucket) and all(suffixes):
|
||||
clear_bucket(bucket)
|
||||
continue
|
||||
|
||||
# protected categories:
|
||||
testnets = [coin for coin in bucket if coin["chain"] in testnet_networks]
|
||||
remaining = [coin for coin in bucket if coin not in testnets]
|
||||
if len(remaining) <= 1:
|
||||
clear_bucket(bucket)
|
||||
|
||||
|
||||
def deduplicate_keys(all_coins: Coins) -> None:
|
||||
dups: CoinBuckets = defaultdict(list)
|
||||
for coin in all_coins:
|
||||
@ -795,9 +746,7 @@ def coin_info_with_duplicates() -> tuple[CoinsInfo, CoinBuckets]:
|
||||
coin_list = all_coins.as_list()
|
||||
# generate duplicity buckets based on shortcuts
|
||||
buckets = mark_duplicate_shortcuts(all_coins.as_list())
|
||||
# apply further processing to ERC20 tokens
|
||||
deduplicate_erc20(buckets, all_coins.eth)
|
||||
# ensure the whole list has unique keys (taking into account changes from deduplicate_erc20)
|
||||
# ensure the whole list has unique keys
|
||||
deduplicate_keys(coin_list)
|
||||
# apply duplicity overrides
|
||||
buckets["_override"] = apply_duplicity_overrides(coin_list)
|
||||
|
||||
@ -1,309 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fetch information about coins supported by Trezor and update it in coins_details.json."""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
import click
|
||||
|
||||
import coin_info
|
||||
import marketcap
|
||||
|
||||
LOG = logging.getLogger(__name__)
|
||||
|
||||
OPTIONAL_KEYS = ("links", "notes", "wallet")
|
||||
ALLOWED_SUPPORT_STATUS = ("yes", "no")
|
||||
|
||||
WALLETS = coin_info.load_json("wallets.json")
|
||||
OVERRIDES = coin_info.load_json("coins_details.override.json")
|
||||
VERSIONS = coin_info.latest_releases()
|
||||
|
||||
|
||||
TREZOR_KNOWN_URLS = (
|
||||
"https://suite.trezor.io",
|
||||
"https://wallet.trezor.io",
|
||||
)
|
||||
|
||||
|
||||
def update_marketcaps(coins):
|
||||
for coin in coins.values():
|
||||
coin["marketcap_usd"] = marketcap.marketcap(coin) or 0
|
||||
|
||||
|
||||
def summary(coins, api_key):
|
||||
t1_coins = 0
|
||||
t2_coins = 0
|
||||
supported_marketcap = 0
|
||||
for coin in coins.values():
|
||||
if coin.get("hidden"):
|
||||
continue
|
||||
|
||||
t1_enabled = coin["t1_enabled"] == "yes"
|
||||
t2_enabled = coin["t2_enabled"] == "yes"
|
||||
if t1_enabled:
|
||||
t1_coins += 1
|
||||
if t2_enabled:
|
||||
t2_coins += 1
|
||||
if t1_enabled or t2_enabled:
|
||||
supported_marketcap += coin.get("marketcap_usd", 0)
|
||||
|
||||
total_marketcap = None
|
||||
try:
|
||||
ret = marketcap.call("global-metrics/quotes/latest", api_key)
|
||||
total_marketcap = int(ret["data"]["quote"]["USD"]["total_market_cap"])
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
marketcap_percent = 100 * supported_marketcap / total_marketcap
|
||||
return dict(
|
||||
updated_at=int(time.time()),
|
||||
updated_at_readable=time.asctime(),
|
||||
t1_coins=t1_coins,
|
||||
t2_coins=t2_coins,
|
||||
marketcap_usd=supported_marketcap,
|
||||
total_marketcap_usd=total_marketcap,
|
||||
marketcap_supported=f"{marketcap_percent:.02f} %",
|
||||
)
|
||||
|
||||
|
||||
def _is_supported(support, trezor_version):
|
||||
# True or version string means YES
|
||||
# False or None means NO
|
||||
return "yes" if support.get(trezor_version) else "no"
|
||||
|
||||
|
||||
def _suite_support(coin, support):
|
||||
"""Check the "suite" support property.
|
||||
If set, check that at least one of the backends run on trezor.io.
|
||||
If yes, assume we support the coin in our wallet.
|
||||
Otherwise it's probably working with a custom backend, which means don't
|
||||
link to our wallet.
|
||||
"""
|
||||
if not support.get("suite"):
|
||||
return False
|
||||
return any(".trezor.io" in url for url in coin["blockbook"])
|
||||
|
||||
|
||||
def dict_merge(orig, new):
|
||||
if isinstance(new, dict) and isinstance(orig, dict):
|
||||
for k, v in new.items():
|
||||
orig[k] = dict_merge(orig.get(k), v)
|
||||
return orig
|
||||
else:
|
||||
return new
|
||||
|
||||
|
||||
def update_simple(coins, support_info, type):
|
||||
res = {}
|
||||
for coin in coins:
|
||||
key = coin["key"]
|
||||
support = support_info[key]
|
||||
|
||||
details = dict(
|
||||
name=coin["name"],
|
||||
shortcut=coin["shortcut"],
|
||||
type=type,
|
||||
t1_enabled=_is_supported(support, "trezor1"),
|
||||
t2_enabled=_is_supported(support, "trezor2"),
|
||||
wallet={},
|
||||
)
|
||||
for k in OPTIONAL_KEYS:
|
||||
if k in coin:
|
||||
details[k] = coin[k]
|
||||
|
||||
details["wallet"].update(WALLETS.get(key, {}))
|
||||
|
||||
res[key] = details
|
||||
|
||||
return res
|
||||
|
||||
|
||||
def update_bitcoin(coins, support_info):
|
||||
res = update_simple(coins, support_info, "coin")
|
||||
for coin in coins:
|
||||
key = coin["key"]
|
||||
support = support_info[key]
|
||||
details = dict(
|
||||
name=coin["coin_label"],
|
||||
links=dict(Homepage=coin["website"], Github=coin["github"]),
|
||||
wallet=coin_info.WALLET_SUITE if _suite_support(coin, support) else {},
|
||||
)
|
||||
dict_merge(res[key], details)
|
||||
|
||||
return res
|
||||
|
||||
|
||||
def update_erc20(coins, networks, support_info):
|
||||
# TODO skip disabled networks?
|
||||
network_support = {n["chain"]: support_info.get(n["key"]) for n in networks}
|
||||
network_testnets = {n["chain"] for n in networks if "Testnet" in n["name"]}
|
||||
res = update_simple(coins, support_info, "erc20")
|
||||
for coin in coins:
|
||||
key = coin["key"]
|
||||
chain = coin["chain"]
|
||||
|
||||
details = dict(
|
||||
network=chain,
|
||||
address=coin["address"],
|
||||
shortcut=coin["shortcut"],
|
||||
links={},
|
||||
)
|
||||
|
||||
if chain in network_testnets:
|
||||
details["hidden"] = True
|
||||
|
||||
if network_support.get(chain, {}).get("suite"):
|
||||
details["wallets"] = coin_info.WALLET_SUITE
|
||||
|
||||
dict_merge(res[key], details)
|
||||
|
||||
return res
|
||||
|
||||
|
||||
def update_ethereum_networks(coins, support_info):
|
||||
res = update_simple(coins, support_info, "coin")
|
||||
for coin in coins:
|
||||
key = coin["key"]
|
||||
details = dict(links=dict(Homepage=coin.get("url")))
|
||||
if support_info[key].get("suite"):
|
||||
details["wallets"] = coin_info.WALLET_SUITE
|
||||
dict_merge(res[key], details)
|
||||
|
||||
return res
|
||||
|
||||
|
||||
def update_nem_mosaics(coins, support_info):
|
||||
res = update_simple(coins, support_info, "mosaic")
|
||||
for coin in coins:
|
||||
key = coin["key"]
|
||||
details = dict(wallet=coin_info.WALLET_NEM)
|
||||
dict_merge(res[key], details)
|
||||
|
||||
return res
|
||||
|
||||
|
||||
def check_missing_data(coins):
|
||||
for k, coin in coins.items():
|
||||
hide = False
|
||||
|
||||
if "Homepage" not in coin.get("links", {}):
|
||||
level = logging.WARNING
|
||||
if k.startswith("erc20:"):
|
||||
level = logging.INFO
|
||||
LOG.log(level, f"{k}: Missing homepage")
|
||||
hide = True
|
||||
if coin["t1_enabled"] not in ALLOWED_SUPPORT_STATUS:
|
||||
LOG.error(f"{k}: Unknown t1_enabled: {coin['t1_enabled']}")
|
||||
hide = True
|
||||
if coin["t2_enabled"] not in ALLOWED_SUPPORT_STATUS:
|
||||
LOG.error(f"{k}: Unknown t2_enabled: {coin['t2_enabled']}")
|
||||
hide = True
|
||||
|
||||
# check wallets
|
||||
for wallet in coin["wallet"]:
|
||||
name = wallet.get("name")
|
||||
url = wallet.get("url")
|
||||
if not name or not url:
|
||||
LOG.warning(f"{k}: Bad wallet entry")
|
||||
hide = True
|
||||
continue
|
||||
if "trezor" in name.lower() and url not in TREZOR_KNOWN_URLS:
|
||||
LOG.warning(f"{k}: Strange URL for Trezor Wallet")
|
||||
|
||||
if coin["t1_enabled"] == "no" and coin["t2_enabled"] == "no":
|
||||
LOG.info(f"{k}: Coin not enabled on either device")
|
||||
hide = True
|
||||
|
||||
if len(coin.get("wallet", [])) == 0:
|
||||
LOG.debug(f"{k}: Missing wallet")
|
||||
|
||||
if "Testnet" in coin["name"] or "Regtest" in coin["name"]:
|
||||
LOG.debug(f"{k}: Hiding testnet")
|
||||
hide = True
|
||||
|
||||
if not hide and coin.get("hidden"):
|
||||
LOG.info(f"{k}: Details are OK, but coin is still hidden")
|
||||
|
||||
if hide:
|
||||
data = marketcap.get_coin(coin)
|
||||
if data and data["cmc_rank"] < 150 and not coin.get("ignore_cmc_rank"):
|
||||
LOG.warning(f"{k}: Hiding coin ranked {data['cmc_rank']} on CMC")
|
||||
coin["hidden"] = 1
|
||||
|
||||
# summary of hidden coins
|
||||
hidden_coins = [k for k, coin in coins.items() if coin.get("hidden")]
|
||||
for key in hidden_coins:
|
||||
del coins[key]
|
||||
|
||||
|
||||
def apply_overrides(coins):
|
||||
for key, override in OVERRIDES.items():
|
||||
if key not in coins:
|
||||
LOG.warning(f"override without coin: {key}")
|
||||
continue
|
||||
|
||||
dict_merge(coins[key], override)
|
||||
|
||||
|
||||
def finalize_wallets(coins):
|
||||
def sort_key(w):
|
||||
if "trezor.io" in w["url"]:
|
||||
return 0, w["name"]
|
||||
else:
|
||||
return 1, w["name"]
|
||||
|
||||
for coin in coins.values():
|
||||
wallets_list = [
|
||||
dict(name=name, url=url) for name, url in coin["wallet"].items() if url
|
||||
]
|
||||
wallets_list.sort(key=sort_key)
|
||||
coin["wallet"] = wallets_list
|
||||
|
||||
|
||||
@click.command()
|
||||
# fmt: off
|
||||
@click.option("-r", "--refresh", "refresh", flag_value=True, default=None, help="Force refresh market cap info")
|
||||
@click.option("-R", "--no-refresh", "refresh", flag_value=False, default=None, help="Force use cached market cap info")
|
||||
@click.option("-A", "--api-key", required=True, envvar="COINMARKETCAP_API_KEY", help="Coinmarketcap API key")
|
||||
@click.option("-v", "--verbose", is_flag=True, help="Display more info")
|
||||
# fmt: on
|
||||
def main(refresh, api_key, verbose):
|
||||
# setup logging
|
||||
log_level = logging.DEBUG if verbose else logging.WARNING
|
||||
root = logging.getLogger()
|
||||
root.setLevel(log_level)
|
||||
handler = logging.StreamHandler(sys.stdout)
|
||||
handler.setLevel(log_level)
|
||||
root.addHandler(handler)
|
||||
|
||||
marketcap.init(api_key, refresh=refresh)
|
||||
|
||||
defs, _ = coin_info.coin_info_with_duplicates()
|
||||
support_info = coin_info.support_info(defs)
|
||||
|
||||
coins = {}
|
||||
coins.update(update_bitcoin(defs.bitcoin, support_info))
|
||||
coins.update(update_erc20(defs.erc20, defs.eth, support_info))
|
||||
coins.update(update_ethereum_networks(defs.eth, support_info))
|
||||
coins.update(update_nem_mosaics(defs.nem, support_info))
|
||||
coins.update(update_simple(defs.misc, support_info, "coin"))
|
||||
|
||||
apply_overrides(coins)
|
||||
finalize_wallets(coins)
|
||||
update_marketcaps(coins)
|
||||
|
||||
check_missing_data(coins)
|
||||
|
||||
info = summary(coins, api_key)
|
||||
details = dict(coins=coins, info=info)
|
||||
|
||||
print(json.dumps(info, sort_keys=True, indent=4))
|
||||
with open(os.path.join(coin_info.DEFS_DIR, "coins_details.json"), "w") as f:
|
||||
json.dump(details, f, sort_keys=True, indent=4)
|
||||
f.write("\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -1,59 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import tempfile
|
||||
|
||||
import click
|
||||
import requests
|
||||
|
||||
LIVE_URL = "https://trezor.io/static/json/coins_details.json"
|
||||
COINS_DETAILS = os.path.join(
|
||||
os.path.dirname(__file__), "..", "defs", "coins_details.json"
|
||||
)
|
||||
|
||||
|
||||
def diffize_file(coins_details, tmp):
|
||||
coins_list = list(coins_details["coins"].values())
|
||||
for coin in coins_list:
|
||||
coin.pop("marketcap_usd", None)
|
||||
links = coin.get("links", {})
|
||||
wallets = coin.get("wallet", {})
|
||||
for link in links:
|
||||
links[link] = links[link].rstrip("/")
|
||||
for wallet in wallets:
|
||||
wallet["url"] = wallet["url"].rstrip("/")
|
||||
|
||||
if not coin.get("wallet"):
|
||||
coin.pop("wallet", None)
|
||||
|
||||
coins_list.sort(key=lambda c: c["name"])
|
||||
|
||||
for coin in coins_list:
|
||||
name = coin["name"]
|
||||
for key in coin:
|
||||
print(name, "\t", key, ":", coin[key], file=tmp)
|
||||
tmp.flush()
|
||||
|
||||
|
||||
@click.command()
|
||||
def cli():
|
||||
"""Compare data from trezor.io/coins with current coins_details.json
|
||||
|
||||
Shows a nicely formatted diff between the live version and the trezor-common
|
||||
version. Useful for catching auto-generation problems, etc.
|
||||
"""
|
||||
live_json = requests.get(LIVE_URL).json()
|
||||
with open(COINS_DETAILS) as f:
|
||||
coins_details = json.load(f)
|
||||
|
||||
Tmp = tempfile.NamedTemporaryFile
|
||||
with Tmp("w") as tmpA, Tmp("w") as tmpB:
|
||||
diffize_file(live_json, tmpA)
|
||||
diffize_file(coins_details, tmpB)
|
||||
subprocess.call(["diff", "-u", "--color=auto", tmpA.name, tmpB.name])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
cli()
|
||||
@ -43,6 +43,7 @@ else:
|
||||
|
||||
DEFINITIONS_CACHE_FILEPATH = pathlib.Path("definitions-cache.json")
|
||||
|
||||
|
||||
# ====== utils ======
|
||||
|
||||
|
||||
|
||||
@ -79,30 +79,6 @@ def init(api_key, refresh=None):
|
||||
COINS_SEARCHABLE = data_searchable
|
||||
|
||||
|
||||
def get_coin(coin):
|
||||
if coin["type"] == "erc20":
|
||||
address = coin["address"].lower()
|
||||
return COINS_SEARCHABLE.get(address)
|
||||
|
||||
data = None
|
||||
if "coinmarketcap_alias" in coin:
|
||||
data = COINS_SEARCHABLE.get(coin["coinmarketcap_alias"])
|
||||
if data is None:
|
||||
slug = coin["name"].replace(" ", "-").lower()
|
||||
data = COINS_SEARCHABLE.get(slug)
|
||||
if data is None:
|
||||
data = COINS_SEARCHABLE.get(coin["shortcut"].lower())
|
||||
return data
|
||||
|
||||
|
||||
def marketcap(coin):
|
||||
data = get_coin(coin)
|
||||
if data is None:
|
||||
return None
|
||||
|
||||
return int(data["quote"]["USD"]["market_cap"])
|
||||
|
||||
|
||||
def fiat_price(coin_symbol):
|
||||
data = COINS_SEARCHABLE.get(coin_symbol)
|
||||
if data is None:
|
||||
|
||||
@ -31,6 +31,4 @@ make -C $HERE/../.. gen
|
||||
|
||||
diff $CHECK_OUTPUT/pre.txt $CHECK_OUTPUT/post.txt
|
||||
|
||||
$HERE/coins_details.py
|
||||
|
||||
$HERE/ethereum_definitions.py prepare-definitions
|
||||
|
||||
@ -31,8 +31,10 @@
|
||||
- [Passphrase](common/communication/passphrase.md)
|
||||
- [Migration](common/communication/passphrase-redesign-migration.md)
|
||||
- [Bitcoin signing](common/communication/bitcoin-signing.md)
|
||||
- [Ethereum definitions binary format](common/communication/ethereum-definitions-binary-format.md)
|
||||
- [Reproducible builds](common/reproducible-build.md)
|
||||
- [Message Workflows](common/message-workflows.md)
|
||||
- [Ethereum definitions](common/ethereum-definitions.md)
|
||||
- [Storage](storage/index.md)
|
||||
- [Tests](tests/index.md)
|
||||
- [Device Tests](tests/device-tests.md)
|
||||
|
||||
@ -0,0 +1,13 @@
|
||||
# Ethereum definition binary format
|
||||
|
||||
Definitions are binary encoded and have a special format:
|
||||
1. prefix:
|
||||
1. format version of the definition (UTF-8 string `trzd` + version number, padded with zeroes if shorter, 8 bytes)
|
||||
2. type of data (unsigned integer, 1 byte)
|
||||
3. data version of the definition (unsigned integer, 4 bytes)
|
||||
4. length of the encoded protobuf message - payload length in bytes (unsigned integer, 2 bytes)
|
||||
2. payload: serialized form of protobuf message EthereumNetworkInfo or EthereumTokenInfo (N bytes)
|
||||
3. suffix:
|
||||
1. length of the Merkle tree proof - number of hashes in the proof (unsigned integer, 1 byte)
|
||||
2. proof - individual hashes used to compute Merkle tree root hash (plain bits, N*32 bytes)
|
||||
3. signed Merkle tree root hash (plain bits, 64 bytes)
|
||||
@ -8,6 +8,10 @@ We use [Protobuf v2](https://developers.google.com/protocol-buffers/) for host-d
|
||||
|
||||
Protobuf messages are defined in the [Common](https://github.com/trezor/trezor-firmware/tree/master/common) project, which is part of this monorepo. This repository is also exported to [trezor/trezor-common](https://github.com/trezor/trezor-common) to be used by third parties, which prefer not to include the whole monorepo. That copy is read-only mirror and all changes are happening in this monorepo.
|
||||
|
||||
## Ethereum definitions binary format
|
||||
|
||||
Ethereum definition binary format is described in [short document](ethereum-definitions-binary-format.md).
|
||||
|
||||
## Notable topics
|
||||
|
||||
- [Sessions](sessions.md)
|
||||
|
||||
176
docs/common/ethereum-definitions.md
Normal file
176
docs/common/ethereum-definitions.md
Normal file
@ -0,0 +1,176 @@
|
||||
# Ethereum definitions
|
||||
|
||||
Ethereum definitions for networks (chains) and tokens are dynamically generated and
|
||||
encoded into binary blobs. These blobs could be send as a part of some messages
|
||||
to the device.
|
||||
|
||||
## Built-in definitions
|
||||
|
||||
In addition to generated binary blobs, small subset of the definitions is also hardcoded
|
||||
in firmware in a decoded form (not as a binary blob).
|
||||
Location of these definitions is for:
|
||||
* networks - [`networks.json`](https://github.com/trezor/trezor-firmware/blob/master/common/defs/ethereum/networks.json)
|
||||
* tokens - [`tokens.json`](https://github.com/trezor/trezor-firmware/blob/master/common/defs/ethereum/tokens.json)
|
||||
|
||||
Built-in definitions are written by hand and are not subject to any generation described
|
||||
below.
|
||||
|
||||
## External definitions
|
||||
|
||||
Generated binary blobs are first saved locally (by the script that generates them)
|
||||
and then published online. By "external" is meant definitions not hardcoded in firmware.
|
||||
|
||||
#### File structure
|
||||
Saved definitions (binary blobs) are stored in directory (e.g. `definitions-latest/`)
|
||||
with specific file structure:
|
||||
````
|
||||
definitions-latest/
|
||||
├── by_chain_id/
|
||||
│ ├── "CHAIN_ID"/
|
||||
│ │ ├── network.dat
|
||||
│ │ ├── token_"TOKEN_ADDRESS".dat
|
||||
│ │ ├── token_"TOKEN_ADDRESS".dat
|
||||
│ │ ...
|
||||
│ ├── "CHAIN_ID"/
|
||||
│ │ ├── network.dat
|
||||
│ │ ├── token_"TOKEN_ADDRESS".dat
|
||||
│ │ ├── token_"TOKEN_ADDRESS".dat
|
||||
│ │ ...
|
||||
│ ...
|
||||
└── by_slip44/
|
||||
├── "SLIP44_ID"/
|
||||
│ └── network.dat
|
||||
├── "SLIP44_ID"/
|
||||
│ └── network.dat
|
||||
...
|
||||
````
|
||||
where:
|
||||
* `CHAIN_ID` is a corresponding chain ID of included network/tokens
|
||||
* `SLIP44_ID` is a corresponding SLIP44 ID of included network
|
||||
* `TOKEN_ADDRESS` is a lowercase token address (stripped of `0x` prefix)
|
||||
|
||||
Notice that token definitions are only accessible by `CHAIN_ID` and `TOKEN_ADDRESS`
|
||||
(directory `by_chain_id`), not by `SLIP44_ID` (directory `by_slip44`).
|
||||
|
||||
#### Definitions online
|
||||
|
||||
Generated binary definitions are available online at [publicly accessible website](https://data.trezor.io/eth_definitions) # TODO: update url.
|
||||
|
||||
To get the desired definition (one at a time), URL can be composed in multiple ways
|
||||
and the structure is the same as it is described in the [file structure](#file-structure)
|
||||
section. Base URL format is `https://data.trezor.io/eth_definitions/LOOKUP_TYPE/ID/NAME`
|
||||
where:
|
||||
* `LOOKUP_TYPE` is one of `by_chain_id` or `by_slip44`
|
||||
* `ID` is either chain ID or SLIP44 ID (depends on the chosen lookup type)
|
||||
* `NAME` is either:
|
||||
* `network.dat` for network definition at given chain ID or SLIP44 ID or
|
||||
* `token_"TOKEN_ADDRESS".dat` for token definition at given chain ID and token address
|
||||
(see [file structure](#file-structure) section on how to format the token address)
|
||||
|
||||
Definitions could be also downloaded by one request in a ZIP file at https://data.trezor.io/eth_definitions/definitions.zip # TODO: update url.
|
||||
|
||||
#### `Trezorctl`
|
||||
|
||||
Automatic manipulation with the definitions is implemented in `trezorctl` tool.
|
||||
All Ethereum commands that do work with definitions contains contains options
|
||||
to automatically download online definitions or use locally stored definitions.
|
||||
For more info look at the `trezorctl` [documentation]
|
||||
(https://github.com/trezor/trezor-firmware/blob/master/python/docs/README.rst).
|
||||
|
||||
## Process of generating the definitions
|
||||
|
||||
Binary definitions are generated by one script -
|
||||
[`ethereum_definitions.py`](https://github.com/trezor/trezor-firmware/blob/master/common/tools/ethereum_definitions.py)
|
||||
|
||||
The process is composed of multiple stages described in the following sections.
|
||||
|
||||
### 1. Prepare the definitions to JSON file
|
||||
|
||||
Preparation stage starts with collection of all data from all [data sources](#data-sources)
|
||||
(or from locally stored cache) and creation of connections between the data (finding
|
||||
the same IDs, etc.). In case that no cache was found at the beginning the cache file
|
||||
is saved (by default `definitions-cache.json`).
|
||||
|
||||
Every definition is checked for the size limitations and user can decide (if he choosed
|
||||
the interactive mode) what should happen if the size of some field is bigger than
|
||||
the allowed maximum. This is needed due to size restrictions on Model 1 (buffers with fixed
|
||||
size).
|
||||
|
||||
Subsequently all the definition are checked for duplicates and user has to decide
|
||||
which ones will be removed. No duplicates are allowed in further processing.
|
||||
|
||||
If there are already locally stored definitions in JSON file the comparison takes place. Simple
|
||||
comparison algorithm looks for these types of changes:
|
||||
* moved definition - definition was moved to other chain ID or address (in case of tokens)
|
||||
* modified definition - definition has same chain ID or address but other fields changed
|
||||
* deleted definition - definition was deleted
|
||||
* resurrected definition - previously deleted definition is available again
|
||||
* changes in symbol - symbol has changed in the definition
|
||||
|
||||
Results of the comparison are printed out.
|
||||
|
||||
After solving all the collisions, size limitations and changes we have "clean" data
|
||||
saved in a local file (by default `definitions-latest.json`).
|
||||
|
||||
### 2. Generate and sign the definitions
|
||||
|
||||
Input for this stage is the JSON file from previous stage with definitions.
|
||||
For the purpose of [verifying the definitions on FW side](#verification-and-validation-on-fw-side)
|
||||
we use Merkle Tree data structure, which gives us the option to effectively split
|
||||
this stage into three sub-stages:
|
||||
1. Getting the Merkle tree root hash - Merkle tree is build from all the definitions
|
||||
and the root hash is computed.
|
||||
2. Signing the hash - computed Merkle tree hash is signed using private key.
|
||||
3. Generating the binary definitions - Merkle tree is build from all the definitions
|
||||
and the root hash is computed again. Then this hash is verified against signed hash
|
||||
from previous step using public key to ensure that nothing has changed. If everything
|
||||
is ok the binary encoded (see
|
||||
[definition binary format](communication/ethereum-definitions-binary-format.md) section)
|
||||
definitions are generated to directory with specific [file structure](#file-structure).
|
||||
|
||||
### 3. Publish the definitions
|
||||
|
||||
Binary definitions are published to our website. See section [definitions online](#definitions-online)
|
||||
for more information.
|
||||
|
||||
## Data sources
|
||||
|
||||
External Ethereum definitions are generated based on data from external APIs and repositories:
|
||||
* [`coingecko.com`](https://www.coingecko.com/) for most of the info about networks and tokens
|
||||
* [defillama](https://defillama.com/) to pair as much networks as we can to CoinGecko ID
|
||||
* [Ethereum Lists - chains](https://github.com/ethereum-lists/chains) as the only source of EVM-based networks
|
||||
* [Ethereum Lists - tokens](https://github.com/ethereum-lists/tokens) as another source of tokens
|
||||
|
||||
## Validation and verification on FW side
|
||||
|
||||
To ensure that the definitions send to device are genuine we have to check them
|
||||
on receiving.
|
||||
|
||||
#### Validation
|
||||
|
||||
First thing that happens when binary definition is received is validation.Validation
|
||||
is a single process when FW compares hardcoded values against values found in received
|
||||
definition.
|
||||
|
||||
Based on the [definition binary format](communication/ethereum-definitions-binary-format.md)
|
||||
FW checks the following values:
|
||||
* format version - format version found in definiton has to be equal or higher than
|
||||
the one specified in FW
|
||||
* type of data - type of data found in definiton has to be the same as FW expects
|
||||
* data version - data version found in definiton has to be equal or higher than
|
||||
the one specified in FW
|
||||
|
||||
If any of these checks fail the definition is rejected.
|
||||
|
||||
#### Verification
|
||||
|
||||
Verification of received definitions is made using Merkle tree data structure
|
||||
in combination with a signed Merkle tree root hash. Every encoded definition
|
||||
is packed with list of Merkle tree proofs and the signed Merkle tree root hash
|
||||
(see [definition binary format](communication/ethereum-definitions-binary-format.md)
|
||||
section).
|
||||
|
||||
When the definition is received, hash from the combination of the encoded definition
|
||||
itself and the list of proofs is computed. This hash is then verified against signed
|
||||
hash included in received definition using the public key hardcoded in FW. This last
|
||||
step ensures that nothing has changed in the definition.
|
||||
@ -6,6 +6,10 @@ Common contains files shared among Trezor projects.
|
||||
|
||||
JSON coin definitions and support tables.
|
||||
|
||||
## Ethereum definitions
|
||||
|
||||
Description of Ethereum definitions and the process of their generation. See [Ethereum definitions](ethereum-definitions.md).
|
||||
|
||||
## Protobuf Definitions
|
||||
|
||||
Common Protobuf definitions for the Trezor protocol. Also see [Communication](communication/index.md).
|
||||
|
||||
Loading…
Reference in New Issue
Block a user