feat(common): switch to external Ethereum definitions, add generators

pull/2914/head
Martin Novák 2 years ago committed by matejcik
parent 4a39df2847
commit f44ef58acb

6
.gitmodules vendored

@ -8,9 +8,6 @@
[submodule "vendor/secp256k1-zkp"]
path = vendor/secp256k1-zkp
url = https://github.com/bitcoin-core/secp256k1.git
[submodule "common/defs/ethereum/tokens"]
path = common/defs/ethereum/tokens
url = https://github.com/ethereum-lists/tokens.git
[submodule "crypto/tests/wycheproof"]
path = crypto/tests/wycheproof
url = https://github.com/google/wycheproof
@ -24,6 +21,3 @@
[submodule "vendor/fido2-tests"]
path = vendor/fido2-tests
url = https://github.com/trezor/fido2-tests.git
[submodule "common/defs/ethereum/chains"]
path = common/defs/ethereum/chains
url = https://github.com/ethereum-lists/chains

@ -33,19 +33,22 @@ Testnet is considered a separate coin, so it must have its own JSON and icon.
We will not support coins that have `address_type` 0, i.e., same as Bitcoin.
#### `eth`
#### `eth` and `erc20`
The file [`ethereum/networks.json`](ethereum/networks.json) has a list of descriptions
of Ethereum networks. Each network must also have a PNG icon in `ethereum/<chain>.png`
file.
Definitions for Ethereum chains (networks) and tokens (erc20) are split in two parts:
#### `erc20`
1. built-in definitions - some of the chain and token definitions are built into the firmware
image. List of built-in chains is stored in [`ethereum/networks.json`](ethereum/networks.json)
and tokens in [`ethereum/tokens.json`](ethereum/tokens.json).
2. external definitions - dynamically generated from multiple sources. Whole process is
described in separate
[document](https://docs.trezor.io/trezor-firmware/common/ethereum-definitions.html).
`ethereum/tokens` is a submodule linking to [Ethereum Lists](https://github.com/ethereum-lists/tokens)
project with descriptions of ERC20 tokens. If you want to add or update a token
definition in Trezor, you need to get your change to the `tokens` repository first.
Trezor will only support tokens that have a unique symbol.
We generally do not accept updates to the built-in definitions. Instead, make sure your
network or token is included in the external definitions. A good place to start is the
[`ethereum-lists` GitHub organization](https://gitub.com/ethereum-lists): add your token
to the [tokens](https://github.com/ethereum-lists/tokens) repository, or your EVM chain to the
[chains](https://github.com/ethereum-lists/chains) repository.
#### `nem`
@ -57,82 +60,32 @@ Supported coins that are not derived from Bitcoin, Ethereum or NEM are currently
and listed in separate file [`misc/misc.json`](misc/misc.json). Each coin must also have
an icon in `misc/<short>.png`, where `short` is lowercased `shortcut` field from the JSON.
## Keys
### Keys
Throughout the system, coins are identified by a _key_ - a colon-separated string
generated from the coin's type and shortcut:
* for Bitcoin-likes, key is `bitcoin:XYZ`
* for Ethereum networks, key is `eth:XYZ`
* for ERC20 tokens, key is `erc20:<chain>:XYZ`
* for NEM mosaic, key is `nem:XYZ`
* for others, key is `misc:XYZ`
* for Bitcoin-likes, key is `bitcoin:<shortcut>`
* for Ethereum networks, key is `eth:<shortcut>`
* for ERC20 tokens, key is `erc20:<chain_symbol>:<token_shortcut>`
* for NEM mosaic, key is `nem:<shortcut>`
* for others, key is `misc:<shortcut>`
If a token shortcut has a suffix, such as `CAT (BlockCat)`, the whole thing is part
of the key (so the key is `erc20:eth:CAT (BlockCat)`).
Sometimes coins end up with duplicate symbols, which in case of ERC20 tokens leads to
key collisions. We do not allow duplicate symbols in the data, so this doesn't affect
everyday use (see below). However, for validation purposes, it is sometimes useful
to work with unfiltered data that includes the duplicates. In such cases, keys are
deduplicated by adding a counter at end, e.g.: `erc20:eth:SMT:0`, `erc20:eth:SMT:1`.
Note that the suffix _is not stable_, so these coins can't be reliably uniquely identified.
## Duplicate Detection
**Duplicate symbols are not allowed** in our data. Tokens that have symbol collisions
are removed from the data set before processing. The duplicate status is mentioned
in `support.json` (see below), but it is impossible to override from there.
Duplicate detection works as follows:
1. a _symbol_ is split off from the shortcut string. E.g., for `CAT (BlockCat)`, symbol
is just `CAT`. It is compared, case-insensitive, with other coins (so `WIC` and `WiC`
are considered the same symbol), and identical symbols are put into a _bucket_.
2. if _all_ coins in the bucket also have a suffix (`CAT (BlockCat)` and `CAT (BitClave)`),
they are _not_ considered duplicate.
3. if _any_ coin in the bucket does _not_ have a suffix (`MIT` and `MIT (Mychatcoin)`),
all coins in the bucket are considered duplicate.
4. Duplicate tokens (coins from the `erc20` group) are automatically removed from data.
Duplicate non-tokens are marked but not removed. For instance, `bitcoin:FTC` (Feathercoin)
and `erc20:eth:FTC` (FTC) are duplicate, and `erc20:eth:FTC` is removed.
5. If two non-tokens collide with each other, it is an error that fails the CI build.
The file [`duplicity_overrides.json`](duplicity_overrides.json) can override detection
results: keys set to `true` are considered duplicate (in a separate bucket), keys set
to `false` are considered non-duplicate even if auto-detected. This is useful for
whitelisting a supported token explicitly, or blacklisting things that the detection
can't match (for instance "Battle" and "Bitlle" have suffixes, but they are too similar).
External contributors should not make changes to `duplicity_overrides.json`, unless
asked to.
Duplicate keys are not allowed and coins that would result in duplicate keys cannot be
added to the dataset.
You can use `./tools/cointool.py check -d all` to inspect duplicate detection in detail.
# Coins Details
The file [`coins_details.json`](coins_details.json) is a list of all known coins
with support status, market cap information and relevant links. This is the source
file for https://trezor.io/coins.
You should never make changes to `coins_details.json` directly. Use `./tools/coins_details.py`
to regenerate it from known data.
If you need to change information in this file, modify the source information instead -
one of the JSON files in the groups listed above, support info in `support.json`, or
make a pull request to the tokens repository.
## Wallet URLs
If you want to add a **wallet link**, modify the file [`wallets.json`](wallets.json).
If this is not viable for some reason, or if there is no source information ,
you can also edit [`coins_details.override.json`](coins_details.override.json).
External contributors should not touch this file unless asked to.
# Support Information
We keep track of support status of each coin over our devices. That is
We keep track of support status of each built-in coin over our devices. That is
`trezor1` for Trezor One, `trezor2` for Trezor T, `connect` for [Connect](https://github.com/trezor/connect)
and `suite` for [Trezor Suite](https://suite.trezor.io/). In further description, the word "device"
applies to Connect and Suite as well.

@ -1,45 +0,0 @@
{
"erc20:eth:BAT": {
"name": "Basic Attention Token"
},
"erc20:eth:LINK (Chainlink)": {
"name": "Chainlink"
},
"erc20:eth:SOL": {
"shortcut": "SOLA"
},
"eth:CLO": {
"coinmarketcap_alias": "callisto-network"
},
"eth:ESN": {
"coinmarketcap_alias": "ethersocial"
},
"nem:DIMTOK": {
"coinmarketcap_alias": "dimcoin"
},
"eth:AUX": {
"links": {
"Github": "https://github.com/auxiliumglobal"
},
"wallet": {
"MyEtherWallet": null
}
},
"eth:EOS": {
"hidden": true,
"ignore_cmc_rank": true,
"reason": "this exists as misc:EOS and the eth: entry is probably a mistake"
},
"eth:XDC": {
"wallet": {
"MyCrypto": null,
"MyEtherWallet": null,
"XDC Wallet": "https://wallet.xinfin.network"
}
},
"misc:LSK": {
"hidden": true,
"ignore_cmc_rank": true,
"reason": "delisted incompatible hardfork"
}
}

@ -1,38 +1,3 @@
{
"erc20:eth:BTL (Battle)": true,
"erc20:eth:BTL (Bitlle)": true,
"erc20:eth:LINK Platform": true,
"erc20:eth:NXX": false,
"erc20:eth:SNX:c011": false,
"erc20:eth:TUSD": false,
"erc20:eth:Hdp": true,
"erc20:eth:Hdp.ф": true,
"erc20:eth:HEX:2b59": false,
"erc20:eth:JOB:dfbc": false,
"misc:BNB": false,
"eth:BNB": false,
"eth:ONE:1666600000": false,
"eth:ONE:1666600001": false,
"eth:ONE:1666600002": false,
"eth:ONE:1666600003": false,
"eth:tGOR:5": false,
"eth:tGOR:420": false,
"eth:tCELO:44787": false,
"eth:tCELO:62320": false,
"eth:QKC:100000": false,
"eth:QKC:100001": false,
"eth:QKC:100002": false,
"eth:QKC:100003": false,
"eth:QKC:100004": false,
"eth:QKC:100005": false,
"eth:QKC:100006": false,
"eth:QKC:100007": false,
"eth:QKC:100008": false,
"eth:xDAI:100": false,
"eth:xDAI:200": false,
"eth:CPAY:3000": false,
"eth:CPAY:3001": false,
"eth:CPAY:21337": false,
"erc20:eth:USDT": false,
"erc20:avax:USDT": false
"misc:BNB": false
}

@ -1 +0,0 @@
Subproject commit 805ae42ecc53aa6493949b1e9c1da41e036c1845

@ -0,0 +1,62 @@
[
{
"chain": "eth",
"chain_id": 1,
"coingecko_id": "ethereum",
"is_testnet": false,
"name": "Ethereum",
"shortcut": "ETH",
"slip44": 60
},
{
"chain": "rop",
"chain_id": 3,
"is_testnet": true,
"name": "Ropsten",
"shortcut": "tROP",
"slip44": 1
},
{
"chain": "rin",
"chain_id": 4,
"is_testnet": true,
"name": "Rinkeby",
"shortcut": "tRIN",
"slip44": 1
},
{
"chain": "gor",
"chain_id": 5,
"is_testnet": true,
"name": "Görli",
"shortcut": "tGOR",
"slip44": 1
},
{
"chain": "bnb",
"chain_id": 56,
"coingecko_id": "binance-smart-chain",
"is_testnet": false,
"name": "Binance Smart Chain",
"shortcut": "BNB",
"slip44": 714
},
{
"chain": "etc",
"chain_id": 61,
"coingecko_id": "ethereum-classic",
"is_testnet": false,
"name": "Ethereum Classic",
"shortcut": "ETC",
"slip44": 61
},
{
"chain": "MATIC",
"chain_id": 137,
"coingecko_id": "polygon-pos",
"is_testnet": false,
"name": "Polygon",
"shortcut": "MATIC",
"slip44": 966
}
]

@ -1 +0,0 @@
Subproject commit 0eeaf9b9f13b5e6538da26d079e2b968dc8bb23f

@ -0,0 +1,189 @@
{
"1;eth": [
{
"address": "0xdac17f958d2ee523a2206206994597c13d831ec7",
"coingecko_id": "tether",
"decimals": 6,
"name": "Tether",
"shortcut": "USDT"
},
{
"address": "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48",
"coingecko_id": "usd-coin",
"decimals": 6,
"name": "USD Coin",
"shortcut": "USDC"
},
{
"address": "0x4fabb145d64652a948d72533023f6e7a623c7c53",
"coingecko_id": "binance-usd",
"decimals": 18,
"name": "Binance USD",
"shortcut": "BUSD"
},
{
"address": "0x95ad61b0a150d79219dcf64e1e6cc01f0b64c4ce",
"coingecko_id": "shiba-inu",
"decimals": 18,
"name": "Shiba Inu",
"shortcut": "SHIB"
},
{
"address": "0x6b175474e89094c44da98b954eedeac495271d0f",
"coingecko_id": "dai",
"decimals": 18,
"name": "Dai",
"shortcut": "DAI"
},
{
"address": "0x7d1afa7b718fb893db30a3abc0cfc608aacfebb0",
"coingecko_id": "matic-network",
"decimals": 18,
"name": "Polygon",
"shortcut": "MATIC"
},
{
"address": "0xae7ab96520de3a18e5e111b5eaab095312d7fe84",
"coingecko_id": "staked-ether",
"decimals": 18,
"name": "Lido Staked Ether",
"shortcut": "STETH"
},
{
"address": "0x1f9840a85d5af5bf1d1762f925bdaddc4201f984",
"coingecko_id": "uniswap",
"decimals": 18,
"name": "Uniswap",
"shortcut": "UNI"
},
{
"address": "0x2260fac5e5542a773aa44fbcfedf7c193bc2c599",
"coingecko_id": "wrapped-bitcoin",
"decimals": 8,
"name": "Wrapped Bitcoin",
"shortcut": "WBTC"
},
{
"address": "0x75231f58b43240c9718dd58b4967c5114342a86c",
"coingecko_id": "okb",
"decimals": 18,
"name": "OKB",
"shortcut": "OKB"
},
{
"address": "0x2af5d2ad76741191d15dfe7bf6ac92d4bd912ca3",
"coingecko_id": "leo-token",
"decimals": 18,
"name": "LEO Token",
"shortcut": "LEO"
},
{
"address": "0x514910771af9ca656af840dff83e8264ecf986ca",
"coingecko_id": "chainlink",
"decimals": 18,
"name": "Chainlink",
"shortcut": "LINK"
},
{
"address": "0x50d1c9771902476076ecfc8b2a83ad6b9355a4c9",
"coingecko_id": "ftx-token",
"decimals": 18,
"name": "FTX",
"shortcut": "FTT"
},
{
"address": "0xa0b73e1ff0b80914ab6fe0444e65848c4c34450b",
"coingecko_id": "crypto-com-chain",
"decimals": 8,
"name": "Cronos",
"shortcut": "CRO"
},
{
"address": "0x4a220e6096b25eadb88358cb44068a3248254675",
"coingecko_id": "quant-network",
"decimals": 18,
"name": "Quant",
"shortcut": "QNT"
},
{
"address": "0x4d224452801aced8b2f0aebe155379bb5d594381",
"coingecko_id": "apecoin",
"decimals": 18,
"name": "ApeCoin",
"shortcut": "APE"
},
{
"address": "0xa2cd3d43c775978a96bdbf12d733d5a1ed94fb18",
"coingecko_id": "chain-2",
"decimals": 18,
"name": "Chain",
"shortcut": "XCN"
},
{
"address": "0x853d955acef822db058eb8505911ed77f175b99e",
"coingecko_id": "frax",
"decimals": 18,
"name": "Frax",
"shortcut": "FRAX"
},
{
"address": "0x3845badade8e6dff049820680d1f14bd3903a5d0",
"coingecko_id": "the-sandbox",
"decimals": 18,
"name": "The Sandbox",
"shortcut": "SAND"
},
{
"address": "0x0f5d2fb29fb7d3cfee444a200298f468908cc942",
"coingecko_id": "decentraland",
"decimals": 18,
"name": "Decentraland",
"shortcut": "MANA"
},
{
"address": "0xbb0e17ef65f82ab018d8edd776e8dd940327b28b",
"coingecko_id": "axie-infinity",
"decimals": 18,
"name": "Axie Infinity",
"shortcut": "AXS"
},
{
"address": "0x3506424f91fd33084466f402d5d97f05f8e3b4af",
"coingecko_id": "chiliz",
"decimals": 18,
"name": "Chiliz",
"shortcut": "CHZ"
},
{
"address": "0x7fc66500c84a76ad7e9c93437bfc5ac33e2ddae9",
"coingecko_id": "aave",
"decimals": 18,
"name": "Aave",
"shortcut": "AAVE"
},
{
"address": "0x86fa049857e0209aa7d9e616f7eb3b3b78ecfdb0",
"decimals": 18,
"name": "EOS",
"shortcut": "EOS"
}
],
"56;bnb": [
{
"address": "0x0eb3a705fc54725037cc9e008bdede697f62f335",
"coingecko_id": "cosmos",
"decimals": 18,
"name": "Cosmos Hub",
"shortcut": "ATOM"
}
],
"137;MATIC": [
{
"address": "0x2c89bbc92bd86f8075d1decc58c7f4e0107f286b",
"coingecko_id": "avalanche-2",
"decimals": 18,
"name": "Avalanche",
"shortcut": "AVAX"
}
]
}

File diff suppressed because it is too large Load Diff

@ -120,16 +120,5 @@
},
"bitcoin:ZCR": {
"Electrum-ZCR": "https://github.com/zcore-coin/electrum-wallet/"
},
"eth:WAN": {
"Wanchain Wallet": "https://www.wanchain.org/getstarted/"
},
"eth:AUX": {
"MyEtherWallet": null
},
"eth:XDC": {
"MyCrypto": null,
"MyEtherWallet": null,
"XDC Wallet": "https://wallet.xinfin.network"
}
}

@ -21,8 +21,6 @@ the following commands:
* **`check`**: check validity of json definitions and associated data. Used in CI.
* **`dump`**: dump coin information, including support status, in JSON format. Various
filtering options are available, check help for details.
* **`coindefs`**: generate signed protobuf descriptions of coins. This is for future use
and could allow us to not need to store coin data in Trezor itself.
Use `cointool.py command --help` to get more information on each command.
@ -41,18 +39,6 @@ The following commands are available:
Use `support.py command --help` to get more information on each command.
### `coins_details.py`
Generates `coins_details.json`, source file for https://trezor.io/coins.
Collects data on coins, downloads market caps and puts everything into a single file.
Caches market cap data so you don't have to download it every time.
### `diffize_coins_details.py`
Compares generated `coins_details.json` to the released version currently served
on https://trezor.io/coins, in a format that is nicely readable to humans and
hard(er) to mess up by diff.
### `coin_info.py`
In case where code generation with `cointool.py render` is impractical or not sufficient,
@ -85,7 +71,7 @@ from the outside.
### `marketcap.py`
Module for obtaining market cap and price data used by `coins_details.py` and `maxfee.py`.
Module for obtaining market cap and price data used by `maxfee.py`.
### `maxfee.py`
@ -130,32 +116,20 @@ Or mark them as unsupported explicitly.
## Releasing a new firmware
#### **Step 1:** update the tokens repo
```sh
pushd defs/ethereum/tokens
git checkout master
git pull
popd
git add defs/ethereum/tokens
```
#### **Step 2:** run the release flow
#### **Step 1:** run the release script
```sh
./tools/support.py release 2
./tools/release.sh
```
The number `2` indicates that you are releasing Trezor 2. The version will be
automatically determined, based on currently released firmwares. Or you can explicitly
specify the version with `-r 2.1.0`.
All currently known unreleased ERC20 tokens are automatically set to the given version.
All coins marked _soon_ are set to the current version. This is automatic - coins that
**_Note that "soon" feature was already removed and following paragraph is deprecated._**
_All coins marked _soon_ are set to the current version. This is automatic - coins that
were marked _soon_ were used in code generation and so should be released. If you want
to avoid this, you will have to manually revert each coin to _soon_ status, either with
`support.py set`, or by manually editing `support.json`.
`support.py set`, or by manually editing `support.json`._
Coins in state _unknown_, i.e., coins that are known in the definitions but not listed
in support files, will be also added. But you will be interactively asked to confirm
@ -170,13 +144,7 @@ Use `-g` or `--git-tag` to automatically tag the current `HEAD` with a version,
XXX this should also commit the changes though, otherwise the tag will apply to the wrong
commit.
#### **Step 3:** review and commit your changes
#### **Step 2:** review and commit your changes
Use `git diff` to review changes made, commit and push. If you tagged the commit in the
previous step, don't forget to `git push --tags` too.
#### **Step 4:** update submodule in your target repository
Go to `trezor-core` or `trezor-mcu` checkout and update the submodule. Checkout the
appropriate tag if you created it. If you're in `trezor-core`, run `make templates`
to update source files.

@ -24,11 +24,7 @@ except ImportError:
log = logging.getLogger(__name__)
ROOT = Path(__file__).resolve().parent.parent
if os.environ.get("DEFS_DIR"):
DEFS_DIR = Path(os.environ.get("DEFS_DIR")).resolve()
else:
DEFS_DIR = ROOT / "defs"
DEFS_DIR = ROOT / "defs"
class SupportItemBool(TypedDict):
@ -107,9 +103,9 @@ class Coin(TypedDict):
icon: str
# Special ETH fields
coingecko_id: str
chain: str
chain_id: str
rskip60: bool
chain_id: int
url: str
# Special erc20 fields
@ -117,7 +113,6 @@ class Coin(TypedDict):
address: str
address_bytes: bytes
dup_key_nontoken: bool
deprecation: dict[str, str]
# Special NEM fields
ticker: str
@ -126,6 +121,7 @@ class Coin(TypedDict):
unsupported: bool
duplicate: bool
support: SupportInfoItem
is_testnet: bool
# Backend-oriented fields
blockchain_link: dict[str, Any]
@ -162,6 +158,10 @@ def load_json(*path: str | Path) -> Any:
return json.loads(file.read_text(), object_pairs_hook=OrderedDict)
def get_btc_testnet_status(name: str) -> bool:
return any((mark in name.lower()) for mark in ("testnet", "regtest"))
# ====== CoinsInfo ======
@ -325,7 +325,7 @@ def validate_btc(coin: Coin) -> list[str]:
if not coin["max_address_length"] >= coin["min_address_length"]:
errors.append("max address length must not be smaller than min address length")
if "testnet" in coin["coin_name"].lower() and coin["slip44"] != 1:
if coin["is_testnet"] and coin["slip44"] != 1:
errors.append("testnet coins must use slip44 coin type 1")
if coin["segwit"]:
@ -359,74 +359,48 @@ def _load_btc_coins() -> Coins:
shortcut=coin["coin_shortcut"],
key=f"bitcoin:{coin['coin_shortcut']}",
icon=str(file.with_suffix(".png")),
is_testnet=get_btc_testnet_status(coin["coin_label"]),
)
coins.append(coin)
return coins
def _load_ethereum_networks() -> Coins:
def _load_builtin_ethereum_networks() -> Coins:
"""Load ethereum networks from `ethereum/networks.json`"""
chains_path = DEFS_DIR / "ethereum" / "chains" / "_data" / "chains"
chains_data = load_json("ethereum", "networks.json")
networks: Coins = []
for chain in sorted(
chains_path.glob("eip155-*.json"),
key=lambda x: int(x.stem.replace("eip155-", "")),
):
chain_data = load_json(chain)
shortcut = chain_data["nativeCurrency"]["symbol"]
name = chain_data["name"]
title = chain_data.get("title", "")
is_testnet = "testnet" in name.lower() or "testnet" in title.lower()
if is_testnet:
slip44 = 1
else:
slip44 = chain_data.get("slip44", 60)
if is_testnet and not shortcut.lower().startswith("t"):
shortcut = "t" + shortcut
rskip60 = shortcut in ("RBTC", "TRBTC")
# strip out bullcrap in network naming
if "mainnet" in name.lower():
name = re.sub(r" mainnet.*$", "", name, flags=re.IGNORECASE)
network = dict(
chain=chain_data["shortName"],
chain_id=chain_data["chainId"],
slip44=slip44,
shortcut=shortcut,
name=name,
rskip60=rskip60,
url=chain_data["infoURL"],
key=f"eth:{shortcut}",
for chain_data in chains_data:
chain_data.update(
chain_id=chain_data["chain_id"],
key=f"eth:{chain_data['shortcut']}",
# is_testnet is present in the JSON
)
networks.append(cast(Coin, network))
networks.append(cast(Coin, chain_data))
return networks
def _load_erc20_tokens() -> Coins:
"""Load ERC20 tokens from `ethereum/tokens` submodule."""
networks = _load_ethereum_networks()
tokens: Coins = []
for network in networks:
chain = network["chain"]
def _load_builtin_erc20_tokens() -> Coins:
"""Load ERC20 tokens from `ethereum/tokens.json`."""
tokens_data = load_json("ethereum", "tokens.json")
all_tokens: Coins = []
chain_path = DEFS_DIR / "ethereum" / "tokens" / "tokens" / chain
for file in sorted(chain_path.glob("*.json")):
token: Coin = load_json(file)
for chain_id_and_chain, tokens in tokens_data.items():
chain_id, chain = chain_id_and_chain.split(";", maxsplit=1)
for token in tokens:
token.update(
chain=chain,
chain_id=network["chain_id"],
chain_id=int(chain_id),
address=token["address"].lower(),
address_bytes=bytes.fromhex(token["address"][2:]),
shortcut=token["symbol"],
key=f"erc20:{chain}:{token['symbol']}",
symbol=token["shortcut"],
key=f"erc20:{chain}:{token['shortcut']}",
is_testnet=False,
)
tokens.append(token)
all_tokens.append(cast(Coin, token))
return tokens
return all_tokens
def _load_nem_mosaics() -> Coins:
@ -434,7 +408,11 @@ def _load_nem_mosaics() -> Coins:
mosaics: Coins = load_json("nem/nem_mosaics.json")
for mosaic in mosaics:
shortcut = mosaic["ticker"].strip()
mosaic.update(shortcut=shortcut, key=f"nem:{shortcut}")
mosaic.update(
shortcut=shortcut,
key=f"nem:{shortcut}",
is_testnet=False,
)
return mosaics
@ -442,7 +420,10 @@ def _load_misc() -> Coins:
"""Loads miscellaneous networks from `misc/misc.json`"""
others: Coins = load_json("misc/misc.json")
for other in others:
other.update(key=f"misc:{other['shortcut']}")
other.update(
key=f"misc:{other['shortcut']}",
is_testnet=False,
)
return others
@ -493,10 +474,6 @@ def latest_releases() -> dict[str, Any]:
return latest
def is_token(coin: Coin) -> bool:
return coin["key"].startswith("erc20:")
def support_info_single(support_data: SupportData, coin: Coin) -> SupportInfoItem:
"""Extract a support dict from `support.json` data.
@ -554,10 +531,6 @@ def support_info(coins: Iterable[Coin] | CoinsInfo | dict[str, Coin]) -> Support
WALLET_SUITE = {"Trezor Suite": "https://suite.trezor.io"}
WALLET_NEM = {"Nano Wallet": "https://nemplatform.com/wallets/#desktop"}
WALLETS_ETH_3RDPARTY = {
"MyEtherWallet": "https://www.myetherwallet.com",
"MyCrypto": "https://mycrypto.com",
}
def get_wallet_data() -> WalletInfo:
@ -579,7 +552,6 @@ def _suite_support(coin: Coin, support: SupportInfoItem) -> bool:
def wallet_info_single(
support_data: SupportInfo,
eth_networks_support_data: SupportInfo,
wallet_data: WalletInfo,
coin: Coin,
) -> WalletItems:
@ -596,26 +568,16 @@ def wallet_info_single(
if key.startswith("bitcoin:"):
if _suite_support(coin, support_data[key]):
wallets.update(WALLET_SUITE)
elif key.startswith("eth:"):
if support_data[key]["suite"]:
wallets.update(WALLET_SUITE)
else:
wallets.update(WALLETS_ETH_3RDPARTY)
elif key.startswith("erc20:"):
if eth_networks_support_data[coin["chain"]]["suite"]:
wallets.update(WALLET_SUITE)
else:
wallets.update(WALLETS_ETH_3RDPARTY)
elif key.startswith("nem:"):
wallets.update(WALLET_NEM)
elif key.startswith("misc:"):
elif key.startswith(("eth:", "erc20:", "misc:")):
pass # no special logic here
else:
raise ValueError(f"Unknown coin category: {key}")
# Add wallets from `wallets.json`
# This must come last as it offers the ability to override existing wallets
# (for example with `"MyEtherWallet": null` we delete the MyEtherWallet from the coin)
# (for example with `"Trezor Suite": null` we delete the "Trezor Suite" from the coin)
wallets.update(wallet_data.get(key, {}))
# Removing potentially disabled wallets from the last step
@ -644,17 +606,9 @@ def wallet_info(coins: Iterable[Coin] | CoinsInfo | dict[str, Coin]) -> WalletIn
support_data = support_info(coins)
wallet_data = get_wallet_data()
# Needed to find out suitable wallets for all the erc20 coins (Suite vs 3rd party)
eth_networks = [coin for coin in coins if coin["key"].startswith("eth:")]
eth_networks_support_data = {
n["chain"]: support_data[n["key"]] for n in eth_networks
}
wallet: WalletInfo = {}
for coin in coins:
wallet[coin["key"]] = wallet_info_single(
support_data, eth_networks_support_data, wallet_data, coin
)
wallet[coin["key"]] = wallet_info_single(support_data, wallet_data, coin)
return wallet
@ -715,74 +669,7 @@ def apply_duplicity_overrides(coins: Coins) -> Coins:
return override_bucket
def deduplicate_erc20(buckets: CoinBuckets, networks: Coins) -> None:
"""Apply further processing to ERC20 duplicate buckets.
This function works on results of `mark_duplicate_shortcuts`.
Buckets that contain at least one non-token are ignored - symbol collisions
with non-tokens always apply.
Otherwise the following rules are applied:
1. If _all tokens_ in the bucket have shortcuts with distinct suffixes, e.g.,
`CAT (BitClave)` and `CAT (Blockcat)`, the bucket is cleared - all are considered
non-duplicate.
(If even one token in the bucket _does not_ have a distinct suffix, e.g.,
`MIT` and `MIT (Mychatcoin)`, this rule does not apply and ALL tokens in the bucket
are still considered duplicate.)
2. If there is only one "main" token in the bucket, the bucket is cleared.
That means that all other tokens must either be on testnets, or they must be marked
as deprecated, with a deprecation pointing to the "main" token.
"""
testnet_networks = {n["chain"] for n in networks if n["slip44"] == 1}
def clear_bucket(bucket: Coins) -> None:
# allow all coins, except those that are explicitly marked through overrides
for coin in bucket:
coin["duplicate"] = False
for bucket in buckets.values():
# Only check buckets that contain purely ERC20 tokens. Collision with
# a non-token is always forbidden.
if not all(is_token(c) for c in bucket):
continue
splits = (symbol_from_shortcut(coin["shortcut"]) for coin in bucket)
suffixes = {suffix for _, suffix in splits}
# if 1. all suffixes are distinct and 2. none of them are empty
if len(suffixes) == len(bucket) and all(suffixes):
clear_bucket(bucket)
continue
# protected categories:
testnets = [coin for coin in bucket if coin["chain"] in testnet_networks]
deprecated_by_same = [
coin
for coin in bucket
if "deprecation" in coin
and any(
other["address"] == coin["deprecation"]["new_address"]
for other in bucket
)
]
remaining = [
coin
for coin in bucket
if coin not in testnets and coin not in deprecated_by_same
]
if len(remaining) <= 1:
for coin in deprecated_by_same:
deprecated_symbol = "[deprecated] " + coin["symbol"]
coin["shortcut"] = coin["symbol"] = deprecated_symbol
coin["key"] += ":deprecated"
clear_bucket(bucket)
def deduplicate_keys(all_coins: Coins) -> None:
def find_duplicate_keys(all_coins: Coins) -> None:
dups: CoinBuckets = defaultdict(list)
for coin in all_coins:
dups[coin["key"]].append(coin)
@ -790,14 +677,8 @@ def deduplicate_keys(all_coins: Coins) -> None:
for coins in dups.values():
if len(coins) <= 1:
continue
for i, coin in enumerate(coins):
if is_token(coin):
coin["key"] += ":" + coin["address"][2:6].lower() # first 4 hex chars
elif "chain_id" in coin:
coin["key"] += ":" + str(coin["chain_id"])
else:
coin["key"] += f":{i}"
coin["dup_key_nontoken"] = True
coin = coins[0]
raise ValueError(f"Duplicate key {coin['key']}")
def fill_blockchain_links(all_coins: CoinsInfo) -> None:
@ -829,8 +710,8 @@ def collect_coin_info() -> CoinsInfo:
"""
all_coins = CoinsInfo(
bitcoin=_load_btc_coins(),
eth=_load_ethereum_networks(),
erc20=_load_erc20_tokens(),
eth=_load_builtin_ethereum_networks(),
erc20=_load_builtin_erc20_tokens(),
nem=_load_nem_mosaics(),
misc=_load_misc(),
)
@ -866,10 +747,8 @@ def coin_info_with_duplicates() -> tuple[CoinsInfo, CoinBuckets]:
coin_list = all_coins.as_list()
# generate duplicity buckets based on shortcuts
buckets = mark_duplicate_shortcuts(all_coins.as_list())
# apply further processing to ERC20 tokens, generate deprecations etc.
deduplicate_erc20(buckets, all_coins.eth)
# ensure the whole list has unique keys (taking into account changes from deduplicate_erc20)
deduplicate_keys(coin_list)
# ensure the whole list has unique keys
find_duplicate_keys(coin_list)
# apply duplicity overrides
buckets["_override"] = apply_duplicity_overrides(coin_list)
sort_coin_infos(all_coins)
@ -883,9 +762,6 @@ def coin_info() -> CoinsInfo:
Does not auto-delete duplicates. This should now be based on support info.
"""
all_coins, _ = coin_info_with_duplicates()
# all_coins["erc20"] = [
# coin for coin in all_coins["erc20"] if not coin.get("duplicate")
# ]
return all_coins

@ -1,331 +0,0 @@
#!/usr/bin/env python3
"""Fetch information about coins and tokens supported by Trezor and update it in coins_details.json."""
import json
import logging
import os
import sys
import time
import click
import coin_info
import marketcap
LOG = logging.getLogger(__name__)
OPTIONAL_KEYS = ("links", "notes", "wallet")
ALLOWED_SUPPORT_STATUS = ("yes", "no")
WALLETS = coin_info.load_json("wallets.json")
OVERRIDES = coin_info.load_json("coins_details.override.json")
VERSIONS = coin_info.latest_releases()
# automatic wallet entries
WALLET_SUITE = {"Trezor Suite": "https://suite.trezor.io"}
WALLET_NEM = {"Nano Wallet": "https://nemplatform.com/wallets/#desktop"}
WALLETS_ETH_3RDPARTY = {
"MyEtherWallet": "https://www.myetherwallet.com",
"MyCrypto": "https://mycrypto.com",
}
TREZOR_KNOWN_URLS = (
"https://suite.trezor.io",
"https://wallet.trezor.io",
)
def update_marketcaps(coins):
for coin in coins.values():
coin["marketcap_usd"] = marketcap.marketcap(coin) or 0
def summary(coins, api_key):
t1_coins = 0
t2_coins = 0
supported_marketcap = 0
for coin in coins.values():
if coin.get("hidden"):
continue
t1_enabled = coin["t1_enabled"] == "yes"
t2_enabled = coin["t2_enabled"] == "yes"
if t1_enabled:
t1_coins += 1
if t2_enabled:
t2_coins += 1
if t1_enabled or t2_enabled:
supported_marketcap += coin.get("marketcap_usd", 0)
total_marketcap = None
try:
ret = marketcap.call("global-metrics/quotes/latest", api_key)
total_marketcap = int(ret["data"]["quote"]["USD"]["total_market_cap"])
except Exception:
pass
marketcap_percent = 100 * supported_marketcap / total_marketcap
return dict(
updated_at=int(time.time()),
updated_at_readable=time.asctime(),
t1_coins=t1_coins,
t2_coins=t2_coins,
marketcap_usd=supported_marketcap,
total_marketcap_usd=total_marketcap,
marketcap_supported=f"{marketcap_percent:.02f} %",
)
def _is_supported(support, trezor_version):
# True or version string means YES
# False or None means NO
return "yes" if support.get(trezor_version) else "no"
def _suite_support(coin, support):
"""Check the "suite" support property.
If set, check that at least one of the backends run on trezor.io.
If yes, assume we support the coin in our wallet.
Otherwise it's probably working with a custom backend, which means don't
link to our wallet.
"""
if not support.get("suite"):
return False
return any(".trezor.io" in url for url in coin["blockbook"])
def dict_merge(orig, new):
if isinstance(new, dict) and isinstance(orig, dict):
for k, v in new.items():
orig[k] = dict_merge(orig.get(k), v)
return orig
else:
return new
def update_simple(coins, support_info, type):
res = {}
for coin in coins:
key = coin["key"]
support = support_info[key]
details = dict(
name=coin["name"],
shortcut=coin["shortcut"],
type=type,
t1_enabled=_is_supported(support, "trezor1"),
t2_enabled=_is_supported(support, "trezor2"),
wallet={},
)
for k in OPTIONAL_KEYS:
if k in coin:
details[k] = coin[k]
details["wallet"].update(WALLETS.get(key, {}))
res[key] = details
return res
def update_bitcoin(coins, support_info):
res = update_simple(coins, support_info, "coin")
for coin in coins:
key = coin["key"]
support = support_info[key]
details = dict(
name=coin["coin_label"],
links=dict(Homepage=coin["website"], Github=coin["github"]),
wallet=WALLET_SUITE if _suite_support(coin, support) else {},
)
dict_merge(res[key], details)
return res
def update_erc20(coins, networks, support_info):
# TODO skip disabled networks?
network_support = {n["chain"]: support_info.get(n["key"]) for n in networks}
network_testnets = {n["chain"] for n in networks if "Testnet" in n["name"]}
res = update_simple(coins, support_info, "erc20")
for coin in coins:
key = coin["key"]
chain = coin["chain"]
hidden = False
if chain in network_testnets:
hidden = True
if "deprecation" in coin:
hidden = True
if network_support.get(chain, {}).get("suite"):
wallets = WALLET_SUITE
else:
wallets = WALLETS_ETH_3RDPARTY
details = dict(
network=chain,
address=coin["address"],
shortcut=coin["shortcut"],
links={},
wallet=wallets,
)
if hidden:
details["hidden"] = True
if coin.get("website"):
details["links"]["Homepage"] = coin["website"]
if coin.get("social", {}).get("github"):
details["links"]["Github"] = coin["social"]["github"]
dict_merge(res[key], details)
return res
def update_ethereum_networks(coins, support_info):
res = update_simple(coins, support_info, "coin")
for coin in coins:
key = coin["key"]
if support_info[key].get("suite"):
wallets = WALLET_SUITE
else:
wallets = WALLETS_ETH_3RDPARTY
details = dict(links=dict(Homepage=coin.get("url")), wallet=wallets)
dict_merge(res[key], details)
return res
def update_nem_mosaics(coins, support_info):
res = update_simple(coins, support_info, "mosaic")
for coin in coins:
key = coin["key"]
details = dict(wallet=WALLET_NEM)
dict_merge(res[key], details)
return res
def check_missing_data(coins):
for k, coin in coins.items():
hide = False
if "Homepage" not in coin.get("links", {}):
level = logging.WARNING
if k.startswith("erc20:"):
level = logging.INFO
LOG.log(level, f"{k}: Missing homepage")
hide = True
if coin["t1_enabled"] not in ALLOWED_SUPPORT_STATUS:
LOG.error(f"{k}: Unknown t1_enabled: {coin['t1_enabled']}")
hide = True
if coin["t2_enabled"] not in ALLOWED_SUPPORT_STATUS:
LOG.error(f"{k}: Unknown t2_enabled: {coin['t2_enabled']}")
hide = True
# check wallets
for wallet in coin["wallet"]:
name = wallet.get("name")
url = wallet.get("url")
if not name or not url:
LOG.warning(f"{k}: Bad wallet entry")
hide = True
continue
if "trezor" in name.lower() and url not in TREZOR_KNOWN_URLS:
LOG.warning(f"{k}: Strange URL for Trezor Wallet")
if coin["t1_enabled"] == "no" and coin["t2_enabled"] == "no":
LOG.info(f"{k}: Coin not enabled on either device")
hide = True
if len(coin.get("wallet", [])) == 0:
LOG.debug(f"{k}: Missing wallet")
if "Testnet" in coin["name"] or "Regtest" in coin["name"]:
LOG.debug(f"{k}: Hiding testnet")
hide = True
if not hide and coin.get("hidden"):
LOG.info(f"{k}: Details are OK, but coin is still hidden")
if hide:
data = marketcap.get_coin(coin)
if data and data["cmc_rank"] < 150 and not coin.get("ignore_cmc_rank"):
LOG.warning(f"{k}: Hiding coin ranked {data['cmc_rank']} on CMC")
coin["hidden"] = 1
# summary of hidden coins
hidden_coins = [k for k, coin in coins.items() if coin.get("hidden")]
for key in hidden_coins:
del coins[key]
def apply_overrides(coins):
for key, override in OVERRIDES.items():
if key not in coins:
LOG.warning(f"override without coin: {key}")
continue
dict_merge(coins[key], override)
def finalize_wallets(coins):
def sort_key(w):
if "trezor.io" in w["url"]:
return 0, w["name"]
else:
return 1, w["name"]
for coin in coins.values():
wallets_list = [
dict(name=name, url=url) for name, url in coin["wallet"].items() if url
]
wallets_list.sort(key=sort_key)
coin["wallet"] = wallets_list
@click.command()
# fmt: off
@click.option("-r", "--refresh", "refresh", flag_value=True, default=None, help="Force refresh market cap info")
@click.option("-R", "--no-refresh", "refresh", flag_value=False, default=None, help="Force use cached market cap info")
@click.option("-A", "--api-key", required=True, envvar="COINMARKETCAP_API_KEY", help="Coinmarketcap API key")
@click.option("-v", "--verbose", is_flag=True, help="Display more info")
# fmt: on
def main(refresh, api_key, verbose):
# setup logging
log_level = logging.DEBUG if verbose else logging.WARNING
root = logging.getLogger()
root.setLevel(log_level)
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(log_level)
root.addHandler(handler)
marketcap.init(api_key, refresh=refresh)
defs, _ = coin_info.coin_info_with_duplicates()
support_info = coin_info.support_info(defs)
coins = {}
coins.update(update_bitcoin(defs.bitcoin, support_info))
coins.update(update_erc20(defs.erc20, defs.eth, support_info))
coins.update(update_ethereum_networks(defs.eth, support_info))
coins.update(update_nem_mosaics(defs.nem, support_info))
coins.update(update_simple(defs.misc, support_info, "coin"))
apply_overrides(coins)
finalize_wallets(coins)
update_marketcaps(coins)
check_missing_data(coins)
info = summary(coins, api_key)
details = dict(coins=coins, info=info)
print(json.dumps(info, sort_keys=True, indent=4))
with open(os.path.join(coin_info.DEFS_DIR, "coins_details.json"), "w") as f:
json.dump(details, f, sort_keys=True, indent=4)
f.write("\n")
if __name__ == "__main__":
main()

@ -17,6 +17,10 @@ import click
import coin_info
from coin_info import Coin, CoinBuckets, Coins, CoinsInfo, FidoApps, SupportInfo
DEFINITIONS_TIMESTAMP_PATH = (
coin_info.DEFS_DIR / "ethereum" / "released-definitions-timestamp.txt"
)
try:
import termcolor
except ImportError:
@ -135,6 +139,7 @@ def render_file(
result = template.render(
support_info=support_info,
supported_on=make_support_filter(support_info),
ethereum_defs_timestamp=int(DEFINITIONS_TIMESTAMP_PATH.read_text()),
**coins,
**MAKO_FILTERS,
)
@ -203,7 +208,7 @@ def check_btc(coins: Coins) -> bool:
for coin in bucket:
name = coin["name"]
prefix = ""
if name.endswith("Testnet") or name.endswith("Regtest"):
if coin["is_testnet"]:
color = "green"
elif name == "Bitcoin":
color = "red"
@ -231,12 +236,7 @@ def check_btc(coins: Coins) -> bool:
"""
failed = False
for key, bucket in buckets.items():
mainnets = [
c
for c in bucket
if not c["name"].endswith("Testnet")
and not c["name"].endswith("Regtest")
]
mainnets = [c for c in bucket if not c["is_testnet"]]
have_bitcoin = any(coin["name"] == "Bitcoin" for coin in mainnets)
supported_mainnets = [c for c in mainnets if not c["unsupported"]]
@ -283,11 +283,9 @@ def check_btc(coins: Coins) -> bool:
return check_passed
def check_dups(buckets: CoinBuckets, print_at_level: int = logging.WARNING) -> bool:
def check_dups(buckets: CoinBuckets) -> bool:
"""Analyze and pretty-print results of `coin_info.mark_duplicate_shortcuts`.
`print_at_level` can be one of logging levels.
The results are buckets of colliding symbols.
If the collision is only between ERC20 tokens, it's DEBUG.
If the collision includes one non-token, it's INFO.
@ -295,15 +293,11 @@ def check_dups(buckets: CoinBuckets, print_at_level: int = logging.WARNING) -> b
"""
def coin_str(coin: Coin) -> str:
"""Colorize coins. Tokens are cyan, nontokens are red. Coins that are NOT
marked duplicate get a green asterisk.
"""
"""Colorize coins according to support / override status."""
prefix = ""
if coin["unsupported"]:
color = "grey"
prefix = crayon("blue", "(X)", bold=True)
elif coin_info.is_token(coin):
color = "cyan"
else:
color = "red"
@ -320,41 +314,24 @@ def check_dups(buckets: CoinBuckets, print_at_level: int = logging.WARNING) -> b
if not bucket:
continue
# supported coins from the bucket
supported = [coin for coin in bucket if not coin["unsupported"]]
nontokens = [
coin
for coin in bucket
if not coin["unsupported"]
and coin.get("duplicate")
and not coin_info.is_token(coin)
] # we do not count override-marked coins as duplicates here
cleared = not any(coin.get("duplicate") for coin in bucket)
eth_testnet = symbol == "teth"
# string generation
dup_str = ", ".join(coin_str(coin) for coin in bucket)
if len(nontokens) > 1 and not eth_testnet:
# Two or more colliding nontokens. This is always fatal.
# XXX consider allowing two nontokens as long as only one is supported?
if any(coin.get("duplicate") for coin in supported):
# At least one supported coin is marked as duplicate.
level = logging.ERROR
check_passed = False
elif len(supported) > 1:
# more than one supported coin in bucket
if cleared:
# some previous step has explicitly marked them as non-duplicate
level = logging.INFO
else:
# at most 1 non-token - we tentatively allow token collisions
# when explicitly marked as supported
level = logging.WARNING
# More than one supported coin in bucket, but no marked duplicates
# --> all must have been cleared by an override.
level = logging.INFO
else:
# At most 1 supported coin, at most 1 non-token. This is informational only.
# At most 1 supported coin in bucket. This is OK.
level = logging.DEBUG
# deciding whether to print
if level < print_at_level:
continue
if symbol == "_override":
print_log(level, "force-set duplicates:", dup_str)
else:
@ -408,7 +385,7 @@ def check_icons(coins: Coins) -> bool:
return check_passed
IGNORE_NONUNIFORM_KEYS = frozenset(("unsupported", "duplicate"))
IGNORE_NONUNIFORM_KEYS = frozenset(("unsupported", "duplicate", "coingecko_id"))
def check_key_uniformity(coins: Coins) -> bool:
@ -600,9 +577,8 @@ def cli(colors: bool) -> None:
# fmt: off
@click.option("--backend/--no-backend", "-b", default=False, help="Check blockbook/bitcore responses")
@click.option("--icons/--no-icons", default=True, help="Check icon files")
@click.option("-d", "--show-duplicates", type=click.Choice(("all", "nontoken", "errors")), default="errors", help="How much information about duplicate shortcuts should be shown.")
# fmt: on
def check(backend: bool, icons: bool, show_duplicates: str) -> None:
def check(backend: bool, icons: bool) -> None:
"""Validate coin definitions.
Checks that every btc-like coin is properly filled out, reports duplicate symbols,
@ -612,14 +588,7 @@ def check(backend: bool, icons: bool, show_duplicates: str) -> None:
Uniformity check ignores NEM mosaics and ERC20 tokens, where non-uniformity is
expected.
The `--show-duplicates` option can be set to:
- all: all shortcut collisions are shown, including colliding ERC20 tokens
- nontoken: only collisions that affect non-ERC20 coins are shown
- errors: only collisions between non-ERC20 tokens are shown. This is the default,
as a collision between two or more non-ERC20 tokens is an error.
All shortcut collisions are shown, including colliding ERC20 tokens.
In the output, duplicate ERC tokens will be shown in cyan; duplicate non-tokens
in red. An asterisk (*) next to symbol name means that even though it was detected
@ -654,14 +623,7 @@ def check(backend: bool, icons: bool, show_duplicates: str) -> None:
if not check_eth(defs.eth):
all_checks_passed = False
if show_duplicates == "all":
dup_level = logging.DEBUG
elif show_duplicates == "nontoken":
dup_level = logging.INFO
else:
dup_level = logging.WARNING
print("Checking unexpected duplicates...")
if not check_dups(buckets, dup_level):
if not check_dups(buckets):
all_checks_passed = False
nontoken_dups = [coin for coin in defs.as_list() if "dup_key_nontoken" in coin]

@ -1,59 +0,0 @@
#!/usr/bin/env python3
import json
import os
import subprocess
import tempfile
import click
import requests
LIVE_URL = "https://trezor.io/static/json/coins_details.json"
COINS_DETAILS = os.path.join(
os.path.dirname(__file__), "..", "defs", "coins_details.json"
)
def diffize_file(coins_details, tmp):
coins_list = list(coins_details["coins"].values())
for coin in coins_list:
coin.pop("marketcap_usd", None)
links = coin.get("links", {})
wallets = coin.get("wallet", {})
for link in links:
links[link] = links[link].rstrip("/")
for wallet in wallets:
wallet["url"] = wallet["url"].rstrip("/")
if not coin.get("wallet"):
coin.pop("wallet", None)
coins_list.sort(key=lambda c: c["name"])
for coin in coins_list:
name = coin["name"]
for key in coin:
print(name, "\t", key, ":", coin[key], file=tmp)
tmp.flush()
@click.command()
def cli():
"""Compare data from trezor.io/coins with current coins_details.json
Shows a nicely formatted diff between the live version and the trezor-common
version. Useful for catching auto-generation problems, etc.
"""
live_json = requests.get(LIVE_URL).json()
with open(COINS_DETAILS) as f:
coins_details = json.load(f)
Tmp = tempfile.NamedTemporaryFile
with Tmp("w") as tmpA, Tmp("w") as tmpB:
diffize_file(live_json, tmpA)
diffize_file(coins_details, tmpB)
subprocess.call(["diff", "-u", "--color=auto", tmpA.name, tmpB.name])
if __name__ == "__main__":
cli()

@ -79,30 +79,6 @@ def init(api_key, refresh=None):
COINS_SEARCHABLE = data_searchable
def get_coin(coin):
if coin["type"] == "erc20":
address = coin["address"].lower()
return COINS_SEARCHABLE.get(address)
data = None
if "coinmarketcap_alias" in coin:
data = COINS_SEARCHABLE.get(coin["coinmarketcap_alias"])
if data is None:
slug = coin["name"].replace(" ", "-").lower()
data = COINS_SEARCHABLE.get(slug)
if data is None:
data = COINS_SEARCHABLE.get(coin["shortcut"].lower())
return data
def marketcap(coin):
data = get_coin(coin)
if data is None:
return None
return int(data["quote"]["USD"]["market_cap"])
def fiat_price(coin_symbol):
data = COINS_SEARCHABLE.get(coin_symbol)
if data is None:

@ -1,10 +1,5 @@
#!/bin/sh
if [ -z "$COINMARKETCAP_API_KEY" ]; then
echo "Please set \$COINMARKETCAP_API_KEY"
exit 1
fi
HERE=$(dirname $0)
CHECK_OUTPUT=$(mktemp -d)
@ -12,17 +7,6 @@ trap "rm -r $CHECK_OUTPUT" EXIT
$HERE/cointool.py check > $CHECK_OUTPUT/pre.txt
ETH_DIR=$HERE/../defs/ethereum
ETH_REPOS="chains tokens"
for dir in $ETH_REPOS; do
(
cd $ETH_DIR/$dir; \
git checkout master; \
git pull origin master
)
done
$HERE/support.py release
$HERE/cointool.py check > $CHECK_OUTPUT/post.txt
@ -30,5 +14,3 @@ $HERE/cointool.py check > $CHECK_OUTPUT/post.txt
make -C $HERE/../.. gen
diff $CHECK_OUTPUT/pre.txt $CHECK_OUTPUT/post.txt
$HERE/coins_details.py

@ -155,69 +155,6 @@ def find_orphaned_support_keys(coins_dict):
return orphans
def find_supported_duplicate_tokens(coins_dict):
result = []
for _, supported, _ in all_support_dicts():
for key in supported:
if not key.startswith("erc20:"):
continue
if coins_dict.get(key, {}).get("duplicate"):
result.append(key)
return result
def process_erc20(coins_dict):
"""Make sure that:
* orphaned ERC20 support info is cleared out
* duplicate ERC20 tokens are not listed as supported
* non-duplicate ERC20 tokens are cleared out from the unsupported list
"""
erc20_dict = {
key: coin.get("duplicate", False)
for key, coin in coins_dict.items()
if coin_info.is_token(coin)
}
for device, supported, unsupported in all_support_dicts():
nondups = set()
dups = set(key for key, value in erc20_dict.items() if value)
for key in supported:
if key not in erc20_dict:
continue
if not erc20_dict[key]:
dups.discard(key)
for key in unsupported:
if key not in erc20_dict:
continue
# ignore dups that are unsupported now
dups.discard(key)
if not erc20_dict[key] and unsupported[key] == ERC20_DUPLICATE_KEY:
# remove duplicate status
nondups.add(key)
for key in dups:
if device in coin_info.MISSING_SUPPORT_MEANS_NO:
clear_support(device, key)
else:
print(f"ERC20 on {device}: adding duplicate {key}")
set_unsupported(device, key, ERC20_DUPLICATE_KEY)
for key in nondups:
print(f"ERC20 on {device}: clearing non-duplicate {key}")
clear_support(device, key)
def clear_erc20_mixed_buckets(buckets):
for bucket in buckets.values():
tokens = [coin for coin in bucket if coin_info.is_token(coin)]
if tokens == bucket:
continue
if len(tokens) == 1:
tokens[0]["duplicate"] = False
@click.group()
def cli():
pass
@ -228,10 +165,9 @@ def cli():
def fix(dry_run):
"""Fix expected problems.
Prunes orphaned keys and ensures that ERC20 duplicate info matches support info.
Currently only prunes orphaned keys.
"""
all_coins, buckets = coin_info.coin_info_with_duplicates()
clear_erc20_mixed_buckets(buckets)
all_coins = coin_info.coin_info()
coins_dict = all_coins.as_dict()
orphaned = find_orphaned_support_keys(coins_dict)
@ -240,32 +176,25 @@ def fix(dry_run):
for device in SUPPORT_INFO:
clear_support(device, orphan)
process_erc20(coins_dict)
if not dry_run:
write_support_info()
@cli.command()
# fmt: off
@click.option("-T", "--check-tokens", is_flag=True, help="Also check unsupported ERC20 tokens, ignored by default")
@click.option("-m", "--ignore-missing", is_flag=True, help="Do not fail on missing supportinfo")
# fmt: on
def check(check_tokens, ignore_missing):
def check(ignore_missing):
"""Check validity of support information.
Ensures that `support.json` data is well formed, there are no keys without
corresponding coins, and there are no coins without corresponding keys.
If `--check-tokens` is specified, the check will also take into account ERC20 tokens
without support info. This is disabled by default, because support info for ERC20
tokens is not strictly required.
If `--ignore-missing` is specified, the check will display coins with missing
support info, but will not fail when missing coins are found. This is
useful in Travis.
"""
all_coins, buckets = coin_info.coin_info_with_duplicates()
clear_erc20_mixed_buckets(buckets)
all_coins = coin_info.coin_info()
coins_dict = all_coins.as_dict()
checks_ok = True
@ -282,8 +211,6 @@ def check(check_tokens, ignore_missing):
missing = find_unsupported_coins(coins_dict)
for device, values in missing.items():
if not check_tokens:
values = [coin for coin in values if not coin_info.is_token(coin)]
if values:
if not ignore_missing:
checks_ok = False
@ -291,12 +218,6 @@ def check(check_tokens, ignore_missing):
for coin in values:
print(f"{coin['key']} - {coin['name']}")
supported_dups = find_supported_duplicate_tokens(coins_dict)
for key in supported_dups:
coin = coins_dict[key]
checks_ok = False
print(f"Token {coin['key']} ({coin['name']}) is duplicate but supported")
if not checks_ok:
print("Some checks have failed")
sys.exit(1)
@ -308,7 +229,6 @@ def check(check_tokens, ignore_missing):
@click.option("--v2", help="Version for TT release (default: guess from latest)")
@click.option("-n", "--dry-run", is_flag=True, help="Do not write changes")
@click.option("-f", "--force", is_flag=True, help="Proceed even with bad version/device info")
@click.option("-v", "--verbose", is_flag=True, help="Be more verbose")
@click.option("--skip-testnets/--no-skip-testnets", default=True, help="Automatically exclude testnets")
# fmt: on
@click.pass_context
@ -318,7 +238,6 @@ def release(
v2,
dry_run,
force,
verbose,
skip_testnets,
):
"""Release a new Trezor firmware.
@ -327,8 +246,7 @@ def release(
By default, marks duplicate tokens and testnets as unsupported, and all coins that
don't have support info are set to the released firmware version.
The tool will ask you to confirm each added coin. ERC20 tokens are added
automatically. Use `--verbose` to see them.
The tool will ask you to confirm each added coin.
"""
latest_releases = coin_info.latest_releases()
@ -391,18 +309,8 @@ def release(
if coin not in missing_list:
missing_list.append(coin)
tokens = [coin for coin in missing_list if coin_info.is_token(coin)]
nontokens = [coin for coin in missing_list if not coin_info.is_token(coin)]
for coin in tokens:
key = coin["key"]
# assert not coin.get("duplicate"), key
if verbose:
print(f"Adding missing {key} ({coin['name']})")
for device, version in versions.items():
support_setdefault(device, key, version)
for coin in nontokens:
if skip_testnets and "testnet" in coin["name"].lower():
for coin in missing_list:
if skip_testnets and coin["is_testnet"]:
for device, version in versions.items():
support_setdefault(device, coin["key"], False, "(AUTO) exclude testnet")
else:
@ -455,10 +363,6 @@ def set_support_value(key, entries, reason):
click.echo("Use 'support.py show' to search for the right one.")
sys.exit(1)
if coins[key].get("duplicate") and coin_info.is_token(coins[key]):
shortcut = coins[key]["shortcut"]
click.echo(f"Note: shortcut {shortcut} is a duplicate.")
for entry in entries:
try:
device, value = entry.split("=", maxsplit=1)

Loading…
Cancel
Save