1
0
mirror of https://github.com/trezor/trezor-firmware.git synced 2024-11-21 23:18:13 +00:00

feat(common): switch to external Ethereum definitions, add generators

This commit is contained in:
Martin Novák 2022-06-15 12:27:48 +02:00 committed by matejcik
parent 4a39df2847
commit f44ef58acb
19 changed files with 431 additions and 6838 deletions

6
.gitmodules vendored
View File

@ -8,9 +8,6 @@
[submodule "vendor/secp256k1-zkp"] [submodule "vendor/secp256k1-zkp"]
path = vendor/secp256k1-zkp path = vendor/secp256k1-zkp
url = https://github.com/bitcoin-core/secp256k1.git url = https://github.com/bitcoin-core/secp256k1.git
[submodule "common/defs/ethereum/tokens"]
path = common/defs/ethereum/tokens
url = https://github.com/ethereum-lists/tokens.git
[submodule "crypto/tests/wycheproof"] [submodule "crypto/tests/wycheproof"]
path = crypto/tests/wycheproof path = crypto/tests/wycheproof
url = https://github.com/google/wycheproof url = https://github.com/google/wycheproof
@ -24,6 +21,3 @@
[submodule "vendor/fido2-tests"] [submodule "vendor/fido2-tests"]
path = vendor/fido2-tests path = vendor/fido2-tests
url = https://github.com/trezor/fido2-tests.git url = https://github.com/trezor/fido2-tests.git
[submodule "common/defs/ethereum/chains"]
path = common/defs/ethereum/chains
url = https://github.com/ethereum-lists/chains

View File

@ -33,19 +33,22 @@ Testnet is considered a separate coin, so it must have its own JSON and icon.
We will not support coins that have `address_type` 0, i.e., same as Bitcoin. We will not support coins that have `address_type` 0, i.e., same as Bitcoin.
#### `eth` #### `eth` and `erc20`
The file [`ethereum/networks.json`](ethereum/networks.json) has a list of descriptions Definitions for Ethereum chains (networks) and tokens (erc20) are split in two parts:
of Ethereum networks. Each network must also have a PNG icon in `ethereum/<chain>.png`
file.
#### `erc20` 1. built-in definitions - some of the chain and token definitions are built into the firmware
image. List of built-in chains is stored in [`ethereum/networks.json`](ethereum/networks.json)
and tokens in [`ethereum/tokens.json`](ethereum/tokens.json).
2. external definitions - dynamically generated from multiple sources. Whole process is
described in separate
[document](https://docs.trezor.io/trezor-firmware/common/ethereum-definitions.html).
`ethereum/tokens` is a submodule linking to [Ethereum Lists](https://github.com/ethereum-lists/tokens) We generally do not accept updates to the built-in definitions. Instead, make sure your
project with descriptions of ERC20 tokens. If you want to add or update a token network or token is included in the external definitions. A good place to start is the
definition in Trezor, you need to get your change to the `tokens` repository first. [`ethereum-lists` GitHub organization](https://gitub.com/ethereum-lists): add your token
to the [tokens](https://github.com/ethereum-lists/tokens) repository, or your EVM chain to the
Trezor will only support tokens that have a unique symbol. [chains](https://github.com/ethereum-lists/chains) repository.
#### `nem` #### `nem`
@ -57,82 +60,32 @@ Supported coins that are not derived from Bitcoin, Ethereum or NEM are currently
and listed in separate file [`misc/misc.json`](misc/misc.json). Each coin must also have and listed in separate file [`misc/misc.json`](misc/misc.json). Each coin must also have
an icon in `misc/<short>.png`, where `short` is lowercased `shortcut` field from the JSON. an icon in `misc/<short>.png`, where `short` is lowercased `shortcut` field from the JSON.
## Keys ### Keys
Throughout the system, coins are identified by a _key_ - a colon-separated string Throughout the system, coins are identified by a _key_ - a colon-separated string
generated from the coin's type and shortcut: generated from the coin's type and shortcut:
* for Bitcoin-likes, key is `bitcoin:XYZ` * for Bitcoin-likes, key is `bitcoin:<shortcut>`
* for Ethereum networks, key is `eth:XYZ` * for Ethereum networks, key is `eth:<shortcut>`
* for ERC20 tokens, key is `erc20:<chain>:XYZ` * for ERC20 tokens, key is `erc20:<chain_symbol>:<token_shortcut>`
* for NEM mosaic, key is `nem:XYZ` * for NEM mosaic, key is `nem:<shortcut>`
* for others, key is `misc:XYZ` * for others, key is `misc:<shortcut>`
If a token shortcut has a suffix, such as `CAT (BlockCat)`, the whole thing is part If a token shortcut has a suffix, such as `CAT (BlockCat)`, the whole thing is part
of the key (so the key is `erc20:eth:CAT (BlockCat)`). of the key (so the key is `erc20:eth:CAT (BlockCat)`).
Sometimes coins end up with duplicate symbols, which in case of ERC20 tokens leads to Duplicate keys are not allowed and coins that would result in duplicate keys cannot be
key collisions. We do not allow duplicate symbols in the data, so this doesn't affect added to the dataset.
everyday use (see below). However, for validation purposes, it is sometimes useful
to work with unfiltered data that includes the duplicates. In such cases, keys are
deduplicated by adding a counter at end, e.g.: `erc20:eth:SMT:0`, `erc20:eth:SMT:1`.
Note that the suffix _is not stable_, so these coins can't be reliably uniquely identified.
## Duplicate Detection
**Duplicate symbols are not allowed** in our data. Tokens that have symbol collisions
are removed from the data set before processing. The duplicate status is mentioned
in `support.json` (see below), but it is impossible to override from there.
Duplicate detection works as follows:
1. a _symbol_ is split off from the shortcut string. E.g., for `CAT (BlockCat)`, symbol
is just `CAT`. It is compared, case-insensitive, with other coins (so `WIC` and `WiC`
are considered the same symbol), and identical symbols are put into a _bucket_.
2. if _all_ coins in the bucket also have a suffix (`CAT (BlockCat)` and `CAT (BitClave)`),
they are _not_ considered duplicate.
3. if _any_ coin in the bucket does _not_ have a suffix (`MIT` and `MIT (Mychatcoin)`),
all coins in the bucket are considered duplicate.
4. Duplicate tokens (coins from the `erc20` group) are automatically removed from data.
Duplicate non-tokens are marked but not removed. For instance, `bitcoin:FTC` (Feathercoin)
and `erc20:eth:FTC` (FTC) are duplicate, and `erc20:eth:FTC` is removed.
5. If two non-tokens collide with each other, it is an error that fails the CI build.
The file [`duplicity_overrides.json`](duplicity_overrides.json) can override detection
results: keys set to `true` are considered duplicate (in a separate bucket), keys set
to `false` are considered non-duplicate even if auto-detected. This is useful for
whitelisting a supported token explicitly, or blacklisting things that the detection
can't match (for instance "Battle" and "Bitlle" have suffixes, but they are too similar).
External contributors should not make changes to `duplicity_overrides.json`, unless
asked to.
You can use `./tools/cointool.py check -d all` to inspect duplicate detection in detail.
# Coins Details
The file [`coins_details.json`](coins_details.json) is a list of all known coins ## Wallet URLs
with support status, market cap information and relevant links. This is the source
file for https://trezor.io/coins.
You should never make changes to `coins_details.json` directly. Use `./tools/coins_details.py`
to regenerate it from known data.
If you need to change information in this file, modify the source information instead -
one of the JSON files in the groups listed above, support info in `support.json`, or
make a pull request to the tokens repository.
If you want to add a **wallet link**, modify the file [`wallets.json`](wallets.json). If you want to add a **wallet link**, modify the file [`wallets.json`](wallets.json).
If this is not viable for some reason, or if there is no source information ,
you can also edit [`coins_details.override.json`](coins_details.override.json).
External contributors should not touch this file unless asked to.
# Support Information # Support Information
We keep track of support status of each coin over our devices. That is We keep track of support status of each built-in coin over our devices. That is
`trezor1` for Trezor One, `trezor2` for Trezor T, `connect` for [Connect](https://github.com/trezor/connect) `trezor1` for Trezor One, `trezor2` for Trezor T, `connect` for [Connect](https://github.com/trezor/connect)
and `suite` for [Trezor Suite](https://suite.trezor.io/). In further description, the word "device" and `suite` for [Trezor Suite](https://suite.trezor.io/). In further description, the word "device"
applies to Connect and Suite as well. applies to Connect and Suite as well.

View File

@ -1,45 +0,0 @@
{
"erc20:eth:BAT": {
"name": "Basic Attention Token"
},
"erc20:eth:LINK (Chainlink)": {
"name": "Chainlink"
},
"erc20:eth:SOL": {
"shortcut": "SOLA"
},
"eth:CLO": {
"coinmarketcap_alias": "callisto-network"
},
"eth:ESN": {
"coinmarketcap_alias": "ethersocial"
},
"nem:DIMTOK": {
"coinmarketcap_alias": "dimcoin"
},
"eth:AUX": {
"links": {
"Github": "https://github.com/auxiliumglobal"
},
"wallet": {
"MyEtherWallet": null
}
},
"eth:EOS": {
"hidden": true,
"ignore_cmc_rank": true,
"reason": "this exists as misc:EOS and the eth: entry is probably a mistake"
},
"eth:XDC": {
"wallet": {
"MyCrypto": null,
"MyEtherWallet": null,
"XDC Wallet": "https://wallet.xinfin.network"
}
},
"misc:LSK": {
"hidden": true,
"ignore_cmc_rank": true,
"reason": "delisted incompatible hardfork"
}
}

View File

@ -1,38 +1,3 @@
{ {
"erc20:eth:BTL (Battle)": true, "misc:BNB": false
"erc20:eth:BTL (Bitlle)": true,
"erc20:eth:LINK Platform": true,
"erc20:eth:NXX": false,
"erc20:eth:SNX:c011": false,
"erc20:eth:TUSD": false,
"erc20:eth:Hdp": true,
"erc20:eth:Hdp.ф": true,
"erc20:eth:HEX:2b59": false,
"erc20:eth:JOB:dfbc": false,
"misc:BNB": false,
"eth:BNB": false,
"eth:ONE:1666600000": false,
"eth:ONE:1666600001": false,
"eth:ONE:1666600002": false,
"eth:ONE:1666600003": false,
"eth:tGOR:5": false,
"eth:tGOR:420": false,
"eth:tCELO:44787": false,
"eth:tCELO:62320": false,
"eth:QKC:100000": false,
"eth:QKC:100001": false,
"eth:QKC:100002": false,
"eth:QKC:100003": false,
"eth:QKC:100004": false,
"eth:QKC:100005": false,
"eth:QKC:100006": false,
"eth:QKC:100007": false,
"eth:QKC:100008": false,
"eth:xDAI:100": false,
"eth:xDAI:200": false,
"eth:CPAY:3000": false,
"eth:CPAY:3001": false,
"eth:CPAY:21337": false,
"erc20:eth:USDT": false,
"erc20:avax:USDT": false
} }

@ -1 +0,0 @@
Subproject commit 805ae42ecc53aa6493949b1e9c1da41e036c1845

View File

@ -0,0 +1,62 @@
[
{
"chain": "eth",
"chain_id": 1,
"coingecko_id": "ethereum",
"is_testnet": false,
"name": "Ethereum",
"shortcut": "ETH",
"slip44": 60
},
{
"chain": "rop",
"chain_id": 3,
"is_testnet": true,
"name": "Ropsten",
"shortcut": "tROP",
"slip44": 1
},
{
"chain": "rin",
"chain_id": 4,
"is_testnet": true,
"name": "Rinkeby",
"shortcut": "tRIN",
"slip44": 1
},
{
"chain": "gor",
"chain_id": 5,
"is_testnet": true,
"name": "Görli",
"shortcut": "tGOR",
"slip44": 1
},
{
"chain": "bnb",
"chain_id": 56,
"coingecko_id": "binance-smart-chain",
"is_testnet": false,
"name": "Binance Smart Chain",
"shortcut": "BNB",
"slip44": 714
},
{
"chain": "etc",
"chain_id": 61,
"coingecko_id": "ethereum-classic",
"is_testnet": false,
"name": "Ethereum Classic",
"shortcut": "ETC",
"slip44": 61
},
{
"chain": "MATIC",
"chain_id": 137,
"coingecko_id": "polygon-pos",
"is_testnet": false,
"name": "Polygon",
"shortcut": "MATIC",
"slip44": 966
}
]

View File

@ -0,0 +1 @@
1669892465

@ -1 +0,0 @@
Subproject commit 0eeaf9b9f13b5e6538da26d079e2b968dc8bb23f

View File

@ -0,0 +1,189 @@
{
"1;eth": [
{
"address": "0xdac17f958d2ee523a2206206994597c13d831ec7",
"coingecko_id": "tether",
"decimals": 6,
"name": "Tether",
"shortcut": "USDT"
},
{
"address": "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48",
"coingecko_id": "usd-coin",
"decimals": 6,
"name": "USD Coin",
"shortcut": "USDC"
},
{
"address": "0x4fabb145d64652a948d72533023f6e7a623c7c53",
"coingecko_id": "binance-usd",
"decimals": 18,
"name": "Binance USD",
"shortcut": "BUSD"
},
{
"address": "0x95ad61b0a150d79219dcf64e1e6cc01f0b64c4ce",
"coingecko_id": "shiba-inu",
"decimals": 18,
"name": "Shiba Inu",
"shortcut": "SHIB"
},
{
"address": "0x6b175474e89094c44da98b954eedeac495271d0f",
"coingecko_id": "dai",
"decimals": 18,
"name": "Dai",
"shortcut": "DAI"
},
{
"address": "0x7d1afa7b718fb893db30a3abc0cfc608aacfebb0",
"coingecko_id": "matic-network",
"decimals": 18,
"name": "Polygon",
"shortcut": "MATIC"
},
{
"address": "0xae7ab96520de3a18e5e111b5eaab095312d7fe84",
"coingecko_id": "staked-ether",
"decimals": 18,
"name": "Lido Staked Ether",
"shortcut": "STETH"
},
{
"address": "0x1f9840a85d5af5bf1d1762f925bdaddc4201f984",
"coingecko_id": "uniswap",
"decimals": 18,
"name": "Uniswap",
"shortcut": "UNI"
},
{
"address": "0x2260fac5e5542a773aa44fbcfedf7c193bc2c599",
"coingecko_id": "wrapped-bitcoin",
"decimals": 8,
"name": "Wrapped Bitcoin",
"shortcut": "WBTC"
},
{
"address": "0x75231f58b43240c9718dd58b4967c5114342a86c",
"coingecko_id": "okb",
"decimals": 18,
"name": "OKB",
"shortcut": "OKB"
},
{
"address": "0x2af5d2ad76741191d15dfe7bf6ac92d4bd912ca3",
"coingecko_id": "leo-token",
"decimals": 18,
"name": "LEO Token",
"shortcut": "LEO"
},
{
"address": "0x514910771af9ca656af840dff83e8264ecf986ca",
"coingecko_id": "chainlink",
"decimals": 18,
"name": "Chainlink",
"shortcut": "LINK"
},
{
"address": "0x50d1c9771902476076ecfc8b2a83ad6b9355a4c9",
"coingecko_id": "ftx-token",
"decimals": 18,
"name": "FTX",
"shortcut": "FTT"
},
{
"address": "0xa0b73e1ff0b80914ab6fe0444e65848c4c34450b",
"coingecko_id": "crypto-com-chain",
"decimals": 8,
"name": "Cronos",
"shortcut": "CRO"
},
{
"address": "0x4a220e6096b25eadb88358cb44068a3248254675",
"coingecko_id": "quant-network",
"decimals": 18,
"name": "Quant",
"shortcut": "QNT"
},
{
"address": "0x4d224452801aced8b2f0aebe155379bb5d594381",
"coingecko_id": "apecoin",
"decimals": 18,
"name": "ApeCoin",
"shortcut": "APE"
},
{
"address": "0xa2cd3d43c775978a96bdbf12d733d5a1ed94fb18",
"coingecko_id": "chain-2",
"decimals": 18,
"name": "Chain",
"shortcut": "XCN"
},
{
"address": "0x853d955acef822db058eb8505911ed77f175b99e",
"coingecko_id": "frax",
"decimals": 18,
"name": "Frax",
"shortcut": "FRAX"
},
{
"address": "0x3845badade8e6dff049820680d1f14bd3903a5d0",
"coingecko_id": "the-sandbox",
"decimals": 18,
"name": "The Sandbox",
"shortcut": "SAND"
},
{
"address": "0x0f5d2fb29fb7d3cfee444a200298f468908cc942",
"coingecko_id": "decentraland",
"decimals": 18,
"name": "Decentraland",
"shortcut": "MANA"
},
{
"address": "0xbb0e17ef65f82ab018d8edd776e8dd940327b28b",
"coingecko_id": "axie-infinity",
"decimals": 18,
"name": "Axie Infinity",
"shortcut": "AXS"
},
{
"address": "0x3506424f91fd33084466f402d5d97f05f8e3b4af",
"coingecko_id": "chiliz",
"decimals": 18,
"name": "Chiliz",
"shortcut": "CHZ"
},
{
"address": "0x7fc66500c84a76ad7e9c93437bfc5ac33e2ddae9",
"coingecko_id": "aave",
"decimals": 18,
"name": "Aave",
"shortcut": "AAVE"
},
{
"address": "0x86fa049857e0209aa7d9e616f7eb3b3b78ecfdb0",
"decimals": 18,
"name": "EOS",
"shortcut": "EOS"
}
],
"56;bnb": [
{
"address": "0x0eb3a705fc54725037cc9e008bdede697f62f335",
"coingecko_id": "cosmos",
"decimals": 18,
"name": "Cosmos Hub",
"shortcut": "ATOM"
}
],
"137;MATIC": [
{
"address": "0x2c89bbc92bd86f8075d1decc58c7f4e0107f286b",
"coingecko_id": "avalanche-2",
"decimals": 18,
"name": "Avalanche",
"shortcut": "AVAX"
}
]
}

File diff suppressed because it is too large Load Diff

View File

@ -120,16 +120,5 @@
}, },
"bitcoin:ZCR": { "bitcoin:ZCR": {
"Electrum-ZCR": "https://github.com/zcore-coin/electrum-wallet/" "Electrum-ZCR": "https://github.com/zcore-coin/electrum-wallet/"
},
"eth:WAN": {
"Wanchain Wallet": "https://www.wanchain.org/getstarted/"
},
"eth:AUX": {
"MyEtherWallet": null
},
"eth:XDC": {
"MyCrypto": null,
"MyEtherWallet": null,
"XDC Wallet": "https://wallet.xinfin.network"
} }
} }

View File

@ -21,8 +21,6 @@ the following commands:
* **`check`**: check validity of json definitions and associated data. Used in CI. * **`check`**: check validity of json definitions and associated data. Used in CI.
* **`dump`**: dump coin information, including support status, in JSON format. Various * **`dump`**: dump coin information, including support status, in JSON format. Various
filtering options are available, check help for details. filtering options are available, check help for details.
* **`coindefs`**: generate signed protobuf descriptions of coins. This is for future use
and could allow us to not need to store coin data in Trezor itself.
Use `cointool.py command --help` to get more information on each command. Use `cointool.py command --help` to get more information on each command.
@ -41,18 +39,6 @@ The following commands are available:
Use `support.py command --help` to get more information on each command. Use `support.py command --help` to get more information on each command.
### `coins_details.py`
Generates `coins_details.json`, source file for https://trezor.io/coins.
Collects data on coins, downloads market caps and puts everything into a single file.
Caches market cap data so you don't have to download it every time.
### `diffize_coins_details.py`
Compares generated `coins_details.json` to the released version currently served
on https://trezor.io/coins, in a format that is nicely readable to humans and
hard(er) to mess up by diff.
### `coin_info.py` ### `coin_info.py`
In case where code generation with `cointool.py render` is impractical or not sufficient, In case where code generation with `cointool.py render` is impractical or not sufficient,
@ -85,7 +71,7 @@ from the outside.
### `marketcap.py` ### `marketcap.py`
Module for obtaining market cap and price data used by `coins_details.py` and `maxfee.py`. Module for obtaining market cap and price data used by `maxfee.py`.
### `maxfee.py` ### `maxfee.py`
@ -130,32 +116,20 @@ Or mark them as unsupported explicitly.
## Releasing a new firmware ## Releasing a new firmware
#### **Step 1:** update the tokens repo #### **Step 1:** run the release script
```sh ```sh
pushd defs/ethereum/tokens ./tools/release.sh
git checkout master
git pull
popd
git add defs/ethereum/tokens
``` ```
#### **Step 2:** run the release flow
```sh
./tools/support.py release 2
```
The number `2` indicates that you are releasing Trezor 2. The version will be
automatically determined, based on currently released firmwares. Or you can explicitly
specify the version with `-r 2.1.0`.
All currently known unreleased ERC20 tokens are automatically set to the given version. All currently known unreleased ERC20 tokens are automatically set to the given version.
All coins marked _soon_ are set to the current version. This is automatic - coins that **_Note that "soon" feature was already removed and following paragraph is deprecated._**
_All coins marked _soon_ are set to the current version. This is automatic - coins that
were marked _soon_ were used in code generation and so should be released. If you want were marked _soon_ were used in code generation and so should be released. If you want
to avoid this, you will have to manually revert each coin to _soon_ status, either with to avoid this, you will have to manually revert each coin to _soon_ status, either with
`support.py set`, or by manually editing `support.json`. `support.py set`, or by manually editing `support.json`._
Coins in state _unknown_, i.e., coins that are known in the definitions but not listed Coins in state _unknown_, i.e., coins that are known in the definitions but not listed
in support files, will be also added. But you will be interactively asked to confirm in support files, will be also added. But you will be interactively asked to confirm
@ -170,13 +144,7 @@ Use `-g` or `--git-tag` to automatically tag the current `HEAD` with a version,
XXX this should also commit the changes though, otherwise the tag will apply to the wrong XXX this should also commit the changes though, otherwise the tag will apply to the wrong
commit. commit.
#### **Step 3:** review and commit your changes #### **Step 2:** review and commit your changes
Use `git diff` to review changes made, commit and push. If you tagged the commit in the Use `git diff` to review changes made, commit and push. If you tagged the commit in the
previous step, don't forget to `git push --tags` too. previous step, don't forget to `git push --tags` too.
#### **Step 4:** update submodule in your target repository
Go to `trezor-core` or `trezor-mcu` checkout and update the submodule. Checkout the
appropriate tag if you created it. If you're in `trezor-core`, run `make templates`
to update source files.

View File

@ -24,11 +24,7 @@ except ImportError:
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
ROOT = Path(__file__).resolve().parent.parent ROOT = Path(__file__).resolve().parent.parent
DEFS_DIR = ROOT / "defs"
if os.environ.get("DEFS_DIR"):
DEFS_DIR = Path(os.environ.get("DEFS_DIR")).resolve()
else:
DEFS_DIR = ROOT / "defs"
class SupportItemBool(TypedDict): class SupportItemBool(TypedDict):
@ -107,9 +103,9 @@ class Coin(TypedDict):
icon: str icon: str
# Special ETH fields # Special ETH fields
coingecko_id: str
chain: str chain: str
chain_id: str chain_id: int
rskip60: bool
url: str url: str
# Special erc20 fields # Special erc20 fields
@ -117,7 +113,6 @@ class Coin(TypedDict):
address: str address: str
address_bytes: bytes address_bytes: bytes
dup_key_nontoken: bool dup_key_nontoken: bool
deprecation: dict[str, str]
# Special NEM fields # Special NEM fields
ticker: str ticker: str
@ -126,6 +121,7 @@ class Coin(TypedDict):
unsupported: bool unsupported: bool
duplicate: bool duplicate: bool
support: SupportInfoItem support: SupportInfoItem
is_testnet: bool
# Backend-oriented fields # Backend-oriented fields
blockchain_link: dict[str, Any] blockchain_link: dict[str, Any]
@ -162,6 +158,10 @@ def load_json(*path: str | Path) -> Any:
return json.loads(file.read_text(), object_pairs_hook=OrderedDict) return json.loads(file.read_text(), object_pairs_hook=OrderedDict)
def get_btc_testnet_status(name: str) -> bool:
return any((mark in name.lower()) for mark in ("testnet", "regtest"))
# ====== CoinsInfo ====== # ====== CoinsInfo ======
@ -325,7 +325,7 @@ def validate_btc(coin: Coin) -> list[str]:
if not coin["max_address_length"] >= coin["min_address_length"]: if not coin["max_address_length"] >= coin["min_address_length"]:
errors.append("max address length must not be smaller than min address length") errors.append("max address length must not be smaller than min address length")
if "testnet" in coin["coin_name"].lower() and coin["slip44"] != 1: if coin["is_testnet"] and coin["slip44"] != 1:
errors.append("testnet coins must use slip44 coin type 1") errors.append("testnet coins must use slip44 coin type 1")
if coin["segwit"]: if coin["segwit"]:
@ -359,74 +359,48 @@ def _load_btc_coins() -> Coins:
shortcut=coin["coin_shortcut"], shortcut=coin["coin_shortcut"],
key=f"bitcoin:{coin['coin_shortcut']}", key=f"bitcoin:{coin['coin_shortcut']}",
icon=str(file.with_suffix(".png")), icon=str(file.with_suffix(".png")),
is_testnet=get_btc_testnet_status(coin["coin_label"]),
) )
coins.append(coin) coins.append(coin)
return coins return coins
def _load_ethereum_networks() -> Coins: def _load_builtin_ethereum_networks() -> Coins:
"""Load ethereum networks from `ethereum/networks.json`""" """Load ethereum networks from `ethereum/networks.json`"""
chains_path = DEFS_DIR / "ethereum" / "chains" / "_data" / "chains" chains_data = load_json("ethereum", "networks.json")
networks: Coins = [] networks: Coins = []
for chain in sorted( for chain_data in chains_data:
chains_path.glob("eip155-*.json"), chain_data.update(
key=lambda x: int(x.stem.replace("eip155-", "")), chain_id=chain_data["chain_id"],
): key=f"eth:{chain_data['shortcut']}",
chain_data = load_json(chain) # is_testnet is present in the JSON
shortcut = chain_data["nativeCurrency"]["symbol"]
name = chain_data["name"]
title = chain_data.get("title", "")
is_testnet = "testnet" in name.lower() or "testnet" in title.lower()
if is_testnet:
slip44 = 1
else:
slip44 = chain_data.get("slip44", 60)
if is_testnet and not shortcut.lower().startswith("t"):
shortcut = "t" + shortcut
rskip60 = shortcut in ("RBTC", "TRBTC")
# strip out bullcrap in network naming
if "mainnet" in name.lower():
name = re.sub(r" mainnet.*$", "", name, flags=re.IGNORECASE)
network = dict(
chain=chain_data["shortName"],
chain_id=chain_data["chainId"],
slip44=slip44,
shortcut=shortcut,
name=name,
rskip60=rskip60,
url=chain_data["infoURL"],
key=f"eth:{shortcut}",
) )
networks.append(cast(Coin, network)) networks.append(cast(Coin, chain_data))
return networks return networks
def _load_erc20_tokens() -> Coins: def _load_builtin_erc20_tokens() -> Coins:
"""Load ERC20 tokens from `ethereum/tokens` submodule.""" """Load ERC20 tokens from `ethereum/tokens.json`."""
networks = _load_ethereum_networks() tokens_data = load_json("ethereum", "tokens.json")
tokens: Coins = [] all_tokens: Coins = []
for network in networks:
chain = network["chain"]
chain_path = DEFS_DIR / "ethereum" / "tokens" / "tokens" / chain for chain_id_and_chain, tokens in tokens_data.items():
for file in sorted(chain_path.glob("*.json")): chain_id, chain = chain_id_and_chain.split(";", maxsplit=1)
token: Coin = load_json(file) for token in tokens:
token.update( token.update(
chain=chain, chain=chain,
chain_id=network["chain_id"], chain_id=int(chain_id),
address=token["address"].lower(),
address_bytes=bytes.fromhex(token["address"][2:]), address_bytes=bytes.fromhex(token["address"][2:]),
shortcut=token["symbol"], symbol=token["shortcut"],
key=f"erc20:{chain}:{token['symbol']}", key=f"erc20:{chain}:{token['shortcut']}",
is_testnet=False,
) )
tokens.append(token) all_tokens.append(cast(Coin, token))
return tokens return all_tokens
def _load_nem_mosaics() -> Coins: def _load_nem_mosaics() -> Coins:
@ -434,7 +408,11 @@ def _load_nem_mosaics() -> Coins:
mosaics: Coins = load_json("nem/nem_mosaics.json") mosaics: Coins = load_json("nem/nem_mosaics.json")
for mosaic in mosaics: for mosaic in mosaics:
shortcut = mosaic["ticker"].strip() shortcut = mosaic["ticker"].strip()
mosaic.update(shortcut=shortcut, key=f"nem:{shortcut}") mosaic.update(
shortcut=shortcut,
key=f"nem:{shortcut}",
is_testnet=False,
)
return mosaics return mosaics
@ -442,7 +420,10 @@ def _load_misc() -> Coins:
"""Loads miscellaneous networks from `misc/misc.json`""" """Loads miscellaneous networks from `misc/misc.json`"""
others: Coins = load_json("misc/misc.json") others: Coins = load_json("misc/misc.json")
for other in others: for other in others:
other.update(key=f"misc:{other['shortcut']}") other.update(
key=f"misc:{other['shortcut']}",
is_testnet=False,
)
return others return others
@ -493,10 +474,6 @@ def latest_releases() -> dict[str, Any]:
return latest return latest
def is_token(coin: Coin) -> bool:
return coin["key"].startswith("erc20:")
def support_info_single(support_data: SupportData, coin: Coin) -> SupportInfoItem: def support_info_single(support_data: SupportData, coin: Coin) -> SupportInfoItem:
"""Extract a support dict from `support.json` data. """Extract a support dict from `support.json` data.
@ -554,10 +531,6 @@ def support_info(coins: Iterable[Coin] | CoinsInfo | dict[str, Coin]) -> Support
WALLET_SUITE = {"Trezor Suite": "https://suite.trezor.io"} WALLET_SUITE = {"Trezor Suite": "https://suite.trezor.io"}
WALLET_NEM = {"Nano Wallet": "https://nemplatform.com/wallets/#desktop"} WALLET_NEM = {"Nano Wallet": "https://nemplatform.com/wallets/#desktop"}
WALLETS_ETH_3RDPARTY = {
"MyEtherWallet": "https://www.myetherwallet.com",
"MyCrypto": "https://mycrypto.com",
}
def get_wallet_data() -> WalletInfo: def get_wallet_data() -> WalletInfo:
@ -579,7 +552,6 @@ def _suite_support(coin: Coin, support: SupportInfoItem) -> bool:
def wallet_info_single( def wallet_info_single(
support_data: SupportInfo, support_data: SupportInfo,
eth_networks_support_data: SupportInfo,
wallet_data: WalletInfo, wallet_data: WalletInfo,
coin: Coin, coin: Coin,
) -> WalletItems: ) -> WalletItems:
@ -596,26 +568,16 @@ def wallet_info_single(
if key.startswith("bitcoin:"): if key.startswith("bitcoin:"):
if _suite_support(coin, support_data[key]): if _suite_support(coin, support_data[key]):
wallets.update(WALLET_SUITE) wallets.update(WALLET_SUITE)
elif key.startswith("eth:"):
if support_data[key]["suite"]:
wallets.update(WALLET_SUITE)
else:
wallets.update(WALLETS_ETH_3RDPARTY)
elif key.startswith("erc20:"):
if eth_networks_support_data[coin["chain"]]["suite"]:
wallets.update(WALLET_SUITE)
else:
wallets.update(WALLETS_ETH_3RDPARTY)
elif key.startswith("nem:"): elif key.startswith("nem:"):
wallets.update(WALLET_NEM) wallets.update(WALLET_NEM)
elif key.startswith("misc:"): elif key.startswith(("eth:", "erc20:", "misc:")):
pass # no special logic here pass # no special logic here
else: else:
raise ValueError(f"Unknown coin category: {key}") raise ValueError(f"Unknown coin category: {key}")
# Add wallets from `wallets.json` # Add wallets from `wallets.json`
# This must come last as it offers the ability to override existing wallets # This must come last as it offers the ability to override existing wallets
# (for example with `"MyEtherWallet": null` we delete the MyEtherWallet from the coin) # (for example with `"Trezor Suite": null` we delete the "Trezor Suite" from the coin)
wallets.update(wallet_data.get(key, {})) wallets.update(wallet_data.get(key, {}))
# Removing potentially disabled wallets from the last step # Removing potentially disabled wallets from the last step
@ -644,17 +606,9 @@ def wallet_info(coins: Iterable[Coin] | CoinsInfo | dict[str, Coin]) -> WalletIn
support_data = support_info(coins) support_data = support_info(coins)
wallet_data = get_wallet_data() wallet_data = get_wallet_data()
# Needed to find out suitable wallets for all the erc20 coins (Suite vs 3rd party)
eth_networks = [coin for coin in coins if coin["key"].startswith("eth:")]
eth_networks_support_data = {
n["chain"]: support_data[n["key"]] for n in eth_networks
}
wallet: WalletInfo = {} wallet: WalletInfo = {}
for coin in coins: for coin in coins:
wallet[coin["key"]] = wallet_info_single( wallet[coin["key"]] = wallet_info_single(support_data, wallet_data, coin)
support_data, eth_networks_support_data, wallet_data, coin
)
return wallet return wallet
@ -715,74 +669,7 @@ def apply_duplicity_overrides(coins: Coins) -> Coins:
return override_bucket return override_bucket
def deduplicate_erc20(buckets: CoinBuckets, networks: Coins) -> None: def find_duplicate_keys(all_coins: Coins) -> None:
"""Apply further processing to ERC20 duplicate buckets.
This function works on results of `mark_duplicate_shortcuts`.
Buckets that contain at least one non-token are ignored - symbol collisions
with non-tokens always apply.
Otherwise the following rules are applied:
1. If _all tokens_ in the bucket have shortcuts with distinct suffixes, e.g.,
`CAT (BitClave)` and `CAT (Blockcat)`, the bucket is cleared - all are considered
non-duplicate.
(If even one token in the bucket _does not_ have a distinct suffix, e.g.,
`MIT` and `MIT (Mychatcoin)`, this rule does not apply and ALL tokens in the bucket
are still considered duplicate.)
2. If there is only one "main" token in the bucket, the bucket is cleared.
That means that all other tokens must either be on testnets, or they must be marked
as deprecated, with a deprecation pointing to the "main" token.
"""
testnet_networks = {n["chain"] for n in networks if n["slip44"] == 1}
def clear_bucket(bucket: Coins) -> None:
# allow all coins, except those that are explicitly marked through overrides
for coin in bucket:
coin["duplicate"] = False
for bucket in buckets.values():
# Only check buckets that contain purely ERC20 tokens. Collision with
# a non-token is always forbidden.
if not all(is_token(c) for c in bucket):
continue
splits = (symbol_from_shortcut(coin["shortcut"]) for coin in bucket)
suffixes = {suffix for _, suffix in splits}
# if 1. all suffixes are distinct and 2. none of them are empty
if len(suffixes) == len(bucket) and all(suffixes):
clear_bucket(bucket)
continue
# protected categories:
testnets = [coin for coin in bucket if coin["chain"] in testnet_networks]
deprecated_by_same = [
coin
for coin in bucket
if "deprecation" in coin
and any(
other["address"] == coin["deprecation"]["new_address"]
for other in bucket
)
]
remaining = [
coin
for coin in bucket
if coin not in testnets and coin not in deprecated_by_same
]
if len(remaining) <= 1:
for coin in deprecated_by_same:
deprecated_symbol = "[deprecated] " + coin["symbol"]
coin["shortcut"] = coin["symbol"] = deprecated_symbol
coin["key"] += ":deprecated"
clear_bucket(bucket)
def deduplicate_keys(all_coins: Coins) -> None:
dups: CoinBuckets = defaultdict(list) dups: CoinBuckets = defaultdict(list)
for coin in all_coins: for coin in all_coins:
dups[coin["key"]].append(coin) dups[coin["key"]].append(coin)
@ -790,14 +677,8 @@ def deduplicate_keys(all_coins: Coins) -> None:
for coins in dups.values(): for coins in dups.values():
if len(coins) <= 1: if len(coins) <= 1:
continue continue
for i, coin in enumerate(coins): coin = coins[0]
if is_token(coin): raise ValueError(f"Duplicate key {coin['key']}")
coin["key"] += ":" + coin["address"][2:6].lower() # first 4 hex chars
elif "chain_id" in coin:
coin["key"] += ":" + str(coin["chain_id"])
else:
coin["key"] += f":{i}"
coin["dup_key_nontoken"] = True
def fill_blockchain_links(all_coins: CoinsInfo) -> None: def fill_blockchain_links(all_coins: CoinsInfo) -> None:
@ -829,8 +710,8 @@ def collect_coin_info() -> CoinsInfo:
""" """
all_coins = CoinsInfo( all_coins = CoinsInfo(
bitcoin=_load_btc_coins(), bitcoin=_load_btc_coins(),
eth=_load_ethereum_networks(), eth=_load_builtin_ethereum_networks(),
erc20=_load_erc20_tokens(), erc20=_load_builtin_erc20_tokens(),
nem=_load_nem_mosaics(), nem=_load_nem_mosaics(),
misc=_load_misc(), misc=_load_misc(),
) )
@ -866,10 +747,8 @@ def coin_info_with_duplicates() -> tuple[CoinsInfo, CoinBuckets]:
coin_list = all_coins.as_list() coin_list = all_coins.as_list()
# generate duplicity buckets based on shortcuts # generate duplicity buckets based on shortcuts
buckets = mark_duplicate_shortcuts(all_coins.as_list()) buckets = mark_duplicate_shortcuts(all_coins.as_list())
# apply further processing to ERC20 tokens, generate deprecations etc. # ensure the whole list has unique keys
deduplicate_erc20(buckets, all_coins.eth) find_duplicate_keys(coin_list)
# ensure the whole list has unique keys (taking into account changes from deduplicate_erc20)
deduplicate_keys(coin_list)
# apply duplicity overrides # apply duplicity overrides
buckets["_override"] = apply_duplicity_overrides(coin_list) buckets["_override"] = apply_duplicity_overrides(coin_list)
sort_coin_infos(all_coins) sort_coin_infos(all_coins)
@ -883,9 +762,6 @@ def coin_info() -> CoinsInfo:
Does not auto-delete duplicates. This should now be based on support info. Does not auto-delete duplicates. This should now be based on support info.
""" """
all_coins, _ = coin_info_with_duplicates() all_coins, _ = coin_info_with_duplicates()
# all_coins["erc20"] = [
# coin for coin in all_coins["erc20"] if not coin.get("duplicate")
# ]
return all_coins return all_coins

View File

@ -1,331 +0,0 @@
#!/usr/bin/env python3
"""Fetch information about coins and tokens supported by Trezor and update it in coins_details.json."""
import json
import logging
import os
import sys
import time
import click
import coin_info
import marketcap
LOG = logging.getLogger(__name__)
OPTIONAL_KEYS = ("links", "notes", "wallet")
ALLOWED_SUPPORT_STATUS = ("yes", "no")
WALLETS = coin_info.load_json("wallets.json")
OVERRIDES = coin_info.load_json("coins_details.override.json")
VERSIONS = coin_info.latest_releases()
# automatic wallet entries
WALLET_SUITE = {"Trezor Suite": "https://suite.trezor.io"}
WALLET_NEM = {"Nano Wallet": "https://nemplatform.com/wallets/#desktop"}
WALLETS_ETH_3RDPARTY = {
"MyEtherWallet": "https://www.myetherwallet.com",
"MyCrypto": "https://mycrypto.com",
}
TREZOR_KNOWN_URLS = (
"https://suite.trezor.io",
"https://wallet.trezor.io",
)
def update_marketcaps(coins):
for coin in coins.values():
coin["marketcap_usd"] = marketcap.marketcap(coin) or 0
def summary(coins, api_key):
t1_coins = 0
t2_coins = 0
supported_marketcap = 0
for coin in coins.values():
if coin.get("hidden"):
continue
t1_enabled = coin["t1_enabled"] == "yes"
t2_enabled = coin["t2_enabled"] == "yes"
if t1_enabled:
t1_coins += 1
if t2_enabled:
t2_coins += 1
if t1_enabled or t2_enabled:
supported_marketcap += coin.get("marketcap_usd", 0)
total_marketcap = None
try:
ret = marketcap.call("global-metrics/quotes/latest", api_key)
total_marketcap = int(ret["data"]["quote"]["USD"]["total_market_cap"])
except Exception:
pass
marketcap_percent = 100 * supported_marketcap / total_marketcap
return dict(
updated_at=int(time.time()),
updated_at_readable=time.asctime(),
t1_coins=t1_coins,
t2_coins=t2_coins,
marketcap_usd=supported_marketcap,
total_marketcap_usd=total_marketcap,
marketcap_supported=f"{marketcap_percent:.02f} %",
)
def _is_supported(support, trezor_version):
# True or version string means YES
# False or None means NO
return "yes" if support.get(trezor_version) else "no"
def _suite_support(coin, support):
"""Check the "suite" support property.
If set, check that at least one of the backends run on trezor.io.
If yes, assume we support the coin in our wallet.
Otherwise it's probably working with a custom backend, which means don't
link to our wallet.
"""
if not support.get("suite"):
return False
return any(".trezor.io" in url for url in coin["blockbook"])
def dict_merge(orig, new):
if isinstance(new, dict) and isinstance(orig, dict):
for k, v in new.items():
orig[k] = dict_merge(orig.get(k), v)
return orig
else:
return new
def update_simple(coins, support_info, type):
res = {}
for coin in coins:
key = coin["key"]
support = support_info[key]
details = dict(
name=coin["name"],
shortcut=coin["shortcut"],
type=type,
t1_enabled=_is_supported(support, "trezor1"),
t2_enabled=_is_supported(support, "trezor2"),
wallet={},
)
for k in OPTIONAL_KEYS:
if k in coin:
details[k] = coin[k]
details["wallet"].update(WALLETS.get(key, {}))
res[key] = details
return res
def update_bitcoin(coins, support_info):
res = update_simple(coins, support_info, "coin")
for coin in coins:
key = coin["key"]
support = support_info[key]
details = dict(
name=coin["coin_label"],
links=dict(Homepage=coin["website"], Github=coin["github"]),
wallet=WALLET_SUITE if _suite_support(coin, support) else {},
)
dict_merge(res[key], details)
return res
def update_erc20(coins, networks, support_info):
# TODO skip disabled networks?
network_support = {n["chain"]: support_info.get(n["key"]) for n in networks}
network_testnets = {n["chain"] for n in networks if "Testnet" in n["name"]}
res = update_simple(coins, support_info, "erc20")
for coin in coins:
key = coin["key"]
chain = coin["chain"]
hidden = False
if chain in network_testnets:
hidden = True
if "deprecation" in coin:
hidden = True
if network_support.get(chain, {}).get("suite"):
wallets = WALLET_SUITE
else:
wallets = WALLETS_ETH_3RDPARTY
details = dict(
network=chain,
address=coin["address"],
shortcut=coin["shortcut"],
links={},
wallet=wallets,
)
if hidden:
details["hidden"] = True
if coin.get("website"):
details["links"]["Homepage"] = coin["website"]
if coin.get("social", {}).get("github"):
details["links"]["Github"] = coin["social"]["github"]
dict_merge(res[key], details)
return res
def update_ethereum_networks(coins, support_info):
res = update_simple(coins, support_info, "coin")
for coin in coins:
key = coin["key"]
if support_info[key].get("suite"):
wallets = WALLET_SUITE
else:
wallets = WALLETS_ETH_3RDPARTY
details = dict(links=dict(Homepage=coin.get("url")), wallet=wallets)
dict_merge(res[key], details)
return res
def update_nem_mosaics(coins, support_info):
res = update_simple(coins, support_info, "mosaic")
for coin in coins:
key = coin["key"]
details = dict(wallet=WALLET_NEM)
dict_merge(res[key], details)
return res
def check_missing_data(coins):
for k, coin in coins.items():
hide = False
if "Homepage" not in coin.get("links", {}):
level = logging.WARNING
if k.startswith("erc20:"):
level = logging.INFO
LOG.log(level, f"{k}: Missing homepage")
hide = True
if coin["t1_enabled"] not in ALLOWED_SUPPORT_STATUS:
LOG.error(f"{k}: Unknown t1_enabled: {coin['t1_enabled']}")
hide = True
if coin["t2_enabled"] not in ALLOWED_SUPPORT_STATUS:
LOG.error(f"{k}: Unknown t2_enabled: {coin['t2_enabled']}")
hide = True
# check wallets
for wallet in coin["wallet"]:
name = wallet.get("name")
url = wallet.get("url")
if not name or not url:
LOG.warning(f"{k}: Bad wallet entry")
hide = True
continue
if "trezor" in name.lower() and url not in TREZOR_KNOWN_URLS:
LOG.warning(f"{k}: Strange URL for Trezor Wallet")
if coin["t1_enabled"] == "no" and coin["t2_enabled"] == "no":
LOG.info(f"{k}: Coin not enabled on either device")
hide = True
if len(coin.get("wallet", [])) == 0:
LOG.debug(f"{k}: Missing wallet")
if "Testnet" in coin["name"] or "Regtest" in coin["name"]:
LOG.debug(f"{k}: Hiding testnet")
hide = True
if not hide and coin.get("hidden"):
LOG.info(f"{k}: Details are OK, but coin is still hidden")
if hide:
data = marketcap.get_coin(coin)
if data and data["cmc_rank"] < 150 and not coin.get("ignore_cmc_rank"):
LOG.warning(f"{k}: Hiding coin ranked {data['cmc_rank']} on CMC")
coin["hidden"] = 1
# summary of hidden coins
hidden_coins = [k for k, coin in coins.items() if coin.get("hidden")]
for key in hidden_coins:
del coins[key]
def apply_overrides(coins):
for key, override in OVERRIDES.items():
if key not in coins:
LOG.warning(f"override without coin: {key}")
continue
dict_merge(coins[key], override)
def finalize_wallets(coins):
def sort_key(w):
if "trezor.io" in w["url"]:
return 0, w["name"]
else:
return 1, w["name"]
for coin in coins.values():
wallets_list = [
dict(name=name, url=url) for name, url in coin["wallet"].items() if url
]
wallets_list.sort(key=sort_key)
coin["wallet"] = wallets_list
@click.command()
# fmt: off
@click.option("-r", "--refresh", "refresh", flag_value=True, default=None, help="Force refresh market cap info")
@click.option("-R", "--no-refresh", "refresh", flag_value=False, default=None, help="Force use cached market cap info")
@click.option("-A", "--api-key", required=True, envvar="COINMARKETCAP_API_KEY", help="Coinmarketcap API key")
@click.option("-v", "--verbose", is_flag=True, help="Display more info")
# fmt: on
def main(refresh, api_key, verbose):
# setup logging
log_level = logging.DEBUG if verbose else logging.WARNING
root = logging.getLogger()
root.setLevel(log_level)
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(log_level)
root.addHandler(handler)
marketcap.init(api_key, refresh=refresh)
defs, _ = coin_info.coin_info_with_duplicates()
support_info = coin_info.support_info(defs)
coins = {}
coins.update(update_bitcoin(defs.bitcoin, support_info))
coins.update(update_erc20(defs.erc20, defs.eth, support_info))
coins.update(update_ethereum_networks(defs.eth, support_info))
coins.update(update_nem_mosaics(defs.nem, support_info))
coins.update(update_simple(defs.misc, support_info, "coin"))
apply_overrides(coins)
finalize_wallets(coins)
update_marketcaps(coins)
check_missing_data(coins)
info = summary(coins, api_key)
details = dict(coins=coins, info=info)
print(json.dumps(info, sort_keys=True, indent=4))
with open(os.path.join(coin_info.DEFS_DIR, "coins_details.json"), "w") as f:
json.dump(details, f, sort_keys=True, indent=4)
f.write("\n")
if __name__ == "__main__":
main()

View File

@ -17,6 +17,10 @@ import click
import coin_info import coin_info
from coin_info import Coin, CoinBuckets, Coins, CoinsInfo, FidoApps, SupportInfo from coin_info import Coin, CoinBuckets, Coins, CoinsInfo, FidoApps, SupportInfo
DEFINITIONS_TIMESTAMP_PATH = (
coin_info.DEFS_DIR / "ethereum" / "released-definitions-timestamp.txt"
)
try: try:
import termcolor import termcolor
except ImportError: except ImportError:
@ -135,6 +139,7 @@ def render_file(
result = template.render( result = template.render(
support_info=support_info, support_info=support_info,
supported_on=make_support_filter(support_info), supported_on=make_support_filter(support_info),
ethereum_defs_timestamp=int(DEFINITIONS_TIMESTAMP_PATH.read_text()),
**coins, **coins,
**MAKO_FILTERS, **MAKO_FILTERS,
) )
@ -203,7 +208,7 @@ def check_btc(coins: Coins) -> bool:
for coin in bucket: for coin in bucket:
name = coin["name"] name = coin["name"]
prefix = "" prefix = ""
if name.endswith("Testnet") or name.endswith("Regtest"): if coin["is_testnet"]:
color = "green" color = "green"
elif name == "Bitcoin": elif name == "Bitcoin":
color = "red" color = "red"
@ -231,12 +236,7 @@ def check_btc(coins: Coins) -> bool:
""" """
failed = False failed = False
for key, bucket in buckets.items(): for key, bucket in buckets.items():
mainnets = [ mainnets = [c for c in bucket if not c["is_testnet"]]
c
for c in bucket
if not c["name"].endswith("Testnet")
and not c["name"].endswith("Regtest")
]
have_bitcoin = any(coin["name"] == "Bitcoin" for coin in mainnets) have_bitcoin = any(coin["name"] == "Bitcoin" for coin in mainnets)
supported_mainnets = [c for c in mainnets if not c["unsupported"]] supported_mainnets = [c for c in mainnets if not c["unsupported"]]
@ -283,11 +283,9 @@ def check_btc(coins: Coins) -> bool:
return check_passed return check_passed
def check_dups(buckets: CoinBuckets, print_at_level: int = logging.WARNING) -> bool: def check_dups(buckets: CoinBuckets) -> bool:
"""Analyze and pretty-print results of `coin_info.mark_duplicate_shortcuts`. """Analyze and pretty-print results of `coin_info.mark_duplicate_shortcuts`.
`print_at_level` can be one of logging levels.
The results are buckets of colliding symbols. The results are buckets of colliding symbols.
If the collision is only between ERC20 tokens, it's DEBUG. If the collision is only between ERC20 tokens, it's DEBUG.
If the collision includes one non-token, it's INFO. If the collision includes one non-token, it's INFO.
@ -295,15 +293,11 @@ def check_dups(buckets: CoinBuckets, print_at_level: int = logging.WARNING) -> b
""" """
def coin_str(coin: Coin) -> str: def coin_str(coin: Coin) -> str:
"""Colorize coins. Tokens are cyan, nontokens are red. Coins that are NOT """Colorize coins according to support / override status."""
marked duplicate get a green asterisk.
"""
prefix = "" prefix = ""
if coin["unsupported"]: if coin["unsupported"]:
color = "grey" color = "grey"
prefix = crayon("blue", "(X)", bold=True) prefix = crayon("blue", "(X)", bold=True)
elif coin_info.is_token(coin):
color = "cyan"
else: else:
color = "red" color = "red"
@ -320,41 +314,24 @@ def check_dups(buckets: CoinBuckets, print_at_level: int = logging.WARNING) -> b
if not bucket: if not bucket:
continue continue
# supported coins from the bucket
supported = [coin for coin in bucket if not coin["unsupported"]] supported = [coin for coin in bucket if not coin["unsupported"]]
nontokens = [
coin
for coin in bucket
if not coin["unsupported"]
and coin.get("duplicate")
and not coin_info.is_token(coin)
] # we do not count override-marked coins as duplicates here
cleared = not any(coin.get("duplicate") for coin in bucket)
eth_testnet = symbol == "teth"
# string generation # string generation
dup_str = ", ".join(coin_str(coin) for coin in bucket) dup_str = ", ".join(coin_str(coin) for coin in bucket)
if len(nontokens) > 1 and not eth_testnet:
# Two or more colliding nontokens. This is always fatal. if any(coin.get("duplicate") for coin in supported):
# XXX consider allowing two nontokens as long as only one is supported? # At least one supported coin is marked as duplicate.
level = logging.ERROR level = logging.ERROR
check_passed = False check_passed = False
elif len(supported) > 1: elif len(supported) > 1:
# more than one supported coin in bucket # More than one supported coin in bucket, but no marked duplicates
if cleared: # --> all must have been cleared by an override.
# some previous step has explicitly marked them as non-duplicate level = logging.INFO
level = logging.INFO
else:
# at most 1 non-token - we tentatively allow token collisions
# when explicitly marked as supported
level = logging.WARNING
else: else:
# At most 1 supported coin, at most 1 non-token. This is informational only. # At most 1 supported coin in bucket. This is OK.
level = logging.DEBUG level = logging.DEBUG
# deciding whether to print
if level < print_at_level:
continue
if symbol == "_override": if symbol == "_override":
print_log(level, "force-set duplicates:", dup_str) print_log(level, "force-set duplicates:", dup_str)
else: else:
@ -408,7 +385,7 @@ def check_icons(coins: Coins) -> bool:
return check_passed return check_passed
IGNORE_NONUNIFORM_KEYS = frozenset(("unsupported", "duplicate")) IGNORE_NONUNIFORM_KEYS = frozenset(("unsupported", "duplicate", "coingecko_id"))
def check_key_uniformity(coins: Coins) -> bool: def check_key_uniformity(coins: Coins) -> bool:
@ -600,9 +577,8 @@ def cli(colors: bool) -> None:
# fmt: off # fmt: off
@click.option("--backend/--no-backend", "-b", default=False, help="Check blockbook/bitcore responses") @click.option("--backend/--no-backend", "-b", default=False, help="Check blockbook/bitcore responses")
@click.option("--icons/--no-icons", default=True, help="Check icon files") @click.option("--icons/--no-icons", default=True, help="Check icon files")
@click.option("-d", "--show-duplicates", type=click.Choice(("all", "nontoken", "errors")), default="errors", help="How much information about duplicate shortcuts should be shown.")
# fmt: on # fmt: on
def check(backend: bool, icons: bool, show_duplicates: str) -> None: def check(backend: bool, icons: bool) -> None:
"""Validate coin definitions. """Validate coin definitions.
Checks that every btc-like coin is properly filled out, reports duplicate symbols, Checks that every btc-like coin is properly filled out, reports duplicate symbols,
@ -612,14 +588,7 @@ def check(backend: bool, icons: bool, show_duplicates: str) -> None:
Uniformity check ignores NEM mosaics and ERC20 tokens, where non-uniformity is Uniformity check ignores NEM mosaics and ERC20 tokens, where non-uniformity is
expected. expected.
The `--show-duplicates` option can be set to: All shortcut collisions are shown, including colliding ERC20 tokens.
- all: all shortcut collisions are shown, including colliding ERC20 tokens
- nontoken: only collisions that affect non-ERC20 coins are shown
- errors: only collisions between non-ERC20 tokens are shown. This is the default,
as a collision between two or more non-ERC20 tokens is an error.
In the output, duplicate ERC tokens will be shown in cyan; duplicate non-tokens In the output, duplicate ERC tokens will be shown in cyan; duplicate non-tokens
in red. An asterisk (*) next to symbol name means that even though it was detected in red. An asterisk (*) next to symbol name means that even though it was detected
@ -654,14 +623,7 @@ def check(backend: bool, icons: bool, show_duplicates: str) -> None:
if not check_eth(defs.eth): if not check_eth(defs.eth):
all_checks_passed = False all_checks_passed = False
if show_duplicates == "all": if not check_dups(buckets):
dup_level = logging.DEBUG
elif show_duplicates == "nontoken":
dup_level = logging.INFO
else:
dup_level = logging.WARNING
print("Checking unexpected duplicates...")
if not check_dups(buckets, dup_level):
all_checks_passed = False all_checks_passed = False
nontoken_dups = [coin for coin in defs.as_list() if "dup_key_nontoken" in coin] nontoken_dups = [coin for coin in defs.as_list() if "dup_key_nontoken" in coin]

View File

@ -1,59 +0,0 @@
#!/usr/bin/env python3
import json
import os
import subprocess
import tempfile
import click
import requests
LIVE_URL = "https://trezor.io/static/json/coins_details.json"
COINS_DETAILS = os.path.join(
os.path.dirname(__file__), "..", "defs", "coins_details.json"
)
def diffize_file(coins_details, tmp):
coins_list = list(coins_details["coins"].values())
for coin in coins_list:
coin.pop("marketcap_usd", None)
links = coin.get("links", {})
wallets = coin.get("wallet", {})
for link in links:
links[link] = links[link].rstrip("/")
for wallet in wallets:
wallet["url"] = wallet["url"].rstrip("/")
if not coin.get("wallet"):
coin.pop("wallet", None)
coins_list.sort(key=lambda c: c["name"])
for coin in coins_list:
name = coin["name"]
for key in coin:
print(name, "\t", key, ":", coin[key], file=tmp)
tmp.flush()
@click.command()
def cli():
"""Compare data from trezor.io/coins with current coins_details.json
Shows a nicely formatted diff between the live version and the trezor-common
version. Useful for catching auto-generation problems, etc.
"""
live_json = requests.get(LIVE_URL).json()
with open(COINS_DETAILS) as f:
coins_details = json.load(f)
Tmp = tempfile.NamedTemporaryFile
with Tmp("w") as tmpA, Tmp("w") as tmpB:
diffize_file(live_json, tmpA)
diffize_file(coins_details, tmpB)
subprocess.call(["diff", "-u", "--color=auto", tmpA.name, tmpB.name])
if __name__ == "__main__":
cli()

View File

@ -79,30 +79,6 @@ def init(api_key, refresh=None):
COINS_SEARCHABLE = data_searchable COINS_SEARCHABLE = data_searchable
def get_coin(coin):
if coin["type"] == "erc20":
address = coin["address"].lower()
return COINS_SEARCHABLE.get(address)
data = None
if "coinmarketcap_alias" in coin:
data = COINS_SEARCHABLE.get(coin["coinmarketcap_alias"])
if data is None:
slug = coin["name"].replace(" ", "-").lower()
data = COINS_SEARCHABLE.get(slug)
if data is None:
data = COINS_SEARCHABLE.get(coin["shortcut"].lower())
return data
def marketcap(coin):
data = get_coin(coin)
if data is None:
return None
return int(data["quote"]["USD"]["market_cap"])
def fiat_price(coin_symbol): def fiat_price(coin_symbol):
data = COINS_SEARCHABLE.get(coin_symbol) data = COINS_SEARCHABLE.get(coin_symbol)
if data is None: if data is None:

View File

@ -1,10 +1,5 @@
#!/bin/sh #!/bin/sh
if [ -z "$COINMARKETCAP_API_KEY" ]; then
echo "Please set \$COINMARKETCAP_API_KEY"
exit 1
fi
HERE=$(dirname $0) HERE=$(dirname $0)
CHECK_OUTPUT=$(mktemp -d) CHECK_OUTPUT=$(mktemp -d)
@ -12,17 +7,6 @@ trap "rm -r $CHECK_OUTPUT" EXIT
$HERE/cointool.py check > $CHECK_OUTPUT/pre.txt $HERE/cointool.py check > $CHECK_OUTPUT/pre.txt
ETH_DIR=$HERE/../defs/ethereum
ETH_REPOS="chains tokens"
for dir in $ETH_REPOS; do
(
cd $ETH_DIR/$dir; \
git checkout master; \
git pull origin master
)
done
$HERE/support.py release $HERE/support.py release
$HERE/cointool.py check > $CHECK_OUTPUT/post.txt $HERE/cointool.py check > $CHECK_OUTPUT/post.txt
@ -30,5 +14,3 @@ $HERE/cointool.py check > $CHECK_OUTPUT/post.txt
make -C $HERE/../.. gen make -C $HERE/../.. gen
diff $CHECK_OUTPUT/pre.txt $CHECK_OUTPUT/post.txt diff $CHECK_OUTPUT/pre.txt $CHECK_OUTPUT/post.txt
$HERE/coins_details.py

View File

@ -155,69 +155,6 @@ def find_orphaned_support_keys(coins_dict):
return orphans return orphans
def find_supported_duplicate_tokens(coins_dict):
result = []
for _, supported, _ in all_support_dicts():
for key in supported:
if not key.startswith("erc20:"):
continue
if coins_dict.get(key, {}).get("duplicate"):
result.append(key)
return result
def process_erc20(coins_dict):
"""Make sure that:
* orphaned ERC20 support info is cleared out
* duplicate ERC20 tokens are not listed as supported
* non-duplicate ERC20 tokens are cleared out from the unsupported list
"""
erc20_dict = {
key: coin.get("duplicate", False)
for key, coin in coins_dict.items()
if coin_info.is_token(coin)
}
for device, supported, unsupported in all_support_dicts():
nondups = set()
dups = set(key for key, value in erc20_dict.items() if value)
for key in supported:
if key not in erc20_dict:
continue
if not erc20_dict[key]:
dups.discard(key)
for key in unsupported:
if key not in erc20_dict:
continue
# ignore dups that are unsupported now
dups.discard(key)
if not erc20_dict[key] and unsupported[key] == ERC20_DUPLICATE_KEY:
# remove duplicate status
nondups.add(key)
for key in dups:
if device in coin_info.MISSING_SUPPORT_MEANS_NO:
clear_support(device, key)
else:
print(f"ERC20 on {device}: adding duplicate {key}")
set_unsupported(device, key, ERC20_DUPLICATE_KEY)
for key in nondups:
print(f"ERC20 on {device}: clearing non-duplicate {key}")
clear_support(device, key)
def clear_erc20_mixed_buckets(buckets):
for bucket in buckets.values():
tokens = [coin for coin in bucket if coin_info.is_token(coin)]
if tokens == bucket:
continue
if len(tokens) == 1:
tokens[0]["duplicate"] = False
@click.group() @click.group()
def cli(): def cli():
pass pass
@ -228,10 +165,9 @@ def cli():
def fix(dry_run): def fix(dry_run):
"""Fix expected problems. """Fix expected problems.
Prunes orphaned keys and ensures that ERC20 duplicate info matches support info. Currently only prunes orphaned keys.
""" """
all_coins, buckets = coin_info.coin_info_with_duplicates() all_coins = coin_info.coin_info()
clear_erc20_mixed_buckets(buckets)
coins_dict = all_coins.as_dict() coins_dict = all_coins.as_dict()
orphaned = find_orphaned_support_keys(coins_dict) orphaned = find_orphaned_support_keys(coins_dict)
@ -240,32 +176,25 @@ def fix(dry_run):
for device in SUPPORT_INFO: for device in SUPPORT_INFO:
clear_support(device, orphan) clear_support(device, orphan)
process_erc20(coins_dict)
if not dry_run: if not dry_run:
write_support_info() write_support_info()
@cli.command() @cli.command()
# fmt: off # fmt: off
@click.option("-T", "--check-tokens", is_flag=True, help="Also check unsupported ERC20 tokens, ignored by default")
@click.option("-m", "--ignore-missing", is_flag=True, help="Do not fail on missing supportinfo") @click.option("-m", "--ignore-missing", is_flag=True, help="Do not fail on missing supportinfo")
# fmt: on # fmt: on
def check(check_tokens, ignore_missing): def check(ignore_missing):
"""Check validity of support information. """Check validity of support information.
Ensures that `support.json` data is well formed, there are no keys without Ensures that `support.json` data is well formed, there are no keys without
corresponding coins, and there are no coins without corresponding keys. corresponding coins, and there are no coins without corresponding keys.
If `--check-tokens` is specified, the check will also take into account ERC20 tokens
without support info. This is disabled by default, because support info for ERC20
tokens is not strictly required.
If `--ignore-missing` is specified, the check will display coins with missing If `--ignore-missing` is specified, the check will display coins with missing
support info, but will not fail when missing coins are found. This is support info, but will not fail when missing coins are found. This is
useful in Travis. useful in Travis.
""" """
all_coins, buckets = coin_info.coin_info_with_duplicates() all_coins = coin_info.coin_info()
clear_erc20_mixed_buckets(buckets)
coins_dict = all_coins.as_dict() coins_dict = all_coins.as_dict()
checks_ok = True checks_ok = True
@ -282,8 +211,6 @@ def check(check_tokens, ignore_missing):
missing = find_unsupported_coins(coins_dict) missing = find_unsupported_coins(coins_dict)
for device, values in missing.items(): for device, values in missing.items():
if not check_tokens:
values = [coin for coin in values if not coin_info.is_token(coin)]
if values: if values:
if not ignore_missing: if not ignore_missing:
checks_ok = False checks_ok = False
@ -291,12 +218,6 @@ def check(check_tokens, ignore_missing):
for coin in values: for coin in values:
print(f"{coin['key']} - {coin['name']}") print(f"{coin['key']} - {coin['name']}")
supported_dups = find_supported_duplicate_tokens(coins_dict)
for key in supported_dups:
coin = coins_dict[key]
checks_ok = False
print(f"Token {coin['key']} ({coin['name']}) is duplicate but supported")
if not checks_ok: if not checks_ok:
print("Some checks have failed") print("Some checks have failed")
sys.exit(1) sys.exit(1)
@ -308,7 +229,6 @@ def check(check_tokens, ignore_missing):
@click.option("--v2", help="Version for TT release (default: guess from latest)") @click.option("--v2", help="Version for TT release (default: guess from latest)")
@click.option("-n", "--dry-run", is_flag=True, help="Do not write changes") @click.option("-n", "--dry-run", is_flag=True, help="Do not write changes")
@click.option("-f", "--force", is_flag=True, help="Proceed even with bad version/device info") @click.option("-f", "--force", is_flag=True, help="Proceed even with bad version/device info")
@click.option("-v", "--verbose", is_flag=True, help="Be more verbose")
@click.option("--skip-testnets/--no-skip-testnets", default=True, help="Automatically exclude testnets") @click.option("--skip-testnets/--no-skip-testnets", default=True, help="Automatically exclude testnets")
# fmt: on # fmt: on
@click.pass_context @click.pass_context
@ -318,7 +238,6 @@ def release(
v2, v2,
dry_run, dry_run,
force, force,
verbose,
skip_testnets, skip_testnets,
): ):
"""Release a new Trezor firmware. """Release a new Trezor firmware.
@ -327,8 +246,7 @@ def release(
By default, marks duplicate tokens and testnets as unsupported, and all coins that By default, marks duplicate tokens and testnets as unsupported, and all coins that
don't have support info are set to the released firmware version. don't have support info are set to the released firmware version.
The tool will ask you to confirm each added coin. ERC20 tokens are added The tool will ask you to confirm each added coin.
automatically. Use `--verbose` to see them.
""" """
latest_releases = coin_info.latest_releases() latest_releases = coin_info.latest_releases()
@ -391,18 +309,8 @@ def release(
if coin not in missing_list: if coin not in missing_list:
missing_list.append(coin) missing_list.append(coin)
tokens = [coin for coin in missing_list if coin_info.is_token(coin)] for coin in missing_list:
nontokens = [coin for coin in missing_list if not coin_info.is_token(coin)] if skip_testnets and coin["is_testnet"]:
for coin in tokens:
key = coin["key"]
# assert not coin.get("duplicate"), key
if verbose:
print(f"Adding missing {key} ({coin['name']})")
for device, version in versions.items():
support_setdefault(device, key, version)
for coin in nontokens:
if skip_testnets and "testnet" in coin["name"].lower():
for device, version in versions.items(): for device, version in versions.items():
support_setdefault(device, coin["key"], False, "(AUTO) exclude testnet") support_setdefault(device, coin["key"], False, "(AUTO) exclude testnet")
else: else:
@ -455,10 +363,6 @@ def set_support_value(key, entries, reason):
click.echo("Use 'support.py show' to search for the right one.") click.echo("Use 'support.py show' to search for the right one.")
sys.exit(1) sys.exit(1)
if coins[key].get("duplicate") and coin_info.is_token(coins[key]):
shortcut = coins[key]["shortcut"]
click.echo(f"Note: shortcut {shortcut} is a duplicate.")
for entry in entries: for entry in entries:
try: try:
device, value = entry.split("=", maxsplit=1) device, value = entry.split("=", maxsplit=1)