Alternate HD path structure: BIP, blog, or wat?
7 messages, 19 to 20 Jun 2015. Started by Matt Smith.
Matt Smith (3), Matt @ Envrin Group (2), Andreas Petersson, Jonas Schnelli
Hey guys,
The crew at Gem is considering a new HD wallet path structure for our wallets, which are coin-agnostic, that separates the coin_type field into two fields as such:
m / purpose' / network' / asset_type' / account' / change / index
where network refers to the blockchain (0 - bitcoin, 1 - testnet3, 2 - litecoin, etc) and the new asset_type refers to the kind of asset to be held in accounts below that path (Open Assets, Omni, Counterparty).
The intent is to allow us to validate the address format, select the appropriate daemon to scan for tokenized assets, and choose multiple blockchain data sources (that may not know anything about token systems running on the blockchain they expose) relevant to an HDNode in the wallet using only information in the HDNode's path -- without having to maintain an explicit mapping of coin_type -> network.
For example, we already have the issue of mapping network identifiers because of the lack of standardization across cryptocurrency libraries which ends up being ugly and obnoxious to maintain, i.e.
netcode_map = {
testnet: testnet3,
bitcoin_testnet: testnet3,
testnet3: testnet3,
XTN: testnet3, ...
}
netcode_i_want = netcode_map[netcode_returned_by_libwhatever]
We want to avoid maintaining a similar asset_type_to_blockchain mapping. Additionally, it would be helpful for utxo selection to exclude utxos tied to assets based on path.
BIP43 seems to suggest that we request a BIP number and publish an informational BIP specifying the new purpose. If that's not appropriate, then maybe we just need to publish the information in a blog post to allow any wallet developers who want to to implement import_from_gem_structure.
I was also wondering if anyone had previously suggested something similar that I missed when cruising the mailing list archives on the subject.
Thanks, – Matt Smith | Gem https://gem.co | GH: @thedoctor
Hi Matt,
I think your best bet is probably just push it out privately via blog post / Github, and see if it gains any traction with other developers.
I'm a little uncertain as to the relevance though. All those variables (purpose, network, asset_type, account, change, index) need to be stored internally within the wallet database, as there's no way to retrieve the path used from just the address, correct? In that case, what's the meaning of that exact path structure when a) it can't be retrieved from just the address, and b) the values will be stored internally within the wallet when you lookup the address.
Matt
On 06/20/2015 03:42 AM, Matt Smith wrote:
Quoted text (58 lines)
> Hey guys,
>
> The crew at Gem is considering a new HD wallet path structure for our
> wallets, which are coin-agnostic, that separates the coin_type field
> into two fields as such:
>
> m / purpose' / network' / asset_type' / account' / change / index
>
> where network refers to the blockchain (0 - bitcoin, 1 - testnet3, 2 -
> litecoin, etc) and the new asset_type refers to the kind of asset to be
> held in accounts below that path (Open Assets, Omni, Counterparty).
>
> The intent is to allow us to validate the address format, select the
> appropriate daemon to scan for tokenized assets, and choose multiple
> blockchain data sources (that may not know anything about token systems
> running on the blockchain they expose) relevant to an HDNode in the
> wallet using only information in the HDNode's path -- without having to
> maintain an explicit mapping of coin_type -> network.
>
> For example, we already have the issue of mapping network identifiers
> because of the lack of standardization across cryptocurrency libraries
> which ends up being ugly and obnoxious to maintain, i.e.
>
> netcode_map = {
> testnet: testnet3,
> bitcoin_testnet: testnet3,
> testnet3: testnet3,
> XTN: testnet3, ...
> }
> netcode_i_want = netcode_map[netcode_returned_by_libwhatever]
>
> We want to avoid maintaining a similar asset_type_to_blockchain mapping.
> Additionally, it would be helpful for utxo selection to exclude utxos
> tied to assets based on path.
>
> BIP43 seems to suggest that we request a BIP number and publish an
> informational BIP specifying the new purpose. If that's not appropriate,
> then maybe we just need to publish the information in a blog post to
> allow any wallet developers who want to to implement
> import_from_gem_structure.
>
> I was also wondering if anyone had previously suggested something
> similar that I missed when cruising the mailing list archives on the
> subject.
>
> Thanks,
> --
> Matt Smith | Gem
> https://gem.co | GH: @thedoctor
>
>
> ------------------------------------------------------------------------------
>
>
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-developmentI'm not sure I understand your question about the need to store paths in the wallet database -- there's no way to infer the path of an address inside an HD wallet from the address alone (short of an exhaustive search), and HD wallets need to store either the paths, addresses, or both that have been previously derived/used to monitor the blockchain usefully, but those facts aren't new or specific to this path format.
The motivation for this path structure over standard bip44 is that it separates the concept of network (or which blockchain I'm using) and coin_type (or what kind of thing I'm sending over that blockchain).
This is useful, for example, if I want to import a wallet into my application and I know that an account was in use at
m/##'/0'/99'/0'
where 99 is the identifier for, say, counterparty - I only need to check the addresses derived below that path for balances against counterpartyd. It may be worth pointing out that I expect multisig HD wallet imports to require master keys and a list of account paths – not a list of addresses, as it's very possible that a new address could be derived between the time when the wallet data was exported and when it will be imported.
This use case might be very specific to our model, but the reason I figured we should request a BIP # for this is that to start using it, we need to pick a number for the purpose field and don't want to do it arbitrarily (and risk having to change it later) or overload 44 (which would be misleading).
Did I either a) answer or b) misunderstand your questions?
Quoted text (3 lines)
>m/##'/0'/99'/0' > >where 99 is the identifier for, say, counterparty
What is stopping you from using m/44'/9'/a'/c/i as descibed here: http://doc.satoshilabs.com/slips/slip-0044.html
to avoid having an internal mapping from 9'-> 0' to find out what blockchain to query, this sounds like it should be trivial for any wallet.
Say you generate a child key using the path m/6'/4'/7'/99'/0/196, which is what your proposed path structure would be, and it results in the address 1DpY7PtPVURvjrGsdAjbZAZ7cL9GD8tc5w.
When the wallet notices a transaction in the blockchain that has 1DpY7PtPVURvjrGsdAjbZAZ7cL9GD8tc5w as an output, it's going to have to lookup the address within its database to get the values 6/4/7/99/0/196, as there's no way to retrieve them from just the address. So technically, you might as well just use m/account'/change/index if using hardened child keys, or m/change/index if not, as recommended, because the wallet will still function the exact same way.
Matt
On 06/20/2015 06:31 AM, Matt Smith wrote:
Quoted text (52 lines)
> I'm not sure I understand your question about the need to store paths in > the wallet database -- there's no way to infer the path of an address > inside an HD wallet from the address alone (short of an exhaustive > search), and HD wallets need to store either the paths, addresses, or > both that have been previously derived/used to monitor the blockchain > usefully, but those facts aren't new or specific to this path format. > > The motivation for this path structure over standard bip44 is that it > separates the concept of network (or which blockchain I'm using) and > coin_type (or what kind of thing I'm sending over that blockchain). > > This is useful, for example, if I want to import a wallet into my > application and I know that an account was in use at > > m/##'/0'/99'/0' > > where 99 is the identifier for, say, counterparty - I only need to check > the addresses derived below that path for balances against > counterpartyd. It may be worth pointing out that I expect multisig HD > wallet imports to require master keys and a list of account paths – not > a list of addresses, as it's very possible that a new address could be > derived between the time when the wallet data was exported and when it > will be imported. > > This use case might be very specific to our model, but the reason I > figured we should request a BIP # for this is that to start using it, we > need to pick a number for the purpose field and don't want to do it > arbitrarily (and risk having to change it later) or overload 44 (which > would be misleading). > > Did I either a) answer or b) misunderstand your questions? > -- > Matt Smith | Gem > https://gem.co | GH: @thedoctor > > > > On 6/19/15 2:25 PM, Matt @ Envrin Group wrote: >> Hi Matt, >> >> I think your best bet is probably just push it out privately via blog >> post / Github, and see if it gains any traction with other developers. >> >> I'm a little uncertain as to the relevance though. All those variables >> (purpose, network, asset_type, account, change, index) need to be stored >> internally within the wallet database, as there's no way to retrieve the >> path used from just the address, correct? In that case, what's the >> meaning of that exact path structure when a) it can't be retrieved from >> just the address, and b) the values will be stored internally within the >> wallet when you lookup the address. >> >> Matt
Quoted text (2 lines)
> to avoid having an internal mapping from 9'-> 0' to find out what > blockchain to query, this sounds like it should be trivial for any wallet.
Trivial to implement, a headache to *maintain*
But if a new platform is released on an existing blockchain, my wallet doesn't need to know about the new magic number it claims in order to handle it correctly.
Say I make a new token layer, BobCoin, which runs on bitcoin and say I use an HD wallet and always generate new BobCoin token addresses as m/##'/0'/808'/*'/*/*. If I import that wallet into older HD wallet software that doesn't know anything about BobCoin, it will still:
- understand what blockchain to query for utxos on the addresses below that path - be able to generate valid BobCoin addresses without any updates
I think this is particularly valuable if you're developing against a platform where updates can't be forced on clients.
To be clear: I am not suggesting this as a general-purpose successor to BIP44.
– Matt Smith | Gem https://gem.co | GH: @thedoctor
On 6/19/15 5:57 PM, Andreas Petersson wrote:
Quoted text (18 lines)
>> m/##'/0'/99'/0' >> >> where 99 is the identifier for, say, counterparty > > > What is stopping you from using m/44'/9'/a'/c/i as descibed here: > http://doc.satoshilabs.com/slips/slip-0044.html > > to avoid having an internal mapping from 9'-> 0' to find out what > blockchain to query, this sounds like it should be trivial for any wallet. > > > ------------------------------------------------------------------------------ > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development >
Quoted text (7 lines)
> m / purpose' / network' / asset_type' / account' / change / index > > ... > > I was also wondering if anyone had previously suggested something > similar that I missed when cruising the mailing list archives on > the subject.
Hi Matt
a) In my opinion, HD chain standards like Bip43/44 are there to allow users to use different wallets with the same seed/wordlist (switch over or use in parallel). In practice, wallet re-importing / re-creating is not trivial and has risks of missing out spent or spendable coins. Take the current landscape. Electrum doesn’t follow the Bip39 standard [1]. Breadwallet does not support Bip44 [2]. If users start to think, as long as they keep the Bip39 wordlist save, nothing can happen, is a material fallacy (and i see a trend into this direction). A hd seed will not recover your non-blockchain-metadata. HD seed recovery is a disaster recovery method.
Because of that I am unsure if standards of hd chainstructurs does lead users and developers into the right directions. In my opinion, reimporting a hd seed is a experts feature that requires manual configuration.
b) Creating deep level chains like m/a/b/c/d/e/f/g/h requires more cpu power to derive private key with a given seed or wordlist. If you are using hardware wallets like Trezor or Digitalbitbox, you probably feel the required time when signing/deriving.
[1] https://bitcointalk.org/index.php?topic=1082586.0 [2] https://github.com/voisine/breadwallet/issues/131
</jonas>