LearnBitcoin

Bitcoin Dev Archive 2014

New BIP32 structure

100 messages, 26 Mar to 24 Apr 2014. Started by Mike Hearn.

Pavol Rusnak (18), slush (13), Thomas Voegtlin (11), Pieter Wuille (10), Luke Dashjr (10), Mike Hearn (9), Tamas Blummer (9), Allen Piscitello (4), +10 more

Permalink Original thread on gnusha.org

Mike Hearn permalink original

Myself, Thomas V (Electrum) and Marek (Trezor) got together to make sure our BIP32 wallet structures would be compatible - and I discovered that only I was planning to use the default structure.

Because I'm hopeful that we can get a lot of interoperability between wallets with regards to importing 12-words paper wallets, we brainstormed to find a structure acceptable to everyone and ended up with:

  /m/cointype/reserved'/account'/change/n

The extra levels require some explanation:

   - cointype:  This is zero for Bitcoin. This is here to support two
   things, one is supporting alt coins based off the same root seed. Right now
   nobody seemed very bothered about alt coins but sometimes feature requests
   do come in for this. Arguably there is no need and alt coins could just use
   the same keys as Bitcoin, but it may help avoid confusion if they don't.
   More usefully, cointype can distinguish between keys intended for things
   like multisig outputs, e.g. for watchdog services. This means if your
   wallet does not know about the extra protocol layers involved in this, it
   can still import the "raw" money and it will just ignore/not see the keys
   used in more complex transactions.
   - reserved is for "other stuff". I actually don't recall why we ended up
   with this. It may have been intended to split out multisig outputs etc from
   cointype. Marek, Thomas?
   - account is for keeping essentially wallets-within-a-wallet to avoid
   mixing of coins. If you want that.
   - change is 0 for receiving addresses, 1 for change addresses.
   - n is the actual key index

For bitcoinj we're targeting a deliberately limited feature set for hdw v1 so I would just set the first three values all to zero and that is a perfectly fine way to be compatible.

The goal here is that the same seed can be written down once, and meet all the users needs, whilst still allowing some drift between what wallets support.

Pieter made the I think valid point that you can't really encode how keys are meant to be used into just an HDW hierarchy and normally you'd need some metadata as well. However, I feel interop between wallets is more important than arriving at the most perfect possible arrangement, which feels a little like bikeshedding, so I'm happy to just go with the flow on this one.

Andreas Schildbach in reply to Mike Hearn permalink original

Thanks for starting the discussion on finding a better structure.

For me, the most important thing is either we're 100% interoperable or 0%. There should not be anything inbetween, as users will delete seeds without knowing there is still money in them on another implementation. I heard from multiple sources that using this standard some wallets will only see a subset of the addresses/keys of some other wallets. Implementation differences can always happen (and should addresses as bugs), but I think its unacceptable that this source of issues is by design.

I suggest we agree on an even simpler least common denominator and wallets that want to implement some feature on top of that can do but are encouraged to pick a totally different "cointype". I guess that would mean removing reserved and account.

I'm still thinking it might be a good idea to have a separate chain for "refunds". Refunds will be rarely used and thus need a much slower moving window than receiving addresses or change.

On 03/26/2014 09:49 PM, Mike Hearn wrote:

Quoted text (62 lines)
> Myself, Thomas V (Electrum) and Marek (Trezor) got together to make sure
> our BIP32 wallet structures would be compatible - and I discovered that
> only I was planning to use the default structure.
>
> Because I'm hopeful that we can get a lot of interoperability between
> wallets with regards to importing 12-words paper wallets, we
> brainstormed to find a structure acceptable to everyone and ended up with:
>
>   /m/cointype/reserved'/account'/change/n
>
> The extra levels require some explanation:
>
>   * cointype:  This is zero for Bitcoin. This is here to support two
>     things, one is supporting alt coins based off the same root seed.
>     Right now nobody seemed very bothered about alt coins but sometimes
>     feature requests do come in for this. Arguably there is no need and
>     alt coins could just use the same keys as Bitcoin, but it may help
>     avoid confusion if they don't.
>
>     More usefully, cointype can distinguish between keys intended for
>     things like multisig outputs, e.g. for watchdog services. This means
>     if your wallet does not know about the extra protocol layers
>     involved in this, it can still import the "raw" money and it will
>     just ignore/not see the keys used in more complex transactions.
>
>   * reserved is for "other stuff". I actually don't recall why we ended
>     up with this. It may have been intended to split out multisig
>     outputs etc from cointype. Marek, Thomas?
>
>   * account is for keeping essentially wallets-within-a-wallet to avoid
>     mixing of coins. If you want that.
>
>   * change is 0 for receiving addresses, 1 for change addresses.
>
>   * n is the actual key index
>
> For bitcoinj we're targeting a deliberately limited feature set for hdw
> v1 so I would just set the first three values all to zero and that is a
> perfectly fine way to be compatible.
>
> The goal here is that the same seed can be written down once, and meet
> all the users needs, whilst still allowing some drift between what
> wallets support.
>
> Pieter made the I think valid point that you can't really encode how
> keys are meant to be used into just an HDW hierarchy and normally you'd
> need some metadata as well. However, I feel interop between wallets is
> more important than arriving at the most perfect possible arrangement,
> which feels a little like bikeshedding, so I'm happy to just go with the
> flow on this one.
>
>
>
> ------------------------------------------------------------------------------
>
>
>
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
Alan Reiner in reply to Andreas Schildbach permalink original

This might be tangential, but the comment about "refund" chains reminded me. Armory will be implementing multi-sig/linked wallets where a each device has a parallel HDW branch and produces P2SH addresses. For those types of wallets, I plan to allocate two chains /per signing authority/. If you have a shared 2-of-2 wallet split between your phone and your spouse's phone, your phone would distribute addresses on P2SH chain 0 and generate change addresses on P2SH chain 1. Your spouse's phone would use chains 2 and 3.

So if you and your spouse switch to a new app that supports M-of-N linked wallets, it should search for coin history along the first 2*N chains. -Alan

On 03/26/2014 07:37 PM, Andreas Schildbach wrote:

Quoted text (90 lines)
> Thanks for starting the discussion on finding a better structure.
>
> For me, the most important thing is either we're 100% interoperable or
> 0%. There should not be anything inbetween, as users will delete seeds
> without knowing there is still money in them on another implementation.
> I heard from multiple sources that using this standard some wallets will
> only see a subset of the addresses/keys of some other wallets.
> Implementation differences can always happen (and should addresses as
> bugs), but I think its unacceptable that this source of issues is by design.
>
> I suggest we agree on an even simpler least common denominator and
> wallets that want to implement some feature on top of that can do but
> are encouraged to pick a totally different "cointype". I guess that
> would mean removing reserved and account.
>
> I'm still thinking it might be a good idea to have a separate chain for
> "refunds". Refunds will be rarely used and thus need a much slower
> moving window than receiving addresses or change.
>
>
> On 03/26/2014 09:49 PM, Mike Hearn wrote:
>> Myself, Thomas V (Electrum) and Marek (Trezor) got together to make sure
>> our BIP32 wallet structures would be compatible - and I discovered that
>> only I was planning to use the default structure.
>>
>> Because I'm hopeful that we can get a lot of interoperability between
>> wallets with regards to importing 12-words paper wallets, we
>> brainstormed to find a structure acceptable to everyone and ended up with:
>>
>>   /m/cointype/reserved'/account'/change/n
>>
>> The extra levels require some explanation:
>>
>>   * cointype:  This is zero for Bitcoin. This is here to support two
>>     things, one is supporting alt coins based off the same root seed.
>>     Right now nobody seemed very bothered about alt coins but sometimes
>>     feature requests do come in for this. Arguably there is no need and
>>     alt coins could just use the same keys as Bitcoin, but it may help
>>     avoid confusion if they don't.
>>
>>     More usefully, cointype can distinguish between keys intended for
>>     things like multisig outputs, e.g. for watchdog services. This means
>>     if your wallet does not know about the extra protocol layers
>>     involved in this, it can still import the "raw" money and it will
>>     just ignore/not see the keys used in more complex transactions.
>>
>>   * reserved is for "other stuff". I actually don't recall why we ended
>>     up with this. It may have been intended to split out multisig
>>     outputs etc from cointype. Marek, Thomas?
>>
>>   * account is for keeping essentially wallets-within-a-wallet to avoid
>>     mixing of coins. If you want that.
>>
>>   * change is 0 for receiving addresses, 1 for change addresses.
>>
>>   * n is the actual key index
>>
>> For bitcoinj we're targeting a deliberately limited feature set for hdw
>> v1 so I would just set the first three values all to zero and that is a
>> perfectly fine way to be compatible.
>>
>> The goal here is that the same seed can be written down once, and meet
>> all the users needs, whilst still allowing some drift between what
>> wallets support.
>>
>> Pieter made the I think valid point that you can't really encode how
>> keys are meant to be used into just an HDW hierarchy and normally you'd
>> need some metadata as well. However, I feel interop between wallets is
>> more important than arriving at the most perfect possible arrangement,
>> which feels a little like bikeshedding, so I'm happy to just go with the
>> flow on this one.
>>
>>
>>
>> ------------------------------------------------------------------------------
>>
>>
>>
>> _______________________________________________
>> Bitcoin-development mailing list
>> Bitcoin-development@lists•sourceforge•net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Allen Piscitello in reply to Mike Hearn permalink original

For every branch (say multiple accounts), how would a new wallet be able to know how many sequence items to scan? It seems like not only do you need to have standard rules for the hierarchy, but how the usage can be detected. The other scanning seems pretty straightforward. For accounts, it seems like you could have a situation where you want to initially set up 10 different accounts, but only account #10 gets any transactions. If a new wallet was trying to scan with this seed, it would have to know to keep scanning each account until it found the account. The user would have to be responsible for knowing how many accounts there are, or some rules would need to be in place to not allow creating accounts until earlier accounts can be proven to have existed in the blockchain. Or I am missing something.

-Allen

On Wed, Mar 26, 2014 at 3:49 PM, Mike Hearn <mike@plan99•net> wrote:

Quoted text (60 lines)
> Myself, Thomas V (Electrum) and Marek (Trezor) got together to make sure
> our BIP32 wallet structures would be compatible - and I discovered that
> only I was planning to use the default structure.
>
> Because I'm hopeful that we can get a lot of interoperability between
> wallets with regards to importing 12-words paper wallets, we brainstormed
> to find a structure acceptable to everyone and ended up with:
>
>   /m/cointype/reserved'/account'/change/n
>
> The extra levels require some explanation:
>
>    - cointype:  This is zero for Bitcoin. This is here to support two
>    things, one is supporting alt coins based off the same root seed. Right now
>    nobody seemed very bothered about alt coins but sometimes feature requests
>    do come in for this. Arguably there is no need and alt coins could just use
>    the same keys as Bitcoin, but it may help avoid confusion if they don't.
>
>    More usefully, cointype can distinguish between keys intended for
>    things like multisig outputs, e.g. for watchdog services. This means if
>    your wallet does not know about the extra protocol layers involved in this,
>    it can still import the "raw" money and it will just ignore/not see the
>    keys used in more complex transactions.
>
>    - reserved is for "other stuff". I actually don't recall why we ended
>    up with this. It may have been intended to split out multisig outputs etc
>    from cointype. Marek, Thomas?
>
>    - account is for keeping essentially wallets-within-a-wallet to avoid
>    mixing of coins. If you want that.
>
>    - change is 0 for receiving addresses, 1 for change addresses.
>
>    - n is the actual key index
>
> For bitcoinj we're targeting a deliberately limited feature set for hdw v1
> so I would just set the first three values all to zero and that is a
> perfectly fine way to be compatible.
>
> The goal here is that the same seed can be written down once, and meet all
> the users needs, whilst still allowing some drift between what wallets
> support.
>
> Pieter made the I think valid point that you can't really encode how keys
> are meant to be used into just an HDW hierarchy and normally you'd need
> some metadata as well. However, I feel interop between wallets is more
> important than arriving at the most perfect possible arrangement, which
> feels a little like bikeshedding, so I'm happy to just go with the flow on
> this one.
>
>
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
Troy Benjegerdes in reply to Mike Hearn permalink original

On Wed, Mar 26, 2014 at 09:49:39PM +0100, Mike Hearn wrote:

Quoted text (17 lines)
> Myself, Thomas V (Electrum) and Marek (Trezor) got together to make sure
> our BIP32 wallet structures would be compatible - and I discovered that
> only I was planning to use the default structure.
>
> Because I'm hopeful that we can get a lot of interoperability between
> wallets with regards to importing 12-words paper wallets, we brainstormed
> to find a structure acceptable to everyone and ended up with:
>
>   /m/cointype/reserved'/account'/change/n
>
> The extra levels require some explanation:
>
>    - cointype:  This is zero for Bitcoin. This is here to support two
>    things, one is supporting alt coins based off the same root seed. Right now
>    nobody seemed very bothered about alt coins but sometimes feature requests
>    do come in for this. Arguably there is no need and alt coins could just use
>    the same keys as Bitcoin, but it may help avoid confusion if they don't.

Using the same keys across different altcoins seems like an exceedingly bad opsec practice. Cointype is critical, as well as having a predictable and deterministic mapping of alt coins to Cointype.

What should I be using for Catcoin, for instance? the CAT symbol all the exchanges use, or do we set up a 'registry', or some other mechanism?

I'd venture to guess the altcoin market is, or soon will be larger in US dollar value trade volume than Bitcoin, so *some* of us are quite bothered by the wailing and gnashing of teeth that occurs on this list at mere thought of such heresy.

Mike Belshe in reply to Alan Reiner permalink original

On Wed, Mar 26, 2014 at 6:01 PM, Alan Reiner <etotheipi@gmail•com> wrote:

Quoted text (12 lines)
>  This might be tangential, but the comment about "refund" chains reminded
> me.  Armory will be implementing multi-sig/linked wallets where a each
> device has a parallel HDW branch and produces P2SH addresses.  For those
> types of wallets, I plan to allocate two chains *per signing authority*.
> If you have a shared 2-of-2 wallet split between your phone and your
> spouse's phone, your phone would distribute addresses on P2SH chain 0 and
> generate change addresses on P2SH chain 1.  Your spouse's phone would use
> chains 2 and 3.
>
> So if you and your spouse switch to a new app that supports M-of-N linked
> wallets, it should search for coin history along the first 2*N chains.
>

In general with multisig, we should probably discourage using a common root keychain for multiple keys in the same P2SH address. The reason is because if you have the single, root private key, you can sign them all. This generally goes against the point of multisig - which was trying to introduce a system which had multiple keys required to unlock :-)

For the BitGo BIP32 implementation we are doing the following:
   - users create 2 extended keys
   - bitgo creates 1 extended key for that user
   - the user can create any number of separate branches (accounts) of P2SH
addresses from those extended keys.
   - change/public addresses are generated from the 0/1 branches
   - new addresses are generated from there.
   - each time a new change address is generated, all 3 keys in the P2SH
are rotated.  This makes it so that for any chain depth *i* of the P2SH
address, we're using the *i*th key in each keychain.  For privacy reasons,
we don't need to rotate all keys.  But I think it makes management simpler
and probably more ready for interop.
So the paths in a user's keychain look exactly like BIP32's layout:
     m/<AccountIndex>/<ExternalOrInternal>/<AddressIndex>

Another issue unique to P2SH addresses is order of the keys. There has been talk in this list of how to sort them. While sorting is simplifying for interop if you never use new change addresses, I believe sorting makes wallet management more difficult when BIP32 is introduced.

Specifically, imagine a P2SH address with keys held by Bob, Charlie, and Dana. If you sort the pubkeys, then with each BIP32 chain, the order of the keys could be different. By maintaining key order, we always know which of the pubkeys belongs to which user without having to do any lookups.

Obviously this can all be calculated too. But the sorting seems gratuitous to me - it adds complexity but offers little value. On the other hand, it can be really handy to know that key #0 for this address is always Bob's key, regardless of how deep the chaining goes.

Mike

Quoted text (105 lines)
>
> -Alan
>
>
>
>
> On 03/26/2014 07:37 PM, Andreas Schildbach wrote:
>
> Thanks for starting the discussion on finding a better structure.
>
> For me, the most important thing is either we're 100% interoperable or
> 0%. There should not be anything inbetween, as users will delete seeds
> without knowing there is still money in them on another implementation.
> I heard from multiple sources that using this standard some wallets will
> only see a subset of the addresses/keys of some other wallets.
> Implementation differences can always happen (and should addresses as
> bugs), but I think its unacceptable that this source of issues is by design.
>
> I suggest we agree on an even simpler least common denominator and
> wallets that want to implement some feature on top of that can do but
> are encouraged to pick a totally different "cointype". I guess that
> would mean removing reserved and account.
>
> I'm still thinking it might be a good idea to have a separate chain for
> "refunds". Refunds will be rarely used and thus need a much slower
> moving window than receiving addresses or change.
>
>
> On 03/26/2014 09:49 PM, Mike Hearn wrote:
>
>  Myself, Thomas V (Electrum) and Marek (Trezor) got together to make sure
> our BIP32 wallet structures would be compatible - and I discovered that
> only I was planning to use the default structure.
>
> Because I'm hopeful that we can get a lot of interoperability between
> wallets with regards to importing 12-words paper wallets, we
> brainstormed to find a structure acceptable to everyone and ended up with:
>
>   /m/cointype/reserved'/account'/change/n
>
> The extra levels require some explanation:
>
>   * cointype:  This is zero for Bitcoin. This is here to support two
>     things, one is supporting alt coins based off the same root seed.
>     Right now nobody seemed very bothered about alt coins but sometimes
>     feature requests do come in for this. Arguably there is no need and
>     alt coins could just use the same keys as Bitcoin, but it may help
>     avoid confusion if they don't.
>
>     More usefully, cointype can distinguish between keys intended for
>     things like multisig outputs, e.g. for watchdog services. This means
>     if your wallet does not know about the extra protocol layers
>     involved in this, it can still import the "raw" money and it will
>     just ignore/not see the keys used in more complex transactions.
>
>   * reserved is for "other stuff". I actually don't recall why we ended
>     up with this. It may have been intended to split out multisig
>     outputs etc from cointype. Marek, Thomas?
>
>   * account is for keeping essentially wallets-within-a-wallet to avoid
>     mixing of coins. If you want that.
>
>   * change is 0 for receiving addresses, 1 for change addresses.
>
>   * n is the actual key index
>
> For bitcoinj we're targeting a deliberately limited feature set for hdw
> v1 so I would just set the first three values all to zero and that is a
> perfectly fine way to be compatible.
>
> The goal here is that the same seed can be written down once, and meet
> all the users needs, whilst still allowing some drift between what
> wallets support.
>
> Pieter made the I think valid point that you can't really encode how
> keys are meant to be used into just an HDW hierarchy and normally you'd
> need some metadata as well. However, I feel interop between wallets is
> more important than arriving at the most perfect possible arrangement,
> which feels a little like bikeshedding, so I'm happy to just go with the
> flow on this one.
>
>
>
> ------------------------------------------------------------------------------
>
>
>
> _______________________________________________
> Bitcoin-development mailing listBitcoin-development@lists•sourceforge•nethttps://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>  ------------------------------------------------------------------------------
> _______________________________________________
> Bitcoin-development mailing listBitcoin-development@lists•sourceforge•nethttps://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
>
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
Tamas Blummer in reply to Mike Hearn permalink original

We had a similar meeting with Andreas Schildbach (Android Bitcoin Wallet), Jan Moller, Andreas Petersson (Mycelium), Thomas V (Electrum), Tamas Blummer, Tamas Bartfai (Bits of Proof) at the Inside Bitcoin Conference in Berlin.

I remember that there were different opinions on how to use a hierarchy and it did seem to me they could eventually be "standardized" for the retail customer but definitelly not for corporate use, where hierarchy will certainly map to organisational hierarchy or cost centres.

A notable suggestion was to instead of building a directory of magic numbers (like 0 for Bitcoin, 1 for Litecoin etc) use a hash of the word "Bitcoin", "Litecoin", "Dogecoin", so collosion is unlikely and cetral directory is not needed.

Regards,

Tamas Blummer http://bitsofproof.com

On 26.03.2014, at 21:49, Mike Hearn <mike@plan99•net> wrote:

Quoted text (29 lines)
> Myself, Thomas V (Electrum) and Marek (Trezor) got together to make sure our BIP32 wallet structures would be compatible - and I discovered that only I was planning to use the default structure.
>
> Because I'm hopeful that we can get a lot of interoperability between wallets with regards to importing 12-words paper wallets, we brainstormed to find a structure acceptable to everyone and ended up with:
>
>   /m/cointype/reserved'/account'/change/n
>
> The extra levels require some explanation:
> cointype:  This is zero for Bitcoin. This is here to support two things, one is supporting alt coins based off the same root seed. Right now nobody seemed very bothered about alt coins but sometimes feature requests do come in for this. Arguably there is no need and alt coins could just use the same keys as Bitcoin, but it may help avoid confusion if they don't.
>
> More usefully, cointype can distinguish between keys intended for things like multisig outputs, e.g. for watchdog services. This means if your wallet does not know about the extra protocol layers involved in this, it can still import the "raw" money and it will just ignore/not see the keys used in more complex transactions.
>
> reserved is for "other stuff". I actually don't recall why we ended up with this. It may have been intended to split out multisig outputs etc from cointype. Marek, Thomas?
>
> account is for keeping essentially wallets-within-a-wallet to avoid mixing of coins. If you want that.
>
> change is 0 for receiving addresses, 1 for change addresses.
>
> n is the actual key index
> For bitcoinj we're targeting a deliberately limited feature set for hdw v1 so I would just set the first three values all to zero and that is a perfectly fine way to be compatible.
>
> The goal here is that the same seed can be written down once, and meet all the users needs, whilst still allowing some drift between what wallets support.
>
> Pieter made the I think valid point that you can't really encode how keys are meant to be used into just an HDW hierarchy and normally you'd need some metadata as well. However, I feel interop between wallets is more important than arriving at the most perfect possible arrangement, which feels a little like bikeshedding, so I'm happy to just go with the flow on this one.
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Mike Hearn in reply to Tamas Blummer permalink original

At this point I'm not sure how much further work people want to do on this: I got the impression that Trezor will ship soon, and Thomas V seemed satisfied too. I'm not sure we can get all wallets to be fully interoperable given the flexibility inherent in BIP32 and people's differing use cases.

Andreas: good point but I really hope nobody ever deletes a seed after all this work we put in to make backups so easy! I'm not sure we can really stop it anyway: not unless we make the seed a full blown data structure with hints to other apps that they should refuse to load it. And it's a bit late for that now.

On Thu, Mar 27, 2014 at 8:09 AM, Tamas Blummer <tamas@bitsofproof•com>wrote:

Quoted text (82 lines)
> We had a similar meeting with Andreas Schildbach (Android Bitcoin Wallet),
> Jan Moller, Andreas  Petersson (Mycelium), Thomas V (Electrum), Tamas
> Blummer, Tamas Bartfai (Bits of Proof)
> at the Inside Bitcoin Conference in Berlin.
>
> I remember that there were different opinions on how to use a hierarchy
> and it did seem to me they could eventually be "standardized" for the
> retail customer but definitelly not for corporate use,
> where hierarchy will certainly map to organisational hierarchy or cost
> centres.
>
> A notable suggestion was to instead of building a directory of magic
> numbers (like 0 for Bitcoin, 1 for Litecoin etc) use a hash of the word
> "Bitcoin", "Litecoin", "Dogecoin", so collosion is unlikely and
> cetral directory is not needed.
>
> Regards,
>
> Tamas Blummer
> http://bitsofproof.com
>
> On 26.03.2014, at 21:49, Mike Hearn <mike@plan99•net> wrote:
>
> Myself, Thomas V (Electrum) and Marek (Trezor) got together to make sure
> our BIP32 wallet structures would be compatible - and I discovered that
> only I was planning to use the default structure.
>
> Because I'm hopeful that we can get a lot of interoperability between
> wallets with regards to importing 12-words paper wallets, we brainstormed
> to find a structure acceptable to everyone and ended up with:
>
>   /m/cointype/reserved'/account'/change/n
>
> The extra levels require some explanation:
>
>    - cointype:  This is zero for Bitcoin. This is here to support two
>    things, one is supporting alt coins based off the same root seed. Right now
>    nobody seemed very bothered about alt coins but sometimes feature requests
>    do come in for this. Arguably there is no need and alt coins could just use
>    the same keys as Bitcoin, but it may help avoid confusion if they don't.
>
>    More usefully, cointype can distinguish between keys intended for
>    things like multisig outputs, e.g. for watchdog services. This means if
>    your wallet does not know about the extra protocol layers involved in this,
>    it can still import the "raw" money and it will just ignore/not see the
>    keys used in more complex transactions.
>
>    - reserved is for "other stuff". I actually don't recall why we ended
>    up with this. It may have been intended to split out multisig outputs etc
>    from cointype. Marek, Thomas?
>
>    - account is for keeping essentially wallets-within-a-wallet to avoid
>    mixing of coins. If you want that.
>
>    - change is 0 for receiving addresses, 1 for change addresses.
>
>    - n is the actual key index
>
> For bitcoinj we're targeting a deliberately limited feature set for hdw v1
> so I would just set the first three values all to zero and that is a
> perfectly fine way to be compatible.
>
> The goal here is that the same seed can be written down once, and meet all
> the users needs, whilst still allowing some drift between what wallets
> support.
>
> Pieter made the I think valid point that you can't really encode how keys
> are meant to be used into just an HDW hierarchy and normally you'd need
> some metadata as well. However, I feel interop between wallets is more
> important than arriving at the most perfect possible arrangement, which
> feels a little like bikeshedding, so I'm happy to just go with the flow on
> this one.
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
>
Thomas Voegtlin in reply to Andreas Schildbach permalink original

Le 27/03/2014 00:37, Andreas Schildbach a écrit :

Quoted text (5 lines)
> Thanks for starting the discussion on finding a better structure.
>
> For me, the most important thing is either we're 100% interoperable or
> 0%. There should not be anything inbetween, as users will delete seeds
> without knowing there is still money in them on another implementation.

I believe you have a good point here: we should not advertise wallets as compatible if they are not 100% compatible.

One issue that I have is bandwidth: Electrum (and mycelium) cannot watch as many addresses as they want, because this will create too much traffic on the servers. (especially when servers send utxo merkle proofs for each address, which is not the case yet, but is planned)

For this reason Electrum imposes a constraint on the number of virgin addresses provided to the user. Although the current strategy used by Electrum can certainly be improved, it will not scale up to having every client watching thousands of addresses.

This constraint is not so important for bloom-filter clients. So I guess that it makes sense for Multibit to provide hundreds, or even thousands of virgin addresses to the user, regardless of how they are used. Such a wallet will in general not be recoverable in Electrum, unless the user "helps" the recovery procedure. (or the seed has metadata telling the software that this is a Multibit wallet). So we have a problem here, if we advertise these wallets as compatible.

My opinion, as far as Electrum is concerned, is that merchant accounts should behave differently from regular user accounts: While merchants need to generate an unlimited number of receiving addresses, it is also acceptable for them to have a slightly more complex wallet recovery procedure (for example, the wallet might show an option to "search for more addresses", and it might not need to watch "old" addresses anymore)

OTOH, I don't think we can ask regular users to do this, not only because it adds complexity to the wallet recovery procedure (which makes it scarier), but also because we want fully automated synchronization between different instances of a wallet, using only no other source of information than the blockchain.

The first versions of Electrum allowed users to set the "gap limit" parameter in their GUI preferences, but I removed it from GUI after I realized it was a bad idea (users messed with it and did not understand what happened..)

With bloom filter clients I guess the distinction between these two use cases is not really necessary, because watching addresses is cheap. So it would be good to hear what you and Mike think about this problem. If you decide to let the user create hundreds of unused addresses (and I think it perfectly makes sense for you), then I guess it would be better for Electrum to give up on compatibility, rather than running the risk of seeing only a subset of addresses. Another option is to handle these seeds as "merchant" accounts.

Quoted text (86 lines)
> I heard from multiple sources that using this standard some wallets will
> only see a subset of the addresses/keys of some other wallets.
> Implementation differences can always happen (and should addresses as
> bugs), but I think its unacceptable that this source of issues is by design.
>
> I suggest we agree on an even simpler least common denominator and
> wallets that want to implement some feature on top of that can do but
> are encouraged to pick a totally different "cointype". I guess that
> would mean removing reserved and account.

>
> I'm still thinking it might be a good idea to have a separate chain for
> "refunds". Refunds will be rarely used and thus need a much slower
> moving window than receiving addresses or change.
>
>
> On 03/26/2014 09:49 PM, Mike Hearn wrote:
>> Myself, Thomas V (Electrum) and Marek (Trezor) got together to make sure
>> our BIP32 wallet structures would be compatible - and I discovered that
>> only I was planning to use the default structure.
>>
>> Because I'm hopeful that we can get a lot of interoperability between
>> wallets with regards to importing 12-words paper wallets, we
>> brainstormed to find a structure acceptable to everyone and ended up with:
>>
>>    /m/cointype/reserved'/account'/change/n
>>
>> The extra levels require some explanation:
>>
>>    * cointype:  This is zero for Bitcoin. This is here to support two
>>      things, one is supporting alt coins based off the same root seed.
>>      Right now nobody seemed very bothered about alt coins but sometimes
>>      feature requests do come in for this. Arguably there is no need and
>>      alt coins could just use the same keys as Bitcoin, but it may help
>>      avoid confusion if they don't.
>>
>>      More usefully, cointype can distinguish between keys intended for
>>      things like multisig outputs, e.g. for watchdog services. This means
>>      if your wallet does not know about the extra protocol layers
>>      involved in this, it can still import the "raw" money and it will
>>      just ignore/not see the keys used in more complex transactions.
>>
>>    * reserved is for "other stuff". I actually don't recall why we ended
>>      up with this. It may have been intended to split out multisig
>>      outputs etc from cointype. Marek, Thomas?
>>
>>    * account is for keeping essentially wallets-within-a-wallet to avoid
>>      mixing of coins. If you want that.
>>
>>    * change is 0 for receiving addresses, 1 for change addresses.
>>
>>    * n is the actual key index
>>
>> For bitcoinj we're targeting a deliberately limited feature set for hdw
>> v1 so I would just set the first three values all to zero and that is a
>> perfectly fine way to be compatible.
>>
>> The goal here is that the same seed can be written down once, and meet
>> all the users needs, whilst still allowing some drift between what
>> wallets support.
>>
>> Pieter made the I think valid point that you can't really encode how
>> keys are meant to be used into just an HDW hierarchy and normally you'd
>> need some metadata as well. However, I feel interop between wallets is
>> more important than arriving at the most perfect possible arrangement,
>> which feels a little like bikeshedding, so I'm happy to just go with the
>> flow on this one.
>>
>>
>>
>> ------------------------------------------------------------------------------
>>
>>
>>
>> _______________________________________________
>> Bitcoin-development mailing list
>> Bitcoin-development@lists•sourceforge•net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
Thomas Voegtlin in reply to Mike Hearn permalink original

Le 26/03/2014 21:49, Mike Hearn a écrit :

Quoted text (5 lines)
>
>   * reserved is for "other stuff". I actually don't recall why we ended
>     up with this. It may have been intended to split out multisig
>     outputs etc from cointype. Marek, Thomas?
>

yes, this was intended to create multisig addresses from the same seed. cointype was proposed after that.

Pavol Rusnak in reply to Tamas Blummer permalink original

On 03/27/2014 08:09 AM, Tamas Blummer wrote:

Quoted text (2 lines)
> A notable suggestion was to instead of building a directory of magic numbers (like 0 for Bitcoin, 1 for Litecoin etc) use a hash of the word "Bitcoin", "Litecoin", "Dogecoin", so collosion is unlikely and
> cetral directory is not needed.

Nice idea, but keep in mind that you are hashing into 2^32 space, so collisions will occur, unfortunately and we'll end up with directory again :-/

Even if they did not occur you would need to keep up the registry of names anyway (is it Peercoin or PPCoin, Testnet or TestNet ...)?

Pavol Rusnak in reply to Mike Hearn permalink original

On 03/26/2014 09:49 PM, Mike Hearn wrote:

Quoted text (3 lines)
>    - cointype:  This is zero for Bitcoin. This is here to support two
>    things, one is supporting alt coins based off the same root seed. Right now
>    nobody seemed very bothered about alt coins but sometimes feature requests

There is one "altcoin" that is pretty important even today and it is Testnet.

Mike Hearn in reply to Thomas Voegtlin permalink original
Quoted text (6 lines)
>
> One issue that I have is bandwidth: Electrum (and mycelium) cannot
> watch as many addresses as they want, because this will create too
> much traffic on the servers. (especially when servers send utxo merkle
> proofs for each address, which is not the case yet, but is planned)
>

This is surprising and the first time I've heard about this. Surely your constraint is CPU or disk seeks? Addresses are small, I find it hard to believe that clients uploading them is a big drain, and mostly addresses that are in the lookahead region won't have any hits and so won't result in any downloads?

This constraint is not so important for bloom-filter clients.

Bloom filters are a neat way to encode addresses and keys but they don't magically let clients save bandwidth. A smaller filter results in less upload bandwidth but more download (from the wallets perspective). So I'm worried if you think this will be an issue for your clients: I haven't investigated bandwidth usage deeply yet, perhaps I should.

FWIW the current bitcoinj HDW alpha preview pre-gens 100 addresses on both receive and change branches. But I'm not sure what the right setting is.

We also have to consider latency. The simplest implementation from a wallets POV is to step through each transaction in the block chain one at a time, and each time you see an address that is yours, calculate the next ones in the chain. But that would be fantastically slow, so we must instead pre-generate a larger lookahead region and request more data in one batch. Then you have to recover if that batch ends up using all the pre-genned addresses. It's just painful.

Quoted text (6 lines)
> My opinion, as far as Electrum is concerned, is that merchant accounts
> should behave differently from regular user accounts: While merchants
> need to generate an unlimited number of receiving addresses, it is also
> acceptable for them to have a slightly more complex wallet recovery
> procedure
>

Maybe. I dislike any distinction between users and merchants though. I don't think it's really safe to assume merchants are more sophisticated than end users.

Quoted text (4 lines)
> but also because we want fully automated synchronization between different
> instances of a wallet, using only no other source of information than
> the blockchain.
>

I think such synchronization won't be possible as we keep adding features, because the block chain cannot sync all the relevant data. For instance Electrum already has a label sync feature. Other wallets need to compete with that, somehow, so we need to build a way to do cross-device wallet sync with non-chain data.

Thomas Voegtlin in reply to Thomas Voegtlin permalink original

Le 27/03/2014 12:30, Marek Palatinus a écrit :

Quoted text (7 lines)
> Ah, I forget to two things, which should be into the BIP as well:
>
> a) Gap factor for addresses; as Thomas mentioned, although some software
> can watch almost unlimited amount of unused addresses, this is serious
> concern for lightweight or server-based wallets like Electrum or
> myTREZOR. myTREZOR currently uses gap factor 10, which is (from my
> experience so far) quite sane for most of users.

Yes, I was planning to increase the number of available unused addresses to 10 or 20 in the bip32 version of Electrum.

Related to this, here is another idea I would like to submit:

Instead of using a "gap limit" (maximal number of consecutive unused addresses), I think we should get rid of the topology, and simply count the number of unused addresses since the beginning of the sequence. Indeed, the topology of the sequence of addresses is of no interest to the user. Users often misinterpret "gap limit" as the "number of unused addresses available", so I think we should just give them what they want :) This is easier to understand, and it makes things more predictable, because the wallet will always display the same number of unused addresses (except when it is waiting for confirmations).

Mike Hearn in reply to Thomas Voegtlin permalink original

By the way, I just noticed that greenaddress.it is creating seeds that have 24 words instead of 12. Does anyone know what's up with that? They claim to be using BIP32 wallets so I wanted to see if they were using the default structure and if so, whether bitcoinj was compatible with it (before I switch to the one discussed here). But it seems we fall at the first hurdle ...

On Thu, Mar 27, 2014 at 1:06 PM, Thomas Voegtlin <thomasv1@gmx•de> wrote:

Quoted text (35 lines)
>
>
> Le 27/03/2014 12:30, Marek Palatinus a écrit :
> > Ah, I forget to two things, which should be into the BIP as well:
> >
> > a) Gap factor for addresses; as Thomas mentioned, although some software
> > can watch almost unlimited amount of unused addresses, this is serious
> > concern for lightweight or server-based wallets like Electrum or
> > myTREZOR. myTREZOR currently uses gap factor 10, which is (from my
> > experience so far) quite sane for most of users.
>
>
> Yes, I was planning to increase the number of available unused addresses
> to 10 or 20 in the bip32 version of Electrum.
>
> Related to this, here is another idea I would like to submit:
>
> Instead of using a "gap limit" (maximal number of consecutive unused
> addresses), I think we should get rid of the topology, and simply count
> the number of unused addresses since the beginning of the sequence.
> Indeed, the topology of the sequence of addresses is of no interest to
> the user. Users often misinterpret "gap limit" as the "number of unused
> addresses available", so I think we should just give them what they want
> :) This is easier to understand, and it makes things more predictable,
> because the wallet will always display the same number of unused
> addresses (except when it is waiting for confirmations).
>
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
Mike Hearn in reply to Mike Hearn permalink original

Ah, BIP32 allows for a range of entropy sizes and it so happens that they picked 256 bits instead of 128 bits.

I'd have thought that there is a right answer for this. 2^128 should not be brute forceable, and longer sizes have a cost in terms of making the seeds harder to write down on paper. So should this be a degree of freedom?

On Thu, Mar 27, 2014 at 1:28 PM, Mike Hearn <mike@plan99•net> wrote:

Quoted text (47 lines)
> By the way, I just noticed that greenaddress.it is creating seeds that
> have 24 words instead of 12. Does anyone know what's up with that? They
> claim to be using BIP32 wallets so I wanted to see if they were using the
> default structure and if so, whether bitcoinj was compatible with it
> (before I switch to the one discussed here). But it seems we fall at the
> first hurdle ...
>
>
> On Thu, Mar 27, 2014 at 1:06 PM, Thomas Voegtlin <thomasv1@gmx•de> wrote:
>
>>
>>
>> Le 27/03/2014 12:30, Marek Palatinus a écrit :
>> > Ah, I forget to two things, which should be into the BIP as well:
>> >
>> > a) Gap factor for addresses; as Thomas mentioned, although some software
>> > can watch almost unlimited amount of unused addresses, this is serious
>> > concern for lightweight or server-based wallets like Electrum or
>> > myTREZOR. myTREZOR currently uses gap factor 10, which is (from my
>> > experience so far) quite sane for most of users.
>>
>>
>> Yes, I was planning to increase the number of available unused addresses
>> to 10 or 20 in the bip32 version of Electrum.
>>
>> Related to this, here is another idea I would like to submit:
>>
>> Instead of using a "gap limit" (maximal number of consecutive unused
>> addresses), I think we should get rid of the topology, and simply count
>> the number of unused addresses since the beginning of the sequence.
>> Indeed, the topology of the sequence of addresses is of no interest to
>> the user. Users often misinterpret "gap limit" as the "number of unused
>> addresses available", so I think we should just give them what they want
>> :) This is easier to understand, and it makes things more predictable,
>> because the wallet will always display the same number of unused
>> addresses (except when it is waiting for confirmations).
>>
>>
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> Bitcoin-development mailing list
>> Bitcoin-development@lists•sourceforge•net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>
>
>
Matias Alejo Garcia in reply to Mike Hearn permalink original

On Thu, Mar 27, 2014 at 9:28 AM, Mike Hearn <mike@plan99•net> wrote:

Quoted text (6 lines)
> By the way, I just noticed that greenaddress.it is creating seeds that have 24
> words instead of 12. Does anyone know what's up with that? They claim to be
> using BIP32 wallets so I wanted to see if they were using the default structure
> and if so, whether bitcoinj was compatible with it (before I switch to the one
> discussed here). But it seems we fall at the first hurdle ...
>

They replied this on reddit: "Good question! We use 256-bit BIP39 seeds for our 512-bit extended HD wallet keypairs, as advised by BIP32. We are aware that 128 is considered ok in Electrum but we wanted to follow BIP32 spec, in particular because we are working with hardware wallets that have been design with BIP32 in mind and wanted to avoid any incompatibility issue."

http://www.reddit.com/r/Bitcoin/comments/20puhg/while_blockchaininfo_is_down_what_about_testing/cg5mr54

BTW, thanks for starting this thread. I am with the bitcore team, working on a new multisig only wallet (Cosign).

The schema originally proposed be Mike (/m/cointype/reserved'/account'/change/n ) seems flexible enough for our needs. I wonder, tough, if we can agree also on a format to exchange the tips (n) of each branch on the tree when the user export/backup the wallet, as Allen suggested.

On Wed, Mar 26, 2014 at 8:37 PM, Andreas Schildbach <andreas@schildbach•de> wrote:

Quoted text (3 lines)
> For me, the most important thing is either we're 100% interoperable or
> 0%. There should not be anything inbetween, as users will delete seeds
> without knowing there is still money in them on another implementation.

Is the ideal schema that the user should be able to use two different wallets at the same time? Does a wallet need to assume that derivate keys could had been used already? That seems complicated. Maybe we could add a /(hash of the wallet name)/ on the BIP32 path.

Thomas Kerin in reply to Mike Hearn permalink original

Isn't the length of the seed arbitrary anyway? Once decoded using whatever mnemonic implementation (electrums, or BIP0039) the bytestream is immediately passed to HMAC-SHA256 to generate the master key. No matter what your initial entropy is, it would be hashed anyway.

On Thu, Mar 27, 2014 at 12:49 PM, Mike Hearn <mike@plan99•net> wrote:

Quoted text (68 lines)
> Ah, BIP32 allows for a range of entropy sizes and it so happens that they
> picked 256 bits instead of 128 bits.
>
> I'd have thought that there is a right answer for this. 2^128 should not
> be brute forceable, and longer sizes have a cost in terms of making the
> seeds harder to write down on paper. So should this be a degree of freedom?
>
>
> On Thu, Mar 27, 2014 at 1:28 PM, Mike Hearn <mike@plan99•net> wrote:
>
>> By the way, I just noticed that greenaddress.it is creating seeds that
>> have 24 words instead of 12. Does anyone know what's up with that? They
>> claim to be using BIP32 wallets so I wanted to see if they were using the
>> default structure and if so, whether bitcoinj was compatible with it
>> (before I switch to the one discussed here). But it seems we fall at the
>> first hurdle ...
>>
>>
>> On Thu, Mar 27, 2014 at 1:06 PM, Thomas Voegtlin <thomasv1@gmx•de> wrote:
>>
>>>
>>>
>>> Le 27/03/2014 12:30, Marek Palatinus a écrit :
>>> > Ah, I forget to two things, which should be into the BIP as well:
>>> >
>>> > a) Gap factor for addresses; as Thomas mentioned, although some
>>> software
>>> > can watch almost unlimited amount of unused addresses, this is serious
>>> > concern for lightweight or server-based wallets like Electrum or
>>> > myTREZOR. myTREZOR currently uses gap factor 10, which is (from my
>>> > experience so far) quite sane for most of users.
>>>
>>>
>>> Yes, I was planning to increase the number of available unused addresses
>>> to 10 or 20 in the bip32 version of Electrum.
>>>
>>> Related to this, here is another idea I would like to submit:
>>>
>>> Instead of using a "gap limit" (maximal number of consecutive unused
>>> addresses), I think we should get rid of the topology, and simply count
>>> the number of unused addresses since the beginning of the sequence.
>>> Indeed, the topology of the sequence of addresses is of no interest to
>>> the user. Users often misinterpret "gap limit" as the "number of unused
>>> addresses available", so I think we should just give them what they want
>>> :) This is easier to understand, and it makes things more predictable,
>>> because the wallet will always display the same number of unused
>>> addresses (except when it is waiting for confirmations).
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> _______________________________________________
>>> Bitcoin-development mailing list
>>> Bitcoin-development@lists•sourceforge•net
>>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>>
>>
>>
>
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
Mike Hearn in reply to Thomas Kerin permalink original

Obviously, SHA256 can't magically generate more entropy out of nothing, it just stretches whatever is put in. If your seed was only 32 bits then hashing wouldn't save you: every possible private key could easily be calculated in advance.

On Thu, Mar 27, 2014 at 2:12 PM, Thomas Kerin <thomas.kerin@gmail•com>wrote:

Quoted text (77 lines)
> Isn't the length of the seed arbitrary anyway? Once decoded using whatever
> mnemonic implementation (electrums, or BIP0039) the bytestream is
> immediately passed to HMAC-SHA256 to generate the master key. No matter
> what your initial entropy is, it would be hashed anyway.
>
>
> On Thu, Mar 27, 2014 at 12:49 PM, Mike Hearn <mike@plan99•net> wrote:
>
>> Ah, BIP32 allows for a range of entropy sizes and it so happens that they
>> picked 256 bits instead of 128 bits.
>>
>> I'd have thought that there is a right answer for this. 2^128 should not
>> be brute forceable, and longer sizes have a cost in terms of making the
>> seeds harder to write down on paper. So should this be a degree of freedom?
>>
>>
>> On Thu, Mar 27, 2014 at 1:28 PM, Mike Hearn <mike@plan99•net> wrote:
>>
>>> By the way, I just noticed that greenaddress.it is creating seeds that
>>> have 24 words instead of 12. Does anyone know what's up with that? They
>>> claim to be using BIP32 wallets so I wanted to see if they were using the
>>> default structure and if so, whether bitcoinj was compatible with it
>>> (before I switch to the one discussed here). But it seems we fall at the
>>> first hurdle ...
>>>
>>>
>>> On Thu, Mar 27, 2014 at 1:06 PM, Thomas Voegtlin <thomasv1@gmx•de>wrote:
>>>
>>>>
>>>>
>>>> Le 27/03/2014 12:30, Marek Palatinus a écrit :
>>>> > Ah, I forget to two things, which should be into the BIP as well:
>>>> >
>>>> > a) Gap factor for addresses; as Thomas mentioned, although some
>>>> software
>>>> > can watch almost unlimited amount of unused addresses, this is serious
>>>> > concern for lightweight or server-based wallets like Electrum or
>>>> > myTREZOR. myTREZOR currently uses gap factor 10, which is (from my
>>>> > experience so far) quite sane for most of users.
>>>>
>>>>
>>>> Yes, I was planning to increase the number of available unused addresses
>>>> to 10 or 20 in the bip32 version of Electrum.
>>>>
>>>> Related to this, here is another idea I would like to submit:
>>>>
>>>> Instead of using a "gap limit" (maximal number of consecutive unused
>>>> addresses), I think we should get rid of the topology, and simply count
>>>> the number of unused addresses since the beginning of the sequence.
>>>> Indeed, the topology of the sequence of addresses is of no interest to
>>>> the user. Users often misinterpret "gap limit" as the "number of unused
>>>> addresses available", so I think we should just give them what they want
>>>> :) This is easier to understand, and it makes things more predictable,
>>>> because the wallet will always display the same number of unused
>>>> addresses (except when it is waiting for confirmations).
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> _______________________________________________
>>>> Bitcoin-development mailing list
>>>> Bitcoin-development@lists•sourceforge•net
>>>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>>>
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>>
>> _______________________________________________
>> Bitcoin-development mailing list
>> Bitcoin-development@lists•sourceforge•net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>
>>
>
Thomas Voegtlin in reply to Mike Hearn permalink original

Le 27/03/2014 12:39, Mike Hearn a écrit :

Quoted text (11 lines)
>     One issue that I have is bandwidth: Electrum (and mycelium) cannot
>     watch as many addresses as they want, because this will create too
>     much traffic on the servers. (especially when servers send utxo merkle
>     proofs for each address, which is not the case yet, but is planned)
>
>
> This is surprising and the first time I've heard about this. Surely your
> constraint is CPU or disk seeks? Addresses are small, I find it hard to
> believe that clients uploading them is a big drain, and mostly addresses
> that are in the lookahead region won't have any hits and so won't result
> in any downloads?

To be honest, I have not carried out a comprehensive examination of server performance. What I can see is that Electrum servers are often slowed down when a wallet with a large number (thousands) of addresses shows up, and this is caused by disk seeks (especially on my slow VPS).

The master branch of electrum-server is also quite wasteful in terms of CPU, because it uses client threads. I have another branch that uses a socket poller, but that branch is not widely deployed yet.

I reckon that I might have been a bit too conservative, in setting the number of unused receiving addresses watched by Electrum clients (until now, the default "gap limit" has always been 5). The reason is that, if I increase that number, then there is no way to go back to a smaller value, because it needs to be compatible with all previously released versions. However, Electrum servers performance has improved over time, so I guess it could safely be raised to 20 (see previous post to slush).

In terms of bandwidth, I am referring to my Android version of Electrum. When it runs on a 3G connection, it sometimes takes up to 1 minute to synchronize (with a wallet that has hundreds of addresses). However, I have not checked if this was caused by addresses or block headers.

Quoted text (13 lines)
>
>     This constraint is not so important for bloom-filter clients.
>
>
> Bloom filters are a neat way to encode addresses and keys but they don't
> magically let clients save bandwidth. A smaller filter results in less
> upload bandwidth but more download (from the wallets perspective). So
> I'm worried if you think this will be an issue for your clients: I
> haven't investigated bandwidth usage deeply yet, perhaps I should.
>
> FWIW the current bitcoinj HDW alpha preview pre-gens 100 addresses on
> both receive and change branches. But I'm not sure what the right
> setting is.

Heh, may I suggest 20 in the receive branch?

For the change branch, there is no need to watch a large number of unused addresses, because the wallet should try to fill all the gaps in the sequence of change.

(Electrum does that. It also watches 3 unused addresses at the end of that sequence, in order to cope with possible blockchain reorgs causing gaps. As an extra safety, it also waits for 3 confirmations before using a new change address, which sometimes results in address reuse, but I guess a smarter strategy could avoid that).

Quoted text (19 lines)
>
> We also have to consider latency. The simplest implementation from a
> wallets POV is to step through each transaction in the block chain one
> at a time, and each time you see an address that is yours, calculate the
> next ones in the chain. But that would be fantastically slow, so we must
> instead pre-generate a larger lookahead region and request more data in
> one batch. Then you have to recover if that batch ends up using all the
> pre-genned addresses. It's just painful.

>
>     My opinion, as far as Electrum is concerned, is that merchant accounts
>     should behave differently from regular user accounts: While merchants
>     need to generate an unlimited number of receiving addresses, it is also
>     acceptable for them to have a slightly more complex wallet recovery
>     procedure
>
>
> Maybe. I dislike any distinction between users and merchants though. I
> don't think it's really safe to assume merchants are more sophisticated
> than end users.

well, it depends what we mean by "merchant". I was thinking more of a website running a script, rather than a brick and mortar ice cream seller. :)

Quoted text (12 lines)
>
>     but also because we want fully automated synchronization between
>     different
>     instances of a wallet, using only no other source of information than
>     the blockchain.
>
>
> I think such synchronization won't be possible as we keep adding
> features, because the block chain cannot sync all the relevant data. For
> instance Electrum already has a label sync feature. Other wallets need
> to compete with that, somehow, so we need to build a way to do
> cross-device wallet sync with non-chain data.

Oh, I was not referring to label sync, but only to the synchronization of the list of addresses in the wallet. Label sync is an Electrum plugin that relies on a centralized server. Using a third party server is acceptable in that case, IMO, because you will not lose your coins if the server fails.

Mike Hearn in reply to Thomas Voegtlin permalink original
Quoted text (6 lines)
>
> To be honest, I have not carried out a comprehensive examination of
> server performance. What I can see is that Electrum servers are often
> slowed down when a wallet with a large number (thousands) of addresses
> shows up, and this is caused by disk seeks (especially on my slow VPS).
>

Yes that sounds more like what I expected.

Quoted text (5 lines)
> In terms of bandwidth, I am referring to my Android version of Electrum.
> When it runs on a 3G connection, it sometimes takes up to 1 minute to
> synchronize (with a wallet that has hundreds of addresses). However, I
> have not checked if this was caused by addresses or block headers.
>

An address is 160 bits. (1000 * 160) / 8 / 1024 = 19.5 kilobytes of data which 3G should be able to transfer in <1 second easily. Of course the encoding may not be optimal. But if it is, I suspect the issue is elsewhere.

Pavol Rusnak in reply to Thomas Voegtlin permalink original

On 03/27/2014 01:06 PM, Thomas Voegtlin wrote:

Quoted text (2 lines)
> Yes, I was planning to increase the number of available unused addresses
> to 10 or 20 in the bip32 version of Electrum.

Also, we'd need to specify a "gap limit" for accounts as well. In TREZOR we currently use 0, which means that the scan will stop as soon as we hit first account with no transaction history (meaning that its first X=10 addresses are unused).

Thomas Voegtlin in reply to Mike Hearn permalink original

Le 27/03/2014 13:49, Mike Hearn a écrit :

Quoted text (7 lines)
> Ah, BIP32 allows for a range of entropy sizes and it so happens that
> they picked 256 bits instead of 128 bits.
>
> I'd have thought that there is a right answer for this. 2^128 should not
> be brute forceable, and longer sizes have a cost in terms of making the
> seeds harder to write down on paper. So should this be a degree of freedom?
>

Here is what I understand:

2^128 iterations is not brute forcable today, and will not be for the foreseeable future.

An EC pubkey of length n can be forced in approximately 2^(n/2) iterations (see http://ecc-challenge.info/) Thus, Bitcoin pubkeys, which are 256 bits, would require 2^128 iterations. This is why unused addresses (160 bits hash) are better protected than already used ones.

However, people tend to believe that a public key of size n requires 2^n iterations. This belief might have been spread by this popular image: https://bitcointalk.org/index.php?topic=508880.msg5616146#msg5616146

Thomas Voegtlin in reply to Pavol Rusnak permalink original

Le 27/03/2014 14:44, Pavol Rusnak a écrit :

Quoted text (9 lines)
> On 03/27/2014 01:06 PM, Thomas Voegtlin wrote:
>> Yes, I was planning to increase the number of available unused addresses
>> to 10 or 20 in the bip32 version of Electrum.
>
> Also, we'd need to specify a "gap limit" for accounts as well. In TREZOR
> we currently use 0, which means that the scan will stop as soon as we
> hit first account with no transaction history (meaning that its first
> X=10 addresses are unused).
>

I agree with that. I was planning to do the same (no gap)

Note: Maybe we could just look at the first address of each account, instead of the first 10 ?

Jim in reply to Thomas Voegtlin permalink original

Good to hear the bip32 wallet structure is _so_ close to being standardised. For MultiBit HD, we have put in support for 12/18/24 words but have the UI 'suggest' to use 12. You can see this on the wallet creation wizard Gary recently blogged about: https://multibit.org/blog/2014/03/26/multibit-hd-welcome-wizard.html

There's a little combo for the seed length, with 12 as the default.

@Thomas. You mention gaps. We are creating new addresses on the UI in a panel marked 'Request' where the user also types in a QR code label and a note to themselves. This gets stored away as a first class 'PaymentRequest'. The UI 'suggests' that each address is used once. There will be some gaps (where the payment request is never paid) but we aren't bulk creating addresses. I am hoping this shouldn't cause Electrum a problem.

We are also storing a timestamp (the number of days since the genesis block) to help wallet restore but that is SPV specific.

On Thu, Mar 27, 2014, at 01:49 PM, Thomas Voegtlin wrote:

Quoted text (3 lines)
>
>
> Le 27/03/2014 13:49, Mike Hearn a écrit :

IP32 allows for a range of entropy sizes and it so happens that

Quoted text (28 lines)
> > they picked 256 bits instead of 128 bits.
> >
> > I'd have thought that there is a right answer for this. 2^128 should not
> > be brute forceable, and longer sizes have a cost in terms of making the
> > seeds harder to write down on paper. So should this be a degree of freedom?
> >
>
>
> Here is what I understand:
>
> 2^128 iterations is not brute forcable today, and will not be for the
> foreseeable future.
>
> An EC pubkey of length n can be forced in approximately 2^(n/2)
> iterations (see http://ecc-challenge.info/) Thus, Bitcoin pubkeys, which
> are 256 bits, would require 2^128 iterations. This is why unused
> addresses (160 bits hash) are better protected than already used ones.
>
> However, people tend to believe that a public key of size n requires 2^n
> iterations. This belief might have been spread by this popular image:
> https://bitcointalk.org/index.php?topic=508880.msg5616146#msg5616146
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Pavol Rusnak in reply to Thomas Voegtlin permalink original

On 03/27/2014 02:53 PM, Thomas Voegtlin wrote:

Quoted text (2 lines)
> Note: Maybe we could just look at the first address of each account,
> instead of the first 10 ?

This is a possible optimization, but it adds unnecessary logic. Also it does not decrease the number of required requests between a client and a server (e.g. when backend sends responses in "bulks" of 10 addresses or more).

Mike Hearn in reply to Jim permalink original

For SPV wallets it's more complicated. There must always be a large lookahead window for latency reasons. We can't query the entire database because we don't know how far ahead the user is. So we have to assume there might be a lot of transaction traffic and create a large window, to reduce the chances that we run out whilst syncing and have to abort/restart the sync after resetting the Bloom filter.

If you have a full db index then you can calculate some addresses, query, if they all get hits, calculate some more, requery, etc. It's a bit simpler.

On Thu, Mar 27, 2014 at 2:58 PM, Jim <jim618@fastmail•co•uk> wrote:

Quoted text (70 lines)
> Good to hear the bip32 wallet structure is _so_ close to being
> standardised.
> For MultiBit HD, we have put in support for 12/18/24 words but have the UI
> 'suggest' to use 12.
> You can see this on the wallet creation wizard Gary recently blogged about:
> https://multibit.org/blog/2014/03/26/multibit-hd-welcome-wizard.html
>
> There's a little combo for the seed length, with 12 as the default.
>
>
> @Thomas. You mention gaps. We are creating new addresses on the UI in a
> panel marked 'Request' where the user also types in a QR code label and a
> note to themselves. This gets stored away as a first class
> 'PaymentRequest'. The UI 'suggests' that each address is used once. There
> will be some gaps (where the payment request is never paid) but we aren't
> bulk creating addresses. I am hoping this shouldn't cause Electrum a
> problem.
>
> We are also storing a timestamp (the number of days since the genesis
> block) to help wallet restore but that is SPV specific.
>
>
> On Thu, Mar 27, 2014, at 01:49 PM, Thomas Voegtlin wrote:
> >
> >
> > Le 27/03/2014 13:49, Mike Hearn a écrit :
> IP32 allows for a range of entropy sizes and it so happens that
> > > they picked 256 bits instead of 128 bits.
> > >
> > > I'd have thought that there is a right answer for this. 2^128 should
> not
> > > be brute forceable, and longer sizes have a cost in terms of making the
> > > seeds harder to write down on paper. So should this be a degree of
> freedom?
> > >
> >
> >
> > Here is what I understand:
> >
> > 2^128 iterations is not brute forcable today, and will not be for the
> > foreseeable future.
> >
> > An EC pubkey of length n can be forced in approximately 2^(n/2)
> > iterations (see http://ecc-challenge.info/) Thus, Bitcoin pubkeys, which
> > are 256 bits, would require 2^128 iterations. This is why unused
> > addresses (160 bits hash) are better protected than already used ones.
> >
> > However, people tend to believe that a public key of size n requires 2^n
> > iterations. This belief might have been spread by this popular image:
> > https://bitcointalk.org/index.php?topic=508880.msg5616146#msg5616146
> >
> >
> >
> ------------------------------------------------------------------------------
> > _______________________________________________
> > Bitcoin-development mailing list
> > Bitcoin-development@lists•sourceforge•net
> > https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
> --
> http://bitcoin-solutions.co.uk
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
Thomas Voegtlin in reply to Thomas Voegtlin permalink original
Quoted text (19 lines)
>
>     Related to this, here is another idea I would like to submit:
>
>     Instead of using a "gap limit" (maximal number of consecutive unused
>     addresses), I think we should get rid of the topology, and simply count
>     the number of unused addresses since the beginning of the sequence.
>     Indeed, the topology of the sequence of addresses is of no interest to
>     the user. Users often misinterpret "gap limit" as the "number of unused
>     addresses available", so I think we should just give them what they want
>     :) This is easier to understand, and it makes things more predictable,
>     because the wallet will always display the same number of unused
>     addresses (except when it is waiting for confirmations).
>
>
> I'm not convinced that it is either simpler to implement nor simpler to
> understand. I considering every "account" as completely independent
> structure (except that it has been derived from master seed). Because of
> this, I don't see a reason why number of generated addresses in one
> account should affect available addresses of other account.

Sorry if I was not clear, but no, I was not proposing that... I propose to count the number of unused addresses in each account, independently.

Here is an explanation with pictures:

Currently, Electrum does this:

x = used address o = unused address

x x x o o x x x x o o o o o
       |           └───────┘ 5 unused addresses at the end (gap limit =5)
      existing gap, size 2

Case 1: Suppose the user decides to use the 4th address starting from the end: x x x o o x x x x x o o o o

After 3 confirmations, Electrum will create one extra address:
x x x o o x x x x x o o o o o
                     └───────┘ 5 final address (one new)

Case 2: The user, who understands nothing about topology, decides to use the final address: x x x o o x x x x o o o o x

After 3 confirmations, Electrum will create 5 new addresses:
x x x o o x x x x o o o o x o o o o o
                     |        └───────┘ 5 new addresses
                  new gap (size 4)

You see that this leads to a number of newly created addresses that depends on the address picked by the user. This is the behaviour that I want to fix. I think it is bad for two reasons: First, because it is unpredictable (I had a user who set his gap limit to 100, and then was systematically using the last address of the list, ending up with a lot of wasted addresses). Second, because the worst case scenario (case 2) creates large gaps, which are less robust to blockchain reorgs than in case 1. A reorg in case 1, where the last tx is cancelled will create no gap. A reorg in case 2 can result in a gap that is larger than the limit (5), if another transaction uses the final addresses. (this is why Electrum has to wait for a few confirmations before adding addresses)

So, what I propose is to always have a constant number of unused addresses in each account, instead of something that depends on the topology. (this number can be 10 or 20, I don't mind, but I guess Mike might prefer 20...)

Troy Benjegerdes in reply to Thomas Voegtlin permalink original

On Thu, Mar 27, 2014 at 02:49:32PM +0100, Thomas Voegtlin wrote:

Quoted text (16 lines)
>
>
> Le 27/03/2014 13:49, Mike Hearn a écrit :
> > Ah, BIP32 allows for a range of entropy sizes and it so happens that
> > they picked 256 bits instead of 128 bits.
> >
> > I'd have thought that there is a right answer for this. 2^128 should not
> > be brute forceable, and longer sizes have a cost in terms of making the
> > seeds harder to write down on paper. So should this be a degree of freedom?
> >
>
>
> Here is what I understand:
>
> 2^128 iterations is not brute forcable today, and will not be for the
> foreseeable future.

I foresee 2^128 being brute forceable in 20-25 years. See below.

Quoted text (8 lines)
> An EC pubkey of length n can be forced in approximately 2^(n/2)
> iterations (see http://ecc-challenge.info/) Thus, Bitcoin pubkeys, which
> are 256 bits, would require 2^128 iterations. This is why unused
> addresses (160 bits hash) are better protected than already used ones.
>
> However, people tend to believe that a public key of size n requires 2^n
> iterations. This belief might have been spread by this popular image:
> https://bitcointalk.org/index.php?topic=508880.msg5616146#msg5616146

So I assume this image is using the Landauer principle for minimum energy ( http://en.wikipedia.org/wiki/Landauer%27s_principle ), however it is unknown (to me at least) if a reversible computing ecdsa forcing algorithm could be implemented. (this may or may not be a quantum computing device)

Let's suppose for a moment you could, and get a million times better than the Landauer pinciple limit of 2.85 zJ per bit, so we have total energy to cycle through 128 bits of address space of:

units "2**128 * 2.85zJ / 1e6" "megawatt*hours"
        * 269.39021

An attacker with a sub-Landauer limit/1e6 cracker would need a lot of silicon area, and a couple of hours energy from a large wind farm, and could siphon that energy out in a rural area without anyone noticing anything other than a few shipping containers and that the wind turbines are running more on windy days.

If we go back to just Landauer limit, and assume a 10MW system that runs 24x7 (much like the NCSA Blue Waters Cray machine), we need: (please check my math, or point out any stupid assumptions I've made)

units "2**128 * 2.85zJ / 10 megawatts" " years"
        * 3073.1914
Or 3000 years. Well that still sounds pretty safe. How about 250MW?
units "2**128 * 2.85zJ / 250 megawatts" " years"
        * 122.92766

Now I'm starting to get worried, because when I started computing, it was on an 8-bit CPU that was measured in thousand operations-per-second. In 1996 the largest supercomputer in the world was ASCII Red, with an amazing 1 trillion floating-point operations per second. This morning I have a 1-2 Teraflop water-cooled graphics processor sitting next to me warming the room.

I expect in 5-10 years we'll have silicon with 256 bit registers that may be able to do thousands or millions of ECDSA calculations per second per computation unit.

So if you stop hearing from me here, it's because I found a better mailing list, or a got a contract to develop and ECDSA cracker, in which case you probably won't hear from me again until I have a talk at DEFCON showing it off.

-- Troy Benjegerdes

Jeff Garzik in reply to Tamas Blummer permalink original

On Thu, Mar 27, 2014 at 3:09 AM, Tamas Blummer <tamas@bitsofproof•com> wrote:

Quoted text (4 lines)
> A notable suggestion was to instead of building a directory of magic numbers
> (like 0 for Bitcoin, 1 for Litecoin etc) use a hash of the word "Bitcoin",
> "Litecoin", "Dogecoin", so collosion is unlikely and
> cetral directory is not needed.

+1 good idea

Allen Piscitello in reply to Jeff Garzik permalink original

Don't most of these coins have a magic number already assigned that is unique? (0xD9B4BEF9 for Bitcoin, 0x0709110B for Testnet, FBC0XB6DB for Litecoin, etc...). This seems like a good candidate for identifying coins, and also supports Testnet cases well. Maybe there are some alts without such a magic number that might prevent that?

-Allen

On Thu, Mar 27, 2014 at 10:43 AM, Jeff Garzik <jgarzik@bitpay•com> wrote:

Quoted text (23 lines)
> On Thu, Mar 27, 2014 at 3:09 AM, Tamas Blummer <tamas@bitsofproof•com>
> wrote:
> > A notable suggestion was to instead of building a directory of magic
> numbers
> > (like 0 for Bitcoin, 1 for Litecoin etc) use a hash of the word
> "Bitcoin",
> > "Litecoin", "Dogecoin", so collosion is unlikely and
> > cetral directory is not needed.
>
> +1 good idea
>
> --
> Jeff Garzik
> Bitcoin core developer and open source evangelist
> BitPay, Inc.      https://bitpay.com/
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
Pavol Rusnak in reply to Allen Piscitello permalink original

On 03/27/2014 04:57 PM, Allen Piscitello wrote:

Quoted text (5 lines)
> Don't most of these coins have a magic number already assigned that is
> unique? (0xD9B4BEF9 for Bitcoin, 0x0709110B for Testnet, FBC0XB6DB for
> Litecoin, etc...).  This seems like a good candidate for identifying coins,
> and also supports Testnet cases well.  Maybe there are some alts without
> such a magic number that might prevent that?

That magic number is something I find very unfortunate and superflous in BIP-32 design. Its only purpose is to distinguish BIP-32 trees for various altcoins, but it doesn't make sense at all once you start storing various altcoins in the same tree using the proposed /m/cointype/reserved'/account'/change/n scheme.

I would love to see that removed from BIP-32 and use always 0x0488B21E/0x0488ADE4 (xpub/xpriv), but that is for different discussion I guess.

Tamas Blummer in reply to Allen Piscitello permalink original

I think not all alts (will) have magic numbers, at least not those defined e.g. with colored coins on top of an other chain.

Also note that the index should have MSB cleared as it would otherwise indicate private derivation.

Regards,

Tamas Blummer http://bitsofproof.com

On 27.03.2014, at 16:57, Allen Piscitello <allen.piscitello@gmail•com> wrote:

Quoted text (25 lines)
> Don't most of these coins have a magic number already assigned that is unique? (0xD9B4BEF9 for Bitcoin, 0x0709110B for Testnet, FBC0XB6DB for Litecoin, etc...).  This seems like a good candidate for identifying coins, and also supports Testnet cases well.  Maybe there are some alts without such a magic number that might prevent that?
>
> -Allen
>
>
> On Thu, Mar 27, 2014 at 10:43 AM, Jeff Garzik <jgarzik@bitpay•com> wrote:
> On Thu, Mar 27, 2014 at 3:09 AM, Tamas Blummer <tamas@bitsofproof•com> wrote:
> > A notable suggestion was to instead of building a directory of magic numbers
> > (like 0 for Bitcoin, 1 for Litecoin etc) use a hash of the word "Bitcoin",
> > "Litecoin", "Dogecoin", so collosion is unlikely and
> > cetral directory is not needed.
>
> +1 good idea
>
> --
> Jeff Garzik
> Bitcoin core developer and open source evangelist
> BitPay, Inc.      https://bitpay.com/
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
Allen Piscitello in reply to Pavol Rusnak permalink original

The idea was to use the magic number as the source for cointype. If it's too big, as Tamas showed, perhaps a hash of it, and for coins without a magic number, a hash of their name (or some unique identifier).

That being said, I agree with Andreas that something that is 90% inter-operable seems very dangerous and will give people false expectations when they miss the corner cases. If the structure isn't going to be shared completely and have all support shared, having it completely incompatible along with a mechanism for converting part of it to another wallet seems superior. The worst types of losses will occur when someone tests out something with a limited use case, sees that it appears to work, makes dangerous assumptions, then gets burned when it's too late.

-Allen

On Thu, Mar 27, 2014 at 11:06 AM, Pavol Rusnak <stick@gk2•sk> wrote:

Quoted text (30 lines)
> On 03/27/2014 04:57 PM, Allen Piscitello wrote:
> > Don't most of these coins have a magic number already assigned that is
> > unique? (0xD9B4BEF9 for Bitcoin, 0x0709110B for Testnet, FBC0XB6DB for
> > Litecoin, etc...).  This seems like a good candidate for identifying
> coins,
> > and also supports Testnet cases well.  Maybe there are some alts without
> > such a magic number that might prevent that?
>
> That magic number is something I find very unfortunate and superflous in
> BIP-32 design. Its only purpose is to distinguish BIP-32 trees for
> various altcoins, but it doesn't make sense at all once you start
> storing various altcoins in the same tree using the proposed
> /m/cointype/reserved'/account'/change/n scheme.
>
> I would love to see that removed from BIP-32 and use always
> 0x0488B21E/0x0488ADE4 (xpub/xpriv), but that is for different discussion
> I guess.
>
> --
> Best Regards / S pozdravom,
>
> Pavol Rusnak <stick@gk2•sk>
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
Pieter Wuille in reply to Allen Piscitello permalink original

Just chiming in...

I'm not opposed to a more generic default key tree, but we need to standardize this soon I believe. There are already existing code bases that implement BIP32 wallets (and more are popping up...); just using a separate one will result in lots of incompibilities.

That said, I'm not convinced about the extra layers. The "cointype" in my opinion isn't necessary inside the derivation. There is already support (4 bytes!) for magic bytes in the serialized form. Inside applications/p2p it should always be known to which chain it applies, and outside of that you shouldn't transfer raw keys. Maybe seeds need some marker, but that's a separate case anyway. Mainnet and testnet have specified magics here already - alts can define what they want imho.

A 'reserved' field for future extensions may be useful, but as already suggested by Mike, I don't believe we can encode how key chains are to be used inside the derivation structure anyway. The most basic case (not losing money in a wallet without special structure) can perhaps be supported with just "the blockchain is your wallet", but I don't believe this principle can scale to more advanced uses anyway, and you need metadata in the wallet to deal with it.

In my view, your wallet just has a bunch of chains, and each chain gets used for a particular purpose, fixing how the derivation beneath it works. Either that is as a wallet, as part of a pair of multisig keys, as a recurring payment receiver, ... or more complex things. Some of these will require extra layers beneath, but that is application specific. You would import a chain into your (advanced) wallet with a particular extpub/extpriv code, and some metadata on how to use it. Serialization formats for such designated extra uses sounds better to me than trying to fit it into the derivation structure.

Pavol Rusnak in reply to Pieter Wuille permalink original

On 03/27/2014 05:14 PM, Pieter Wuille wrote:

Quoted text (3 lines)
> That said, I'm not convinced about the extra layers. The "cointype" in
> my opinion isn't necessary inside the derivation. There is already
> support (4 bytes!) for magic bytes in the serialized form. Inside

Cointype in path is for separation purposes, not for identification.

Pieter Wuille in reply to Pavol Rusnak permalink original

On Thu, Mar 27, 2014 at 5:21 PM, Pavol Rusnak <stick@gk2•sk> wrote:

Quoted text (1 line)
> Cointype in path is for separation purposes, not for identification.

I don't understand what that gains you.

Allen Piscitello in reply to Pieter Wuille permalink original

The benefit I see is avoiding reuse of keys between coins while not having each wallet implementation have to know about each coin in order to scan for transactions. Wallet X supports Doge and Bitcoin. If both used a shared sequence of keys, say the first two end up Bitcoin, then 10 Doge, then some more Bitcoin. If you took this seed to Wallet Y, which only supports Bitcoin (either the wallet's support or what is installed on the system it's being used), it will see a gap of 10 addresses, and presume no more scanning with a 5 gap limit. The alternative is to reuse keys for each coin.

It also seems like a solution might be to only expect interoperability on a single sequence, and provide backups of each final sequence to use between different wallet implementations. This allows flexibility in hierarchies depending on needs and support of wallet, but allows sharing. The short seed would only be useful for the same wallet, but sharing between wallets would use the longer keys. That will give predictable behavior for users (although less friendly) and lead to less errors.

-Allen

On Thu, Mar 27, 2014 at 11:28 AM, Pieter Wuille <pieter.wuille@gmail•com>wrote:

Quoted text (15 lines)
> On Thu, Mar 27, 2014 at 5:21 PM, Pavol Rusnak <stick@gk2•sk> wrote:
> > Cointype in path is for separation purposes, not for identification.
>
> I don't understand what that gains you.
>
> --
> Pieter
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
slush in reply to Pieter Wuille permalink original

I agree that 'version' field of bip32 is not necessary and xpriv/xpub should be enough for all cases; there's actually no need to use different BIP32 roots for different altcoins.

I'm happily using one xpub for Bitcoin/Testnet/Litecoin at once, and by having the "cointype" distinction in the bip32 path itself, I'm sure that I don't reuse the same pubkey across blockchains which may be a privacy issue otherwise.

Marek

On Thu, Mar 27, 2014 at 5:28 PM, Pieter Wuille <pieter.wuille@gmail•com>wrote:

Quoted text (15 lines)
> On Thu, Mar 27, 2014 at 5:21 PM, Pavol Rusnak <stick@gk2•sk> wrote:
> > Cointype in path is for separation purposes, not for identification.
>
> I don't understand what that gains you.
>
> --
> Pieter
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
slush in reply to slush permalink original

After some off-list discussion about details with wallet developers, it seems that structure

m/<cointype>'/<account>'/<change>/<n>

fulfill requirements of all wallet developers around, including myTrezor, Electrum, Multibit, Wallet32 and other software is willing to adapt once anything will be standardized (i.e. they don't care).

Because I think that everybody told their comments to the topic already and because it seems that there's quite wide agreement on that, I would like to close the discussion and finally implement these paths into our software.

Cheers, Marek

On Fri, Mar 28, 2014 at 3:59 PM, slush <slush@centrum•cz> wrote:

Quoted text (31 lines)
> I agree that 'version' field of bip32 is not necessary and xpriv/xpub
> should be enough for all cases; there's actually no need to use different
> BIP32 roots for different altcoins.
>
> I'm happily using one xpub for Bitcoin/Testnet/Litecoin at once, and by
> having the "cointype" distinction in the bip32 path itself, I'm sure that I
> don't reuse the same pubkey across blockchains which may be a privacy issue
> otherwise.
>
> Marek
>
>
> On Thu, Mar 27, 2014 at 5:28 PM, Pieter Wuille <pieter.wuille@gmail•com>wrote:
>
>> On Thu, Mar 27, 2014 at 5:21 PM, Pavol Rusnak <stick@gk2•sk> wrote:
>> > Cointype in path is for separation purposes, not for identification.
>>
>> I don't understand what that gains you.
>>
>> --
>> Pieter
>>
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> Bitcoin-development mailing list
>> Bitcoin-development@lists•sourceforge•net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>
>
>
Pieter Wuille in reply to slush permalink original

I still don't understand the purpose of cointype. If you don't want to risk reusing the same keys across different currencies, just don't use the same seed or the same account? That is purely a client-side issue.

If the consensus is to add the cointype anyway, can we fix it to be equal to the 4-byte magic in the serialization (after setting the high bit to true)? That way there aren't two 4-byte magic codes that need to be defined for each, and at the same time make it obvious from the serialized form what it is for.

On Tue, Apr 8, 2014 at 2:43 PM, slush <slush@centrum•cz> wrote:

Quoted text (63 lines)
> After some off-list discussion about details with wallet developers, it
> seems that structure
>
> m/<cointype>'/<account>'/<change>/<n>
>
> fulfill requirements of all wallet developers around, including myTrezor,
> Electrum, Multibit, Wallet32 and other software is willing to adapt once
> anything will be standardized (i.e. they don't care).
>
> Because I think that everybody told their comments to the topic already and
> because it seems that there's quite wide agreement on that, I would like to
> close the discussion and finally implement these paths into our software.
>
> Cheers,
> Marek
>
>
> On Fri, Mar 28, 2014 at 3:59 PM, slush <slush@centrum•cz> wrote:
>>
>> I agree that 'version' field of bip32 is not necessary and xpriv/xpub
>> should be enough for all cases; there's actually no need to use different
>> BIP32 roots for different altcoins.
>>
>> I'm happily using one xpub for Bitcoin/Testnet/Litecoin at once, and by
>> having the "cointype" distinction in the bip32 path itself, I'm sure that I
>> don't reuse the same pubkey across blockchains which may be a privacy issue
>> otherwise.
>>
>> Marek
>>
>>
>> On Thu, Mar 27, 2014 at 5:28 PM, Pieter Wuille <pieter.wuille@gmail•com>
>> wrote:
>>>
>>> On Thu, Mar 27, 2014 at 5:21 PM, Pavol Rusnak <stick@gk2•sk> wrote:
>>> > Cointype in path is for separation purposes, not for identification.
>>>
>>> I don't understand what that gains you.
>>>
>>> --
>>> Pieter
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> _______________________________________________
>>> Bitcoin-development mailing list
>>> Bitcoin-development@lists•sourceforge•net
>>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>
>>
>
>
> ------------------------------------------------------------------------------
> Put Bad Developers to Shame
> Dominate Development with Jenkins Continuous Integration
> Continuously Automate Build, Test & Deployment
> Start a new project now. Try Jenkins in the cloud.
> http://p.sf.net/sfu/13600_Cloudbees
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
slush in reply to Pieter Wuille permalink original

On Tue, Apr 8, 2014 at 3:18 PM, Pieter Wuille <pieter.wuille@gmail•com>wrote:

Quoted text (5 lines)
> I still don't understand the purpose of cointype. If you don't want to
> risk reusing the same keys across different currencies, just don't use
> the same seed or the same account? That is purely a client-side issue.
>
>

Of course it is purely client-side issue, but it matters.

There's actually no reason to generate, backup and store individual seeds for various coins and purposes. User can handle all his identities and altcoins with single seed, avoiding potential issues with using wrong seed for other purposes.

Actually with accounts and cointypes in the path, you can have all your crypto funds stored on single seed, which I see as very comfortable solution.

But to gain advantages of such solution and avoid reusing the same path across blockchains, we need to separate the space, which is achieved by cointype.

Quoted text (7 lines)
> If the consensus is to add the cointype anyway, can we fix it to be
> equal to the 4-byte magic in the serialization (after setting the high
> bit to true)? That way there aren't two 4-byte magic codes that need
> to be defined for each, and at the same time make it obvious from the
> serialized form what it is for.
>
>

Serialization magic of bip32 seed is in my opinion completely unnecessary. Most of software does not care about it anyway; You can use xprv/xpub pair for main net, testnet, litecoin, dogecoin, whatevercoin.

Instead using the same seed (xprv) and then separate the chains *inside* the bip32 path seems more useful to me.

Marek

slush in reply to slush permalink original

tl;dr;

It is dangerous to expect that other seed than "xprv" does not contain bitcoins or that "xprv" contains only bitcoins, because technically are both situations possible. It is still safer to do the lookup; the magic itself is ambiguous.

Marek

On Tue, Apr 8, 2014 at 3:40 PM, slush <slush@centrum•cz> wrote:

Quoted text (10 lines)
>
> Serialization magic of bip32 seed is in my opinion completely unnecessary.
> Most of software does not care about it anyway; You can use xprv/xpub pair
> for main net, testnet, litecoin, dogecoin, whatevercoin.
>
> Instead using the same seed (xprv) and then separate the chains *inside*
> the bip32 path seems more useful to me.
>
> Marek
>
Pieter Wuille in reply to slush permalink original

I see the cause of our disagreement now.

You actually want to share a single BIP32 tree across different currency types, but do it in a way that guarantees that they never use the same keys.

I would have expected that different chains would use independent chains, and have serializations encode which chain they belong to.

Let me offer an alternative suggestion, which is compatible with the original default BIP32 structure: * You can use one seed across different chains, but the master nodes are separate. * To derive the master node from the seed, the key string "Bitcoin seed" is replaced by something chain-specific. * Every encoded node (including master nodes) has a chain-specific serialization magic.

This is in practice almost the same as your suggestion, except that the m/cointype' in m/cointype'/account'/change/n is replaced by different masters. The only disadvantage I see is that you do not have a way to encode the "super master" that is the parent of all chain-specific masters. You can - and with the same security properties - encode the seed, though.

Thomas Voegtlin in reply to Pieter Wuille permalink original

+1

I would prefer that solution...

Le 08/04/2014 15:53, Pieter Wuille a écrit :

Quoted text (25 lines)
> I see the cause of our disagreement now.
>
> You actually want to share a single BIP32 tree across different
> currency types, but do it in a way that guarantees that they never use
> the same keys.
>
> I would have expected that different chains would use independent
> chains, and have serializations encode which chain they belong to.
>
> Let me offer an alternative suggestion, which is compatible with the
> original default BIP32 structure:
> * You can use one seed across different chains, but the master nodes
> are separate.
> * To derive the master node from the seed, the key string "Bitcoin
> seed" is replaced by something chain-specific.
> * Every encoded node (including master nodes) has a chain-specific
> serialization magic.
>
> This is in practice almost the same as your suggestion, except that
> the m/cointype' in m/cointype'/account'/change/n is replaced by
> different masters. The only disadvantage I see is that you do not have
> a way to encode the "super master" that is the parent of all
> chain-specific masters. You can - and with the same security
> properties - encode the seed, though.
>
Tamas Blummer in reply to Pieter Wuille permalink original

Pieter,

your suggestion has charm since “Bitcoin seed” would even not need a global dictionary like the interpretation of the first level, since it would be self describing.

Regards,

Tamas Blummer http://bitsofproof.com

On 08.04.2014, at 15:53, Pieter Wuille <pieter.wuille@gmail•com> wrote:

Quoted text (64 lines)
> I see the cause of our disagreement now.
>
> You actually want to share a single BIP32 tree across different
> currency types, but do it in a way that guarantees that they never use
> the same keys.
>
> I would have expected that different chains would use independent
> chains, and have serializations encode which chain they belong to.
>
> Let me offer an alternative suggestion, which is compatible with the
> original default BIP32 structure:
> * You can use one seed across different chains, but the master nodes
> are separate.
> * To derive the master node from the seed, the key string "Bitcoin
> seed" is replaced by something chain-specific.
> * Every encoded node (including master nodes) has a chain-specific
> serialization magic.
>
> This is in practice almost the same as your suggestion, except that
> the m/cointype' in m/cointype'/account'/change/n is replaced by
> different masters. The only disadvantage I see is that you do not have
> a way to encode the "super master" that is the parent of all
> chain-specific masters. You can - and with the same security
> properties - encode the seed, though.
>
> --
> Pieter
>
>
> On Tue, Apr 8, 2014 at 3:43 PM, slush <slush@centrum•cz> wrote:
>> tl;dr;
>>
>> It is dangerous to expect that other seed than "xprv" does not contain
>> bitcoins or that "xprv" contains only bitcoins, because technically are both
>> situations possible. It is still safer to do the lookup; the magic itself is
>> ambiguous.
>>
>> Marek
>>
>> On Tue, Apr 8, 2014 at 3:40 PM, slush <slush@centrum•cz> wrote:
>>>
>>>
>>> Serialization magic of bip32 seed is in my opinion completely unnecessary.
>>> Most of software does not care about it anyway; You can use xprv/xpub pair
>>> for main net, testnet, litecoin, dogecoin, whatevercoin.
>>>
>>> Instead using the same seed (xprv) and then separate the chains *inside*
>>> the bip32 path seems more useful to me.
>>>
>>> Marek
>>
>>
>
> ------------------------------------------------------------------------------
> Put Bad Developers to Shame
> Dominate Development with Jenkins Continuous Integration
> Continuously Automate Build, Test & Deployment
> Start a new project now. Try Jenkins in the cloud.
> http://p.sf.net/sfu/13600_Cloudbees
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
Pavol Rusnak in reply to Pieter Wuille permalink original

On 04/08/2014 03:53 PM, Pieter Wuille wrote:

Quoted text (8 lines)
> Let me offer an alternative suggestion, which is compatible with the
> original default BIP32 structure:
> * You can use one seed across different chains, but the master nodes
> are separate.
> * To derive the master node from the seed, the key string "Bitcoin
> seed" is replaced by something chain-specific.
> * Every encoded node (including master nodes) has a chain-specific
> serialization magic.

This is possible, but I find it much more practical to use just one list (assignment of coins to indexes) than to use two lists (assignment of coins to key strings and to serialization magic).

Keeping two lists is harder and adds unnecessary friction. (Also I am not very happy for the possibility we'll have to deal with key strings "sCAMCo1N RULEZZZZ!!!! bRoUghT TO YoU bY M4rty" and serialization magic that leads to prefix "lulz").

Also from wallet's implementer perspective it is a little easier to use just one root node and then descend in tree as needed than to use method you described.

Andreas Schildbach in reply to slush permalink original

On 04/08/2014 02:43 PM, slush wrote:

Quoted text (13 lines)
> After some off-list discussion about details with wallet developers, it
> seems that structure
>
> m/<cointype>'/<account>'/<change>/<n>
>
> fulfill requirements of all wallet developers around, including
> myTrezor, Electrum, Multibit, Wallet32 and other software is willing to
> adapt once anything will be standardized (i.e. they don't care).
>
> Because I think that everybody told their comments to the topic already
> and because it seems that there's quite wide agreement on that, I would
> like to close the discussion and finally implement these paths into our
> software.

While there is an agreement that a standard would be useful for sharing wallets, we certainly didn't agree on every aspect of a standard. At least not on this thread, and also not at the Berlin meeting.

I understand each client will implement things a little bit different, for example the current plan is bitcoinj will hold all keys in memory and start reusing keys on low resources. Electrum uses a chain for their private purpose. Etc.

If we cannot 100% agree on a standard and also agree it will not be extended in future, I think we should not even try. It's dangerous for the money of users.

I propose we keep our chains deliberately separate and only try standardizing after each of us has a feel for the specific requirements.

slush in reply to Pieter Wuille permalink original

On Tue, Apr 8, 2014 at 3:53 PM, Pieter Wuille <pieter.wuille@gmail•com>wrote:

Quoted text (16 lines)
> I see the cause of our disagreement now.
>
> You actually want to share a single BIP32 tree across different
> currency types, but do it in a way that guarantees that they never use
> the same keys.
>
> I would have expected that different chains would use independent
> chains, and have serializations encode which chain they belong to.
>
> Let me offer an alternative suggestion, which is compatible with the
> original default BIP32 structure:
> * You can use one seed across different chains, but the master nodes
> are separate.
> * To derive the master node from the seed, the key string "Bitcoin
> seed" is replaced by something chain-specific.
>

I've discussed the solution of "Litecoin seed" in BIP32 HMAC with Litecoin devs already, and after long discussion we've concluded that it is generally bad idea.

When changing "Bitcoin seed" constant to something different, same *entropy* will produce different *master node*. That's actually the opposite what's requested, because xprv serialization format stores *node*, not *entropy*. By changing HMAC constant, you still won't be able to store one node and derive wallets for multiple coins at same time.

Quoted text (3 lines)
> * Every encoded node (including master nodes) has a chain-specific
> serialization magic.
>

This is in practice almost the same as your suggestion, except that

Quoted text (7 lines)
> the m/cointype' in m/cointype'/account'/change/n is replaced by
> different masters. The only disadvantage I see is that you do not have
> a way to encode the "super master" that is the parent of all
> chain-specific masters. You can - and with the same security
> properties - encode the seed, though.
>
>

Actually I don't understand why there's such disagreement about "cointype" level here, what it breaks? I see it as the cleanest solution so far. It is forward and backward compatible, does need any special extension to bip32 (to be strict, bip32 says "Bitcoin seed", so client using "Litecoin seed" cannot be "bip32 compatible").

Of course, the problem of "cointype" can be solved in zillion ways, but still using cointype in bip32 path seem to be the most elegant way so far, because it fullfill all requirements for single backup, for separating pubkeys and for handling all coins by one master...

Marek

slush in reply to Andreas Schildbach permalink original

On Tue, Apr 8, 2014 at 4:49 PM, Andreas Schildbach <andreas@schildbach•de>wrote:

Quoted text (5 lines)
> While there is an agreement that a standard would be useful for sharing
> wallets, we certainly didn't agree on every aspect of a standard. At
> least not on this thread, and also not at the Berlin meeting.
>
>

We're going to write down BIP describing such structure. If any wallet want to be BIP XX compatible, then it has chance to. Of course if any wallet want to use another format, then it cannot call itself BIP XX compatible, thus nobody will expect that such software will see/recover all keys generated by BIP XX wallet.

Quoted text (6 lines)
> I understand each client will implement things a little bit different,
> for example the current plan is bitcoinj will hold all keys in memory
> and start reusing keys on low resources. Electrum uses a chain for their
> private purpose. Etc.
>
>

It still doesn't mean that bitcoinj or Electrum cannot share the bare minimum of BIP XX. Of course if somebody will use Electrum for 2to3 transactions and then move wallet to client which does not offer such feature, it won't work. But I don't see that as a problem.

Quoted text (3 lines)
> If we cannot 100% agree on a standard and also agree it will not be
> extended in future, I think we should not even try. It's dangerous for
> the money of users.

If some developers agree on some specific BIP, then it should be cross compatible. Of course if somebody implements BIP XX in different way, then it isn't BIP XX compatible.

Quoted text (2 lines)
>
>

I propose we keep our chains deliberately separate and only try

Quoted text (2 lines)
> standardizing after each of us has a feel for the specific requirements.
>

Of course, if somebody don't want to generate compatible bip32 paths, no problem. It will be the same situation as now.

Marek

Andreas Schildbach in reply to slush permalink original

On 04/08/2014 05:46 PM, slush wrote:

Quoted text (9 lines)
>     I understand each client will implement things a little bit different,
>     for example the current plan is bitcoinj will hold all keys in memory
>     and start reusing keys on low resources. Electrum uses a chain for their
>     private purpose. Etc.
>
> It still doesn't mean that bitcoinj or Electrum cannot share the bare
> minimum of BIP XX. Of course if somebody will use Electrum for 2to3
> transactions and then move wallet to client which does not offer such
> feature, it won't work. But I don't see that as a problem.

There is no "bare minimum". Either you implement the "BIP" fully or not. There is no inbetween. Likewise, the standard cannot be extended unless you create a new standard that is based on the old (without re-using the path, of course).

We're lightyears away from a BIP. Lets first create implementations and see if they are compatible in all possible combinations and situations. The moment any two apps have a different view on their wallets generated from the same seed, they're incompatible. In this case they should either fix the issue or intentionally choose incompatible paths, so that they don't see and spend "subsets" of each other.

slush in reply to Andreas Schildbach permalink original

On Tue, Apr 8, 2014 at 5:58 PM, Andreas Schildbach <andreas@schildbach•de>wrote:

Quoted text (9 lines)
> On 04/08/2014 05:46 PM, slush wrote:
>
> > It still doesn't mean that bitcoinj or Electrum cannot share the bare
> > minimum of BIP XX. Of course if somebody will use Electrum for 2to3
> > transactions and then move wallet to client which does not offer such
> > feature, it won't work. But I don't see that as a problem.
>
> There is no "bare minimum". Either you implement the "BIP" fully or not.
>

I didn't mean "Bare minimum of some spec", but "spec describing bare minimum", which is different. I'm sorry for confusion.

Marek

Pieter Wuille in reply to slush permalink original

On Tue, Apr 8, 2014 at 5:41 PM, slush <slush@centrum•cz> wrote:

Quoted text (9 lines)
> I've discussed the solution of "Litecoin seed" in BIP32 HMAC with Litecoin
> devs already, and after long discussion we've concluded that it is generally
> bad idea.
>
> When changing "Bitcoin seed" constant to something different, same *entropy*
> will produce different *master node*. That's actually the opposite what's
> requested, because xprv serialization format stores *node*, not *entropy*.
> By changing HMAC constant, you still won't be able to store one node and
> derive wallets for multiple coins at same time.

Storing the seed is superior to storing the master node already (whether coin specific or not), as it is smaller.

All this changes is making the seed the "super master" which allows generating the coin-specific masters (which get an actual useful function: revealing your entire-tree, but only one coin's subset of it).

Quoted text (16 lines)
>> * Every encoded node (including master nodes) has a chain-specific
>> serialization magic.
>>
>> This is in practice almost the same as your suggestion, except that
>> the m/cointype' in m/cointype'/account'/change/n is replaced by
>> different masters. The only disadvantage I see is that you do not have
>> a way to encode the "super master" that is the parent of all
>> chain-specific masters. You can - and with the same security
>> properties - encode the seed, though.
>>
>
> Actually I don't understand why there's such disagreement about "cointype"
> level here, what it breaks? I see it as the cleanest solution so far. It is
> forward and backward compatible, does need any special extension to bip32
> (to be strict, bip32 says "Bitcoin seed", so client using "Litecoin seed"
> cannot be "bip32 compatible").

Fair enough, it would break strictly BIP32. Then again, BIP32 is a *Bitcoin* improvement proposal, and not something that necessarily applies to other coins (they can adopt it of course, I don't care).

What I dislike is that this removes the ability of using the magic in the serialization to prevent importing a chain from the wrong coin. The standard could just say that instead of "Bitcoin seed", you'd use "Coin seed: " + magic, so you don't need an extra mapping from cointype to seed strings.

slush in reply to Pieter Wuille permalink original

On Wed, Apr 23, 2014 at 7:42 PM, Pieter Wuille <pieter.wuille@gmail•com>wrote:

Quoted text (5 lines)
>
> Storing the seed is superior to storing the master node already
> (whether coin specific or not), as it is smaller.
>
>

...Except that you're loosing flexibility (serialization, deserialization) which gives you BIP32 node.

I see "bip32 seed" as some transitional, internal state from raw entropy to bip32 master node and this seed should not be handled by the end user in any form. In the oposite, well-serialized bip32 node (in xpriv, or even in mnemonic format) can be used very widely and have no downsides against using raw "bip32 seed".

Quoted text (6 lines)
>
> Fair enough, it would break strictly BIP32. Then again, BIP32 is a
> *Bitcoin* improvement proposal, and not something that necessarily
> applies to other coins (they can adopt it of course, I don't care).
>
>

I also don't care too much about altcoins, but people want them so me, as infrastructure developer, need to think about it. And I don't see any reason for breaking compatibility between Bitcoin and other altcoins. I would be happier if there will be another sentence than "Bitcoin seed", but honestly, who cares. It is just some magic string for hashing the raw seed...

Quoted text (3 lines)
> What I dislike is that this removes the ability of using the magic in
> the serialization to prevent importing a chain from the wrong coin.
>

The truth is that even existing software which handle bip32 don't care about 'version' at all. I think that "xpub/xprv" distinction is the only useful feature of version, so user se if it stores public or private information.

But using prefixes which doesn't enforce anything is even more dangerous. If somebody exports node "dogeblablabla", it creates false exceptations that there's only dogecoin stored.

Marek

slush in reply to slush permalink original

For those who don't follow github pull requests regularly; there's pull request for BIP64 defining HD wallet structure as discussed in this thread:

https://github.com/bitcoin/bips/pull/52

On Wed, Apr 23, 2014 at 8:01 PM, slush <slush@centrum•cz> wrote:

Quoted text (48 lines)
>
>
>
> On Wed, Apr 23, 2014 at 7:42 PM, Pieter Wuille <pieter.wuille@gmail•com>wrote:
>>
>> Storing the seed is superior to storing the master node already
>> (whether coin specific or not), as it is smaller.
>>
>>
> ...Except that you're loosing flexibility (serialization, deserialization)
> which gives you BIP32 node.
>
> I see "bip32 seed" as some transitional, internal state from raw entropy
> to bip32 master node and this seed should not be handled by the end user in
> any form. In the oposite, well-serialized bip32 node (in xpriv, or even in
> mnemonic format) can be used very widely and have no downsides against
> using raw "bip32 seed".
>
>
>>
>> Fair enough, it would break strictly BIP32. Then again, BIP32 is a
>> *Bitcoin* improvement proposal, and not something that necessarily
>> applies to other coins (they can adopt it of course, I don't care).
>>
>>
> I also don't care too much about altcoins, but people want them so me, as
> infrastructure developer, need to think about it. And I don't see any
> reason for breaking compatibility between Bitcoin and other altcoins. I
> would be happier if there will be another sentence than "Bitcoin seed", but
> honestly, who cares. It is just some magic string for hashing the raw
> seed...
>
>
>> What I dislike is that this removes the ability of using the magic in
>> the serialization to prevent importing a chain from the wrong coin.
>>
>
> The truth is that even existing software which handle bip32 don't care
> about 'version' at all. I think that "xpub/xprv" distinction is the only
> useful feature of version, so user se if it stores public or private
> information.
>
> But using prefixes which doesn't enforce anything is even more dangerous.
> If somebody exports node "dogeblablabla", it creates false exceptations
> that there's only dogecoin stored.
>
>  Marek
>
Tier Nolan in reply to slush permalink original

Different users could have different gap limit requirements. 20 seems very low as the default.

A merchant could easily send 20 addresses in a row to customers and none of them bother to actually buy anything.

Setting the gap limit to high is just a small extra cost in that case.

Bip-32 serialization doesn't have a way of adding meta data though.

On Wed, Apr 23, 2014 at 7:18 PM, slush <slush@centrum•cz> wrote:

Quoted text (72 lines)
> For those who don't follow github pull requests regularly; there's pull
> request for BIP64 defining HD wallet structure as discussed in this thread:
>
> https://github.com/bitcoin/bips/pull/52
>
>
>
> On Wed, Apr 23, 2014 at 8:01 PM, slush <slush@centrum•cz> wrote:
>
>>
>>
>>
>> On Wed, Apr 23, 2014 at 7:42 PM, Pieter Wuille <pieter.wuille@gmail•com>wrote:
>>>
>>> Storing the seed is superior to storing the master node already
>>> (whether coin specific or not), as it is smaller.
>>>
>>>
>> ...Except that you're loosing flexibility (serialization,
>> deserialization) which gives you BIP32 node.
>>
>> I see "bip32 seed" as some transitional, internal state from raw entropy
>> to bip32 master node and this seed should not be handled by the end user in
>> any form. In the oposite, well-serialized bip32 node (in xpriv, or even in
>> mnemonic format) can be used very widely and have no downsides against
>> using raw "bip32 seed".
>>
>>
>>>
>>> Fair enough, it would break strictly BIP32. Then again, BIP32 is a
>>> *Bitcoin* improvement proposal, and not something that necessarily
>>> applies to other coins (they can adopt it of course, I don't care).
>>>
>>>
>> I also don't care too much about altcoins, but people want them so me, as
>> infrastructure developer, need to think about it. And I don't see any
>> reason for breaking compatibility between Bitcoin and other altcoins. I
>> would be happier if there will be another sentence than "Bitcoin seed", but
>> honestly, who cares. It is just some magic string for hashing the raw
>> seed...
>>
>>
>>> What I dislike is that this removes the ability of using the magic in
>>> the serialization to prevent importing a chain from the wrong coin.
>>>
>>
>> The truth is that even existing software which handle bip32 don't care
>> about 'version' at all. I think that "xpub/xprv" distinction is the only
>> useful feature of version, so user se if it stores public or private
>> information.
>>
>> But using prefixes which doesn't enforce anything is even more dangerous.
>> If somebody exports node "dogeblablabla", it creates false exceptations
>> that there's only dogecoin stored.
>>
>>  Marek
>>
>
>
>
> ------------------------------------------------------------------------------
> Start Your Social Network Today - Download eXo Platform
> Build your Enterprise Intranet with eXo Platform Software
> Java Based Open Source Intranet - Social, Extensible, Cloud Ready
> Get Started Now And Turn Your Intranet Into A Collaboration Platform
> http://p.sf.net/sfu/ExoPlatform
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
Pavol Rusnak in reply to Tier Nolan permalink original

On 04/23/2014 08:39 PM, Tier Nolan wrote:

Quoted text (2 lines)
> A merchant could easily send 20 addresses in a row to customers and none of
> them bother to actually buy anything.

Such merchant would surely use some merchant system instead of generic wallet software.

Quoted text (1 line)
> Setting the gap limit to high is just a small extra cost in that case.

Not if you have 100 accounts on 10 different devices.

Tamas Blummer in reply to Tier Nolan permalink original

The most useful meta data to optimize chain scan is the key birth date, then the allowed gap size.

Tamas Blummer http://bitsofproof.com

On 23.04.2014, at 20:39, Tier Nolan <tier.nolan@gmail•com> wrote:

Quoted text (9 lines)
> Different users could have different gap limit requirements.  20 seems very low as the default.
>
> A merchant could easily send 20 addresses in a row to customers and none of them bother to actually buy anything.
>
> Setting the gap limit to high is just a small extra cost in that case.
>
> Bip-32 serialization doesn't have a way of adding meta data though.
>
>
Tier Nolan in reply to Pavol Rusnak permalink original

On Wed, Apr 23, 2014 at 7:46 PM, Pavol Rusnak <stick@gk2•sk> wrote:

Quoted text (5 lines)
>
> > Setting the gap limit to high is just a small extra cost in that case.
>
> Not if you have 100 accounts on 10 different devices.
>

I meant for a merchant with a server that is handing out hundreds of addresses.

The point is to have a single system that is compatible over a large number of systems.

slush in reply to Tier Nolan permalink original

Using higher gap limit in the software is not prohibited, but then it breaks the standard "as is", in mean that importing such wallet to another BIP64 wallet won't recover the funds properly, without proper settings of gap limit...

Gap limit 20 is the most sane defaults for majority of users (actually I think it is a bit higher than is really necessary for 99% users) and majority of users won't need to change it at any time.

Marek

On Wed, Apr 23, 2014 at 9:00 PM, Tier Nolan <tier.nolan@gmail•com> wrote:

Quoted text (27 lines)
> On Wed, Apr 23, 2014 at 7:46 PM, Pavol Rusnak <stick@gk2•sk> wrote:
>
>>
>> > Setting the gap limit to high is just a small extra cost in that case.
>>
>> Not if you have 100 accounts on 10 different devices.
>>
>
> I meant for a merchant with a server that is handing out hundreds of
> addresses.
>
> The point is to have a single system that is compatible over a large
> number of systems.
>
>
> ------------------------------------------------------------------------------
> Start Your Social Network Today - Download eXo Platform
> Build your Enterprise Intranet with eXo Platform Software
> Java Based Open Source Intranet - Social, Extensible, Cloud Ready
> Get Started Now And Turn Your Intranet Into A Collaboration Platform
> http://p.sf.net/sfu/ExoPlatform
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
Tamas Blummer in reply to Tier Nolan permalink original

I built such a merchant system handing out BIP32 addresses.

The gap size problem does not arise there since such a system has to have an extra database keeping track of requests, so there is no added cost of storing the key coordinates used by them. A scan is not needed the keys can be accessed at random order.

Tamas Blummer http://bitsofproof.com

On 23.04.2014, at 21:00, Tier Nolan <tier.nolan@gmail•com> wrote:

Quoted text (9 lines)
> On Wed, Apr 23, 2014 at 7:46 PM, Pavol Rusnak <stick@gk2•sk> wrote:
>
> > Setting the gap limit to high is just a small extra cost in that case.
>
> Not if you have 100 accounts on 10 different devices.
>
> I meant for a merchant with a server that is handing out hundreds of addresses.
>
> The point is to have a single system that is compatible over a large number of systems.
Pavol Rusnak in reply to Tier Nolan permalink original

On 04/23/2014 09:00 PM, Tier Nolan wrote:

Quoted text (2 lines)
> The point is to have a single system that is compatible over a large number
> of systems.

There is such system and it is called BIP32.

On the other hand, in BIP64 we try to put a set of restrictions and rules on top of BIP32. There will always be some special usecases where BIP64 is not a good fit and there's no reason why you cannot use BIP32 in a different manner using a different "purpose" field.

Examples: Electrum does not want to use accounts and they start to use scheme m/65'/change/address (where change = 0 or 1). Or Andreas Schildbach wants to have refunds chain so he uses m/66'/chain/address (where chain = 0, 1 or 2).

We wanted to find one good solution that fits all, but unfortunately it turned out everyone wants something a little bit different.

The point of the whole effort is that you can have ONE SINGLE BACKUP (master node) for all these independent purposes and suddenly claims such as "my wallet is BIP64 and BIP66 compatible" actually mean something as opposed to "BIP32 compatible" which actually means nothing except that the wallet author is capable of using HMAC in context of generating the tree.

Tamas Blummer in reply to slush permalink original

Pieter suggested in IRC couple of months ago to append key birth to key serialization in xprv….@unixtime format.

What about picking this idea up in BIP64? It would greatly help the importing client.

Regards,

Tamas Blummer http://bitsofproof.com

Luke Dashjr in reply to Pavol Rusnak permalink original

On Wednesday, April 23, 2014 7:29:04 PM Pavol Rusnak wrote:

Quoted text (18 lines)
> On 04/23/2014 09:00 PM, Tier Nolan wrote:
> > The point is to have a single system that is compatible over a large
> > number of systems.
>
> There is such system and it is called BIP32.
>
> On the other hand, in BIP64 we try to put a set of restrictions and
> rules on top of BIP32. There will always be some special usecases where
> BIP64 is not a good fit and there's no reason why you cannot use BIP32
> in a different manner using a different "purpose" field.
>
> Examples: Electrum does not want to use accounts and they start to use
> scheme m/65'/change/address (where change = 0 or 1). Or Andreas
> Schildbach wants to have refunds chain so he uses m/66'/chain/address
> (where chain = 0, 1 or 2).
>
> We wanted to find one good solution that fits all, but unfortunately it
> turned out everyone wants something a little bit different.

Why do clients need to use the features in BIP 64? If Electrum doesn't want to use accounts, then it can just use account 0 for everything. Refund chains are definitely a third case that should be added to the external and internal/change address division... and a wallet not implementing refund addresses would simply not use that chain.

Luke

slush in reply to Tamas Blummer permalink original

We do not want BIP64 to be incompatible with BIP32 in any way. BIP64 is just set of some recommendations for wallet developers how to browse bip32 tree.

Modifying serialization format would break the compatibility.

However we have our solution for storing wallet birth time, which is out of scope of BIP64, but we'll communicate it as soon as we'll write down that idea to some more specific format.

Marek

On Wed, Apr 23, 2014 at 9:36 PM, Tamas Blummer <tamas@bitsofproof•com>wrote:

Quoted text (13 lines)
> Pieter suggested in IRC couple of months ago to append key birth to key
> serialization in xprv....@unixtime format.
>
> What about picking this idea up in BIP64? It would greatly help the
> importing client.
>
> Regards,
>
> Tamas Blummer
> http://bitsofproof.com
>
>
>
Pavol Rusnak in reply to Luke Dashjr permalink original

On 04/23/2014 09:44 PM, Luke-Jr wrote:

Quoted text (2 lines)
> Why do clients need to use the features in BIP 64? If Electrum doesn't want to
> use accounts, then it can just use account 0 for everything. Refund chains are

As Andreas wrote earlier in this thread: "There is no "bare minimum". Either you implement the "BIP" fully or not."

What you suggest does not follow the principle of least surprise. Suppose user imports his BIP64 compatible wallet into Electrum, which claims it is BIP64 compatible, but actually implements just a subset of the spec (sticking account index to 0). The user now sees just a fraction of his coins and is puzzled.

Luke Dashjr in reply to Pavol Rusnak permalink original

On Wednesday, April 23, 2014 7:49:38 PM Pavol Rusnak wrote:

Quoted text (13 lines)
> On 04/23/2014 09:44 PM, Luke-Jr wrote:
> > Why do clients need to use the features in BIP 64? If Electrum doesn't
> > want to use accounts, then it can just use account 0 for everything.
> > Refund chains are
>
> As Andreas wrote earlier in this thread: "There is no "bare minimum".
> Either you implement the "BIP" fully or not."
>
> What you suggest does not follow the principle of least surprise.
> Suppose user imports his BIP64 compatible wallet into Electrum, which
> claims it is BIP64 compatible, but actually implements just a subset of
> the spec (sticking account index to 0). The user now sees just a
> fraction of his coins and is puzzled.

Any wallet should import all the coins just fine, it just wouldn't *use* any account other than 0. Remember addresses are used to receive bitcoins; once the UTXOs are in the wallet, they are no longer associated with the address or any other details of how they were received.

Luke

slush in reply to Luke Dashjr permalink original

On Wed, Apr 23, 2014 at 9:55 PM, Luke-Jr <luke@dashjr•org> wrote:

Quoted text (7 lines)
> Any wallet should import all the coins just fine, it just wouldn't *use*
> any
> account other than 0. Remember addresses are used to receive bitcoins; once
> the UTXOs are in the wallet, they are no longer associated with the
> address or
> any other details of how they were received.
>

Wallet don't see UTXO until it scans all branches/accounts on HD node import.

Luke Dashjr in reply to slush permalink original

On Wednesday, April 23, 2014 7:57:46 PM slush wrote:

Quoted text (10 lines)
> On Wed, Apr 23, 2014 at 9:55 PM, Luke-Jr <luke@dashjr•org> wrote:
> > Any wallet should import all the coins just fine, it just wouldn't *use*
> > any
> > account other than 0. Remember addresses are used to receive bitcoins;
> > once the UTXOs are in the wallet, they are no longer associated with the
> > address or
> > any other details of how they were received.
>
> Wallet don't see UTXO until it scans all branches/accounts on HD node
> import.

Yes, it should scan all possible (under the BIP-defined structure) branches regardless of which features it supports.

Luke

Tamas Blummer in reply to Luke Dashjr permalink original

On 23.04.2014, at 21:55, Luke-Jr <luke@dashjr•org> wrote:

Quoted text (4 lines)
> Any wallet should import all the coins just fine, it just wouldn't *use* any
> account other than 0. Remember addresses are used to receive bitcoins; once
> the UTXOs are in the wallet, they are no longer associated with the address or
> any other details of how they were received.

This is a bit idealistic. The wallet has to know how it got the UTXO in order to be able to spend it.

Luke Dashjr in reply to Tamas Blummer permalink original

On Wednesday, April 23, 2014 8:01:16 PM Tamas Blummer wrote:

Quoted text (9 lines)
> On 23.04.2014, at 21:55, Luke-Jr <luke@dashjr•org> wrote:
> > Any wallet should import all the coins just fine, it just wouldn't *use*
> > any account other than 0. Remember addresses are used to receive
> > bitcoins; once the UTXOs are in the wallet, they are no longer
> > associated with the address or any other details of how they were
> > received.
>
> This is a bit idealistic.
> The wallet has to know how it got the UTXO in order to be able to spend it.

No it doesn't... Just the assigned scriptPubKey and secret(s) required to satisfy it.

Pavol Rusnak in reply to Luke Dashjr permalink original

On 04/23/2014 10:01 PM, Luke-Jr wrote:

Quoted text (2 lines)
> Yes, it should scan all possible (under the BIP-defined structure) branches
> regardless of which features it supports.

So you suggest to scan for accounts, show balances but don't allow user to spend them? Does not seem right to me.

Tamas Blummer in reply to Luke Dashjr permalink original

On 23.04.2014, at 22:02, Luke-Jr <luke@dashjr•org> wrote:

Quoted text (14 lines)
> On Wednesday, April 23, 2014 8:01:16 PM Tamas Blummer wrote:
>> On 23.04.2014, at 21:55, Luke-Jr <luke@dashjr•org> wrote:
>>> Any wallet should import all the coins just fine, it just wouldn't *use*
>>> any account other than 0. Remember addresses are used to receive
>>> bitcoins; once the UTXOs are in the wallet, they are no longer
>>> associated with the address or any other details of how they were
>>> received.
>>
>> This is a bit idealistic.
>> The wallet has to know how it got the UTXO in order to be able to spend it.
>
> No it doesn't... Just the assigned scriptPubKey and secret(s) required to
> satisfy it.
>

To know the secret it needs to know which key coordinate to use that is logically the same as knowing the address it used to receive it.

Pieter Wuille in reply to Luke Dashjr permalink original

That's the point. BIP64 specifies such a structure, and you have to scan all that it defines.

If you want to write wallet software that does not have the complexity to deal with just one account, it is not BIP64 compliant. It could try to define its own purpose system, with a hierarchy without accounts in it. I'm not sure this is a very interesting use case, but I like how strict it is. Construction of related chains for multisig addresses is perhaps a better example of a different purpose structure. On 23 Apr 2014 22:03, "Luke-Jr" <luke@dashjr•org> wrote:

Quoted text (31 lines)
> On Wednesday, April 23, 2014 7:57:46 PM slush wrote:
> > On Wed, Apr 23, 2014 at 9:55 PM, Luke-Jr <luke@dashjr•org> wrote:
> > > Any wallet should import all the coins just fine, it just wouldn't
> *use*
> > > any
> > > account other than 0. Remember addresses are used to receive bitcoins;
> > > once the UTXOs are in the wallet, they are no longer associated with
> the
> > > address or
> > > any other details of how they were received.
> >
> > Wallet don't see UTXO until it scans all branches/accounts on HD node
> > import.
>
> Yes, it should scan all possible (under the BIP-defined structure) branches
> regardless of which features it supports.
>
> Luke
>
>
> ------------------------------------------------------------------------------
> Start Your Social Network Today - Download eXo Platform
> Build your Enterprise Intranet with eXo Platform Software
> Java Based Open Source Intranet - Social, Extensible, Cloud Ready
> Get Started Now And Turn Your Intranet Into A Collaboration Platform
> http://p.sf.net/sfu/ExoPlatform
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
Luke Dashjr in reply to Pavol Rusnak permalink original

On Wednesday, April 23, 2014 8:04:03 PM Pavol Rusnak wrote:

Quoted text (6 lines)
> On 04/23/2014 10:01 PM, Luke-Jr wrote:
> > Yes, it should scan all possible (under the BIP-defined structure)
> > branches regardless of which features it supports.
>
> So you suggest to scan for accounts, show balances but don't allow user
> to spend them? Does not seem right to me.

Scan all branches for UTXOs, then you have the balance for the wallet. Account balances are metadata, so cannot be known from the seed alone. If you want to have a way to restore accounts, you must define some more detailed wallet file format (which could be built on top of this).

On Wednesday, April 23, 2014 8:04:35 PM you wrote:

Quoted text (10 lines)
> On 23.04.2014, at 22:02, Luke-Jr <luke@dashjr•org> wrote:
> > On Wednesday, April 23, 2014 8:01:16 PM Tamas Blummer wrote:
> >> The wallet has to know how it got the UTXO in order to be able to spend
> >> it.
> >
> > No it doesn't... Just the assigned scriptPubKey and secret(s) required to
> > satisfy it.
>
> To know the secret it needs to know which key coordinate to use that is
> logically the same as knowing the address it used to receive it.

Sure, it *knows* what address was used to receive it. But at the point it's a UTXO, that address is no longer relevant.

Luke

Pieter Wuille in reply to Pavol Rusnak permalink original

I believe Luke means scanning chains as defined by the structure, but not handing out addresses from other accounts than the first one.

That's certainly a possibly way to compatibly implement BIP64, but it doesn't reduce all that much complexity. I hope people would choose that over defining their own accountless structure though. On 23 Apr 2014 22:06, "Pavol Rusnak" <stick@gk2•sk> wrote:

Quoted text (25 lines)
> On 04/23/2014 10:01 PM, Luke-Jr wrote:
> > Yes, it should scan all possible (under the BIP-defined structure)
> branches
> > regardless of which features it supports.
>
> So you suggest to scan for accounts, show balances but don't allow user
> to spend them? Does not seem right to me.
>
> --
> Best Regards / S pozdravom,
>
> Pavol Rusnak <stick@gk2•sk>
>
>
> ------------------------------------------------------------------------------
> Start Your Social Network Today - Download eXo Platform
> Build your Enterprise Intranet with eXo Platform Software
> Java Based Open Source Intranet - Social, Extensible, Cloud Ready
> Get Started Now And Turn Your Intranet Into A Collaboration Platform
> http://p.sf.net/sfu/ExoPlatform
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
Pavol Rusnak in reply to Luke Dashjr permalink original

On 04/23/2014 10:09 PM, Luke-Jr wrote:

Quoted text (4 lines)
> Scan all branches for UTXOs, then you have the balance for the wallet. Account
> balances are metadata, so cannot be known from the seed alone. If you want to
> have a way to restore accounts, you must define some more detailed wallet file
> format (which could be built on top of this).

I think one of the following is happening:

a) you have not read the document
b) you don't understand what accounts are, because you have not read
   the document
c) you don't understand how restoring accounts work, because you
   have not read the document
d) you don't understand that accounts funds are not meant to be mixed
   together, because you have not read the document
e) I got your emails wrong because I am not a native speaker,
   but please read the the document ;-)
Tamas Blummer in reply to Luke Dashjr permalink original

On 23.04.2014, at 22:09, Luke-Jr <luke@dashjr•org> wrote:

Quoted text (28 lines)
> On Wednesday, April 23, 2014 8:04:03 PM Pavol Rusnak wrote:
>> On 04/23/2014 10:01 PM, Luke-Jr wrote:
>>> Yes, it should scan all possible (under the BIP-defined structure)
>>> branches regardless of which features it supports.
>>
>> So you suggest to scan for accounts, show balances but don't allow user
>> to spend them? Does not seem right to me.
>
> Scan all branches for UTXOs, then you have the balance for the wallet. Account
> balances are metadata, so cannot be known from the seed alone. If you want to
> have a way to restore accounts, you must define some more detailed wallet file
> format (which could be built on top of this).
>
> On Wednesday, April 23, 2014 8:04:35 PM you wrote:
>> On 23.04.2014, at 22:02, Luke-Jr <luke@dashjr•org> wrote:
>>> On Wednesday, April 23, 2014 8:01:16 PM Tamas Blummer wrote:
>>>> The wallet has to know how it got the UTXO in order to be able to spend
>>>> it.
>>>
>>> No it doesn't... Just the assigned scriptPubKey and secret(s) required to
>>> satisfy it.
>>
>> To know the secret it needs to know which key coordinate to use that is
>> logically the same as knowing the address it used to receive it.
>
> Sure, it *knows* what address was used to receive it. But at the point it's a
> UTXO, that address is no longer relevant.
>

In context of BIP32 one does not store secrets but re-create them on-the-fly if needed using key coordinates known to the UTXO. Individual secrets per UTXO are about as irrelevant and accessible as addresses.

Luke Dashjr in reply to Pavol Rusnak permalink original

On Wednesday, April 23, 2014 8:16:45 PM Pavol Rusnak wrote:

Quoted text (17 lines)
> On 04/23/2014 10:09 PM, Luke-Jr wrote:
> > Scan all branches for UTXOs, then you have the balance for the wallet.
> > Account balances are metadata, so cannot be known from the seed alone.
> > If you want to have a way to restore accounts, you must define some more
> > detailed wallet file format (which could be built on top of this).
>
> I think one of the following is happening:
>
> a) you have not read the document
> b) you don't understand what accounts are, because you have not read
>    the document
> c) you don't understand how restoring accounts work, because you
>    have not read the document
> d) you don't understand that accounts funds are not meant to be mixed
>    together, because you have not read the document
> e) I got your emails wrong because I am not a native speaker,
>    but please read the the document ;-)

You missed one: f) I missed the part where BIP 32 redefined "account" to mean "subwallet" instead of what has traditionally been called "account" in Bitcoin.

:)

In that case, single-subwallet wallet software probably needs to have the user provide a subwallet number in addition to the seed.

Luke

Pavol Rusnak in reply to Luke Dashjr permalink original

On 04/23/2014 10:32 PM, Luke-Jr wrote:

Quoted text (2 lines)
> f) I missed the part where BIP 32 redefined "account" to mean "subwallet"
> instead of what has traditionally been called "account" in Bitcoin.

Ah, okay. The last time I saw Bitcoin-qt it was still using independent addresses.

Quoted text (2 lines)
> In that case, single-subwallet wallet software probably needs to have the user
> provide a subwallet number in addition to the seed.

Which brings us back to my original complaint that the user can be confused because he doesn't see all his funds.

Luke Dashjr in reply to Pavol Rusnak permalink original

On Wednesday, April 23, 2014 8:35:50 PM Pavol Rusnak wrote:

Quoted text (6 lines)
> On 04/23/2014 10:32 PM, Luke-Jr wrote:
> > f) I missed the part where BIP 32 redefined "account" to mean "subwallet"
> > instead of what has traditionally been called "account" in Bitcoin.
>
> Ah, okay. The last time I saw Bitcoin-qt it was still using independent
> addresses.

It is. Accounts have been a bitcoind feature since before 0.4.

Quoted text (5 lines)
> > In that case, single-subwallet wallet software probably needs to have the
> > user provide a subwallet number in addition to the seed.
>
> Which brings us back to my original complaint that the user can be
> confused because he doesn't see all his funds.

I don't see how. The user knows he has money in different subwallets. As long as he has a way to specify which subwallet he is accessing in single-subwallet clients, there shouldn't be a problem.

Luke

Pavol Rusnak in reply to Luke Dashjr permalink original

On 04/23/2014 10:41 PM, Luke-Jr wrote:

Quoted text (3 lines)
> I don't see how. The user knows he has money in different subwallets. As long
> as he has a way to specify which subwallet he is accessing in single-subwallet
> clients, there shouldn't be a problem.

Right. But these clients have no right to call themselves BIP64 compatible then.

Pieter Wuille in reply to Pavol Rusnak permalink original

On Wed, Apr 23, 2014 at 10:43 PM, Pavol Rusnak <stick@gk2•sk> wrote:

Quoted text (7 lines)
> On 04/23/2014 10:41 PM, Luke-Jr wrote:
>> I don't see how. The user knows he has money in different subwallets. As long
>> as he has a way to specify which subwallet he is accessing in single-subwallet
>> clients, there shouldn't be a problem.
>
> Right. But these clients have no right to call themselves BIP64
> compatible then.

Would you consider software which scans all accounts as specified by BIP64, but has no user interface option to distinguish them in any way, view them independently, and has no ability to keep the coins apart... compatible with BIP64?

According to the argument here mentioned earlier ("all or nothing"), it is, as it will not break interoperability with other BIP64 software. Still, it doesn't support the accounts feature, and perhaps that's fine?

Luke Dashjr in reply to Pavol Rusnak permalink original

On Wednesday, April 23, 2014 8:43:57 PM Pavol Rusnak wrote:

Quoted text (7 lines)
> On 04/23/2014 10:41 PM, Luke-Jr wrote:
> > I don't see how. The user knows he has money in different subwallets. As
> > long as he has a way to specify which subwallet he is accessing in
> > single-subwallet clients, there shouldn't be a problem.
>
> Right. But these clients have no right to call themselves BIP64
> compatible then.

Then BIP 64 is pretty restrictive. Most end users really have no need for subwallet support.

Pavol Rusnak in reply to Pieter Wuille permalink original

On 04/23/2014 10:54 PM, Pieter Wuille wrote:

Quoted text (4 lines)
> Would you consider software which scans all accounts as specified by
> BIP64, but has no user interface option to distinguish them in any
> way, view them independently, and has no ability to keep the coins
> apart... compatible with BIP64?

This is not a desired behavior. Do you have any idea how to make it even more explicit in the spec? Currently we just have (in Account sectrion):

"This level splits the key space into independent user identities, so the wallet never mixes the coins across different accounts."

I would like to make it obvious from the spec that if you mix funds accross the accounts you are not doing a right thing and you are not compliant to the spec.

slush in reply to Pieter Wuille permalink original

On Wed, Apr 23, 2014 at 10:54 PM, Pieter Wuille <pieter.wuille@gmail•com>wrote:

Quoted text (7 lines)
>
> Would you consider software which scans all accounts as specified by
> BIP64, but has no user interface option to distinguish them in any
> way, view them independently, and has no ability to keep the coins
> apart... compatible with BIP64?
>
>

No, because one of requirement of BIP64 is to do not mix addressess between accounts. Without handling those accounts fully, it won't pass this requirement.

("This level [accounts] splits the key space into independent user identities, so the wallet never mixes the coins across different accounts.")

Marek

Luke Dashjr in reply to Pavol Rusnak permalink original

On Wednesday, April 23, 2014 9:06:26 PM Pavol Rusnak wrote:

Quoted text (16 lines)
> On 04/23/2014 10:54 PM, Pieter Wuille wrote:
> > Would you consider software which scans all accounts as specified by
> > BIP64, but has no user interface option to distinguish them in any
> > way, view them independently, and has no ability to keep the coins
> > apart... compatible with BIP64?
>
> This is not a desired behavior. Do you have any idea how to make it even
> more explicit in the spec? Currently we just have (in Account sectrion):
>
> "This level splits the key space into independent user identities, so
> the wallet never mixes the coins across different accounts."
>
>
> I would like to make it obvious from the spec that if you mix funds
> accross the accounts you are not doing a right thing and you are not
> compliant to the spec.

How do you get the more expected/usual behaviour of mixing funds between accounts? Only a very niche user base needs funds isolated...

Luke

Greg Maxwell in reply to Luke Dashjr permalink original

On Wed, Apr 23, 2014 at 2:18 PM, Luke-Jr <luke@dashjr•org> wrote:

Quoted text (2 lines)
> How do you get the more expected/usual behaviour of mixing funds between
> accounts? Only a very niche user base needs funds isolated...

Hopefully it would be clarified as only a MUST NOT do so silently... "I have funds split across two wallets and it WONT LET ME SPEND THEM" sounds like a terrible user experience. :)

Pavol Rusnak in reply to Luke Dashjr permalink original

On 04/23/2014 11:18 PM, Luke-Jr wrote:

Quoted text (1 line)
> Only a very niche user base needs funds isolated...

Our users do and we are creating this BIP for them, because we love them. ;)

Pavol Rusnak in reply to Greg Maxwell permalink original

On 04/23/2014 11:22 PM, Gregory Maxwell wrote:

Quoted text (3 lines)
> Hopefully it would be clarified as only a MUST NOT do so silently...
> "I have funds split across two wallets and it WONT LET ME SPEND THEM"
> sounds like a terrible user experience. :)

That is a subjective matter. To me it makes PERFECT SENSE that funds across accounts NEVER MIX. One can still send funds from one account to another and then perform another spend.

That's what I consider a consistent and thus GOOD user experience. What's the purpose of Accounts if wallet mixes coins among them anyway?

I know it's not good to use classic bank accounts as analogy, but that's exactly how they work. Or have you every done ONE transaction from two bank accounts simultaneously?

Pieter Wuille in reply to Pavol Rusnak permalink original

On Wed, Apr 23, 2014 at 11:33 PM, Pavol Rusnak <stick@gk2•sk> wrote:

Quoted text (8 lines)
> On 04/23/2014 11:22 PM, Gregory Maxwell wrote:
>> Hopefully it would be clarified as only a MUST NOT do so silently...
>> "I have funds split across two wallets and it WONT LET ME SPEND THEM"
>> sounds like a terrible user experience. :)
>
> That is a subjective matter. To me it makes PERFECT SENSE that funds
> across accounts NEVER MIX. One can still send funds from one account to
> another and then perform another spend.

In that case, maybe it makes sense to define another purpose id without accounts as well already.

I believe many simple wallets will find multiple subwallets too burdening for the user experience, or not worth the technical complexity.

Pavol Rusnak in reply to Pieter Wuille permalink original

On 04/23/2014 11:42 PM, Pieter Wuille wrote:

Quoted text (6 lines)
> In that case, maybe it makes sense to define another purpose id
> without accounts as well already.
>
> I believe many simple wallets will find multiple subwallets too
> burdening for the user experience, or not worth the technical
> complexity.

Right. See my previous email where I describe hypothetical creation of BIP65 and BIP66 which the exact thing you describe.

Greg Maxwell in reply to Pieter Wuille permalink original

On Wed, Apr 23, 2014 at 2:42 PM, Pieter Wuille <pieter.wuille@gmail•com> wrote:

Quoted text (6 lines)
> In that case, maybe it makes sense to define another purpose id
> without accounts as well already.
>
> I believe many simple wallets will find multiple subwallets too
> burdening for the user experience, or not worth the technical
> complexity.

Or implement them but in a form where the different wallets can have different security policies and thus wouldn't share a common piece of private key material. I can see it being pretty confusing to have multiple wallets which are both sharing a private key and not sharing a private key.

Luke Dashjr in reply to Pavol Rusnak permalink original

On Wednesday, April 23, 2014 9:33:48 PM Pavol Rusnak wrote:

Quoted text (8 lines)
> On 04/23/2014 11:22 PM, Gregory Maxwell wrote:
> > Hopefully it would be clarified as only a MUST NOT do so silently...
> > "I have funds split across two wallets and it WONT LET ME SPEND THEM"
> > sounds like a terrible user experience. :)
>
> That is a subjective matter. To me it makes PERFECT SENSE that funds
> across accounts NEVER MIX. One can still send funds from one account to
> another and then perform another spend.

Only by crossing the blockchain unnecessarily.

Quoted text (2 lines)
> That's what I consider a consistent and thus GOOD user experience.
> What's the purpose of Accounts if wallet mixes coins among them anyway?

To keep balances separated. For example, I might have an account for each of my children, move their allowance money to it every day/week (off-chain), and they can spend only what they have in their account. Or an exchange might have an account for each user's deposits. In neither case, would it make sense to pay special attention to which UTXOs are consumed in transactions from any of the account.

The only use case that requires tracking UTXOs per-account is when you want to have multiple uncoordinated copies of the wallet in use at the same time, and therefore need to immediately identify which account a transaction came from based on the UTXOs it consumed - although even then, you could still mix funds as long as you use only the first UTXO for tracking, so maybe there isn't even a niche use case! In any case, Trezor, being a hardware wallet, should never overlap with this niche...?

Quoted text (3 lines)
> I know it's not good to use classic bank accounts as analogy, but that's
> exactly how they work. Or have you every done ONE transaction from two
> bank accounts simultaneously?

Bad analogy. Banks *always* mix funds. You don't expect your bank wire to use exactly the specific dollar bills you deposited, do you??

Luke

Thomas Voegtlin in reply to Luke Dashjr permalink original

Le 23/04/2014 21:44, Luke-Jr a écrit :

Quoted text (5 lines)
> On Wednesday, April 23, 2014 7:29:04 PM Pavol Rusnak wrote:
>>
>> Examples: Electrum does not want to use accounts [...]

> Why do clients need to use the features in BIP 64? If Electrum doesn't want to
> use accounts, [...]

To clarify: Electrum plans to have bip32 accounts; Multibit will not, afaik.

I think Pavol just wanted to make an example.

Pieter Wuille in reply to Thomas Voegtlin permalink original

On Thu, Apr 24, 2014 at 8:54 AM, Thomas Voegtlin <thomasv1@gmx•de> wrote:

Quoted text (5 lines)
>> Why do clients need to use the features in BIP 64? If Electrum doesn't want to
>> use accounts, [...]
>
> To clarify:
> Electrum plans to have bip32 accounts; Multibit will not, afaik.

To clarify: BIP64 has a much stricter definition for accounts than BIP32.

In BIP32, it is not well specified what accounts are used for. They can be used for "subwallets", "receive accounts" (as in bitcoind's account feature), "recurring payments", part of a chain used as multisig addresses, ... determined individually for each index.

In BIP64, they are strictly used for subwallets, and can't be used by anything else.

Greg Maxwell in reply to Pieter Wuille permalink original

On Thu, Apr 24, 2014 at 12:10 AM, Pieter Wuille <pieter.wuille@gmail•com> wrote:

Quoted text (17 lines)
> On Thu, Apr 24, 2014 at 8:54 AM, Thomas Voegtlin <thomasv1@gmx•de> wrote:
>>> Why do clients need to use the features in BIP 64? If Electrum doesn't want to
>>> use accounts, [...]
>>
>> To clarify:
>> Electrum plans to have bip32 accounts; Multibit will not, afaik.
>
> To clarify:
> BIP64 has a much stricter definition for accounts than BIP32.
>
> In BIP32, it is not well specified what accounts are used for. They
> can be used for "subwallets", "receive accounts" (as in bitcoind's
> account feature), "recurring payments", part of a chain used as
> multisig addresses, ... determined individually for each index.
>
> In BIP64, they are strictly used for subwallets, and can't be used by
> anything else.

It doesn't appear to me that reoccurring payments, receive accounts, multisig addresses, etc can be used with this proposal, but instead you must use a different purpose code and another BIP and are not compatible with the draft here.

Am I misunderstanding it? Will Electrum be limiting itself in this way? I'd consider it a unfortunate loss of functionality if wallets couldn't implement reoccurring payment chains without making users generate entirely different wallets (which they couldn't share funds across) since addresses for recurring payments was one of the main motivations in BIP32.

Thomas Voegtlin in reply to Pieter Wuille permalink original

Le 24/04/2014 09:10, Pieter Wuille a écrit :

Quoted text (11 lines)
> To clarify:
> BIP64 has a much stricter definition for accounts than BIP32.
>
> In BIP32, it is not well specified what accounts are used for. They
> can be used for "subwallets", "receive accounts" (as in bitcoind's
> account feature), "recurring payments", part of a chain used as
> multisig addresses, ... determined individually for each index.
>
> In BIP64, they are strictly used for subwallets, and can't be used by
> anything else.
>

yes, I saw that.

In particular, bip64 stipulates that the wallet "never mixes coins across different accounts". This is not what Electrum does currently. The UI allows you to chose between two modes: activate a single account (and the wallet will only use UTXOs from that acccount), or activate all accounts (and spend from all of them simultaneously).

Concerning multisig addresses, I have changed my mind: Electrum will use parallel BIP32 trees. A wallet will not mix standard and multisig accounts. I think that is better in terms of UX.

I agree with Mike Hearn's view that wallets with multiple accounts are probably too difficult to deal with for most users. If a user feels the need to have different "accounting identities", they will probably create different wallet files, instead of creating bip32 subwallets.

However, since multiple subwallets have been asked by many users, Electrum will propose them. But this should not be the default. More important, multiple accounts should never be required (in my previous implementation, they were required for multisig, because the wallet was creating multisig addresses in dedicated multisig accounts)

Thomas

Mike Hearn in reply to Pieter Wuille permalink original

Right. So part of this is my fault, I'm afraid, because I do not intend to implement any kind of subwallet/account support in bitcoinj. My reasons are:

   1. The bitcoinj API already lets you create and use multiple wallets.
   What's more, because of the desire to do key rotation (think rotating a
   previously unencrypted wallet to an encrypted one that is stored on SSD's
   that cannot reliably erase data), a bitcoinj wallet can actually contain
   multiple BIP32 seeds and hierarchies at once, although only the last one
   will be used for vending addresses. So adding subwallet support onto this
   makes it even more complicated.
   2. If there was a much better user experience to be enabled by this, it
   may be worth it, but I believe many people will find subwallets rather
   confusing. They don't match the analogy of bank accounts in several ways.
   For instance, transferring money across them leaks private data and costs
   miners fees, neither of which are true with banks.
   Also it differs in a more important way. People have different bank
   accounts because those accounts implement different policies. Current
   accounts may pay a lower interest rate than savings accounts, but have
   different features, and accounts can be used as security boundaries i.e. no
   card withdrawals from savings. But "subwallets" are not like this. The only
   justification for their existence is to avoid outputs being merged together
   to make payments - a subtle technical detail of the protocol that users are
   ill equipped to understand. If someone asked me "why should I create a
   second account" I would be unable to give them a satisfying answer without
   first teaching them about how the Bitcoin protocol works and the privacy
   implications of that, which is practically a lecture sized topic.
   3. MultiBit did support multiple wallets for a long time (just by
   creating multiple wallet files and using the support in bitcoinj for
   running them in parallel), but they decided to remove this feature in
   MultiBit HD because it caused support headaches. People would stash money
   in one wallet or the other, close the wallet and then forget and think they
   had lost it, etc. It may be that TREZOR type subwallets don't suffer this
   confusion because they can't be moved around or "closed" in the same way a
   file can be, but still, this is a data point against multiple simultaneous
   wallets. At least for products targeting entry level consumers.

Whilst I can well believe there are TREZOR users who are asking for this feature today, currently the costs feel a bit higher than the benefits.

It would be rather nice to be able to type in a mnemonic code that myTREZOR was initialised with and duplicate that wallet into a bitcoinj based wallet app. But if I have to implement subwallets and expose this in the API, and if all wallet authors that want to be able to share a wallet with myTREZOR have to expose subwallets in their GUIs too, even though the concept may prove confusing and hard to explain, then it might be more tempting to just tell users that want to switch wallet apps to send the money via the block chain instead.

On Thu, Apr 24, 2014 at 9:10 AM, Pieter Wuille <pieter.wuille@gmail•com>wrote:

Quoted text (34 lines)
> On Thu, Apr 24, 2014 at 8:54 AM, Thomas Voegtlin <thomasv1@gmx•de> wrote:
> >> Why do clients need to use the features in BIP 64? If Electrum doesn't
> want to
> >> use accounts, [...]
> >
> > To clarify:
> > Electrum plans to have bip32 accounts; Multibit will not, afaik.
>
> To clarify:
> BIP64 has a much stricter definition for accounts than BIP32.
>
> In BIP32, it is not well specified what accounts are used for. They
> can be used for "subwallets", "receive accounts" (as in bitcoind's
> account feature), "recurring payments", part of a chain used as
> multisig addresses, ... determined individually for each index.
>
> In BIP64, they are strictly used for subwallets, and can't be used by
> anything else.
>
> --
> Pieter
>
>
> ------------------------------------------------------------------------------
> Start Your Social Network Today - Download eXo Platform
> Build your Enterprise Intranet with eXo Platform Software
> Java Based Open Source Intranet - Social, Extensible, Cloud Ready
> Get Started Now And Turn Your Intranet Into A Collaboration Platform
> http://p.sf.net/sfu/ExoPlatform
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
Thomas Voegtlin in reply to Greg Maxwell permalink original

Le 24/04/2014 09:21, Gregory Maxwell a écrit :

Quoted text (14 lines)
>
> It doesn't appear to me that reoccurring payments, receive accounts,
> multisig addresses, etc can be used with this proposal, but instead
> you must use a different purpose code and another BIP and are not
> compatible with the draft here.
>
> Am I misunderstanding it?   Will Electrum be limiting itself in this
> way?  I'd consider it a unfortunate loss of functionality if wallets
> couldn't implement reoccurring payment chains without making users
> generate entirely different wallets (which they couldn't share funds
> across) since addresses for recurring payments was one of the main
> motivations in BIP32.
>
>

No, Electrum will not be limiting itself in this way. I believe that we are only at the beginning of exploring the different possibilities opened by HD wallets. It will probably take years until we have clear ideas on what users need, what choices they make, and how to organize everything. Therefore it is too early to take decisions that might limit future functionality.

I can see that it is very difficult today to find a consensus on wallet structure between wallet developers. In addition, I changed my mind several times on these questions, so I guess I will probably need to change things again in the future.

This is why I decided to include a version number in Electrum seeds. The version number will be updated everytime the wallet structure changes. I know many developers do not follow me on this, but that is something I am quite sure Electrum needs, despite all the other things I am not sure about :)

I think it is too early to aim for inter-wallet compatibility today. I guess we should postpone this goal, and move on with software releases. As Andreas pointed out, we should just make sure that we do not import an incompatible seed in another wallet, because not recovering all your bitcoins would be a terrible user experience; the version number built in the seed will ensure that for Electrum.