BIP70: why Google Protocol Buffers for encoding?
37 messages, 19 Jan to 24 Mar 2015. Started by Richard Brady.
Mike Hearn (7), Jeff Garzik (6), Nicolas Dorier (6), Richard Brady (3), Matt Whitlock (2), Peter Todd (2), Ross Nicoll (2), Isidor Zeuner (2), +7 more
Hi Gavin, Mike and co
Is there a strong driver behind the choice of Google Protocol Buffers for payment request encoding in BIP-0070?
Performance doesn't feel that relevant when you think that: 1. Payment requests are not broadcast, this is a request / response flow, much more akin to a web request. 2. One would be cramming this data into a binary format just so you can then attach it to a no-so-binary format such as HTTP.
Some great things about protocols/encodings such as HTTP/JSON/XML are: 1. They are human readable on-the-wire. No Wireshark plugin required, tcpdump or ngrep will do. 2. There are tons of great open source libraries and API for parsing / manipulating / generating. 3. It's really easy to hand-craft a test message for debugging. 4. The standards are much easier to read and write. They don't need to contain code like BIP-0070 currently does and they can contain examples, which BIP70 does not. 5. They are thoroughly specified by independent standards bodies such as the IETF. Gotta love a bit of MUST / SHOULD / MAY in a standard. 6. They're a family ;-)
Keen to hear your thoughts on this and very keen to watch the payment protocol grow regardless of encoding choice! My background is SIP / VoIP and I think that could be a fascinating use case for this protocol which I'm hoping to do some work on.
Best, Richard
Text formats such as XML or JSON are far less deterministic, are more loosely specified, have wide variance in parsing, are not very hash-able, the list goes on.
On Mon, Jan 19, 2015 at 2:07 PM, Richard Brady <rnbrady@gmail•com> wrote:
Quoted text (46 lines)
> Hi Gavin, Mike and co > > Is there a strong driver behind the choice of Google Protocol Buffers for > payment request encoding in BIP-0070? > > Performance doesn't feel that relevant when you think that: > 1. Payment requests are not broadcast, this is a request / response flow, > much more akin to a web request. > 2. One would be cramming this data into a binary format just so you can > then attach it to a no-so-binary format such as HTTP. > > Some great things about protocols/encodings such as HTTP/JSON/XML are: > 1. They are human readable on-the-wire. No Wireshark plugin required, > tcpdump or ngrep will do. > 2. There are tons of great open source libraries and API for parsing / > manipulating / generating. > 3. It's really easy to hand-craft a test message for debugging. > 4. The standards are much easier to read and write. They don't need to > contain code like BIP-0070 currently does and they can contain examples, > which BIP70 does not. > 5. They are thoroughly specified by independent standards bodies such as > the IETF. Gotta love a bit of MUST / SHOULD / MAY in a standard. > 6. They're a family ;-) > > Keen to hear your thoughts on this and very keen to watch the payment > protocol grow regardless of encoding choice! My background is SIP / VoIP > and I think that could be a fascinating use case for this protocol which > I'm hoping to do some work on. > > Best, > Richard > > > > ------------------------------------------------------------------------------ > New Year. New Location. New Benefits. New Data Center in Ashburn, VA. > GigeNET is offering a free month of service with a new server in Ashburn. > Choose from 2 high performing configs, both with 100TB of bandwidth. > Higher redundancy.Lower latency.Increased capacity.Completely compliant. > http://p.sf.net/sfu/gigenet > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > >
Fair points, although for me the line is blurred between which of those are security considerations vs performance considerations.
Richard
On 19 January 2015 at 19:09, Jeff Garzik <jgarzik@bitpay•com> wrote:
Quoted text (40 lines)
> Text formats such as XML or JSON are far less deterministic, are more > loosely specified, have wide variance in parsing, are not very hash-able, > the list goes on. > > > On Mon, Jan 19, 2015 at 2:07 PM, Richard Brady <rnbrady@gmail•com> wrote: > >> Hi Gavin, Mike and co >> >> Is there a strong driver behind the choice of Google Protocol Buffers for >> payment request encoding in BIP-0070? >> >> Performance doesn't feel that relevant when you think that: >> 1. Payment requests are not broadcast, this is a request / response flow, >> much more akin to a web request. >> 2. One would be cramming this data into a binary format just so you can >> then attach it to a no-so-binary format such as HTTP. >> >> Some great things about protocols/encodings such as HTTP/JSON/XML are: >> 1. They are human readable on-the-wire. No Wireshark plugin required, >> tcpdump or ngrep will do. >> 2. There are tons of great open source libraries and API for parsing / >> manipulating / generating. >> 3. It's really easy to hand-craft a test message for debugging. >> 4. The standards are much easier to read and write. They don't need to >> contain code like BIP-0070 currently does and they can contain examples, >> which BIP70 does not. >> 5. They are thoroughly specified by independent standards bodies such as >> the IETF. Gotta love a bit of MUST / SHOULD / MAY in a standard. >> 6. They're a family ;-) >> >> Keen to hear your thoughts on this and very keen to watch the payment >> protocol grow regardless of encoding choice! My background is SIP / VoIP >> and I think that could be a fascinating use case for this protocol which >> I'm hoping to do some work on. >> >> Best, >> Richard >> >>
Even if a compact binary encoding is a high priority, there are more "standard" choices than Google Protocol Buffers. For example, ASN.1 is a very rigorously defined standard that has been around for decades, and ASN.1 even has an XML encoding (XER) that is directly convertible to/from the binary encoding (BER/DER), given the schema. In practice, I'm mostly agnostic about what encoding is actually used in BIP70, and I wouldn't fault BIP70 for choosing Google Protocol Buffers, but the very existence of Protobuf perplexes me, as it apparently re-solves a problem that was solved 40 years ago by ASN.1. It's as though the engineers at Google weren't aware that ASN.1 existed.
On Monday, 19 January 2015, at 7:07 pm, Richard Brady wrote:
Quoted text (31 lines)
> Hi Gavin, Mike and co > > Is there a strong driver behind the choice of Google Protocol Buffers for > payment request encoding in BIP-0070? > > Performance doesn't feel that relevant when you think that: > 1. Payment requests are not broadcast, this is a request / response flow, > much more akin to a web request. > 2. One would be cramming this data into a binary format just so you can > then attach it to a no-so-binary format such as HTTP. > > Some great things about protocols/encodings such as HTTP/JSON/XML are: > 1. They are human readable on-the-wire. No Wireshark plugin required, > tcpdump or ngrep will do. > 2. There are tons of great open source libraries and API for parsing / > manipulating / generating. > 3. It's really easy to hand-craft a test message for debugging. > 4. The standards are much easier to read and write. They don't need to > contain code like BIP-0070 currently does and they can contain examples, > which BIP70 does not. > 5. They are thoroughly specified by independent standards bodies such as > the IETF. Gotta love a bit of MUST / SHOULD / MAY in a standard. > 6. They're a family ;-) > > Keen to hear your thoughts on this and very keen to watch the payment > protocol grow regardless of encoding choice! My background is SIP / VoIP > and I think that could be a fascinating use case for this protocol which > I'm hoping to do some work on. > > Best, > Richard
None of those listed were in the context of performance. Parsing of binary or text is quite fast these days, and is not really a consideration versus other needs such as a predictable encoding for a single data representation. XML and JSON both can represent the same post-evaluation user data a million different ways, which is awful for anything you are signing and hashing. Text formats also transit binary data very poorly, leading to unnecessary wrapping and unwrappiing (a programmatic, visibility & bug; again performance not a primary concern).
This is evident because both XML and JSON have standards efforts under way to correct some of these problems and make them more deterministic. However, such standards are not field deployed and widely supported by parsers and generators alike.
On Mon, Jan 19, 2015 at 2:16 PM, Richard Brady <rnbrady@gmail•com> wrote:
Quoted text (47 lines)
> Fair points, although for me the line is blurred between which of those > are security considerations vs performance considerations. > > Richard > > On 19 January 2015 at 19:09, Jeff Garzik <jgarzik@bitpay•com> wrote: > >> Text formats such as XML or JSON are far less deterministic, are more >> loosely specified, have wide variance in parsing, are not very hash-able, >> the list goes on. >> >> >> On Mon, Jan 19, 2015 at 2:07 PM, Richard Brady <rnbrady@gmail•com> wrote: >> >>> Hi Gavin, Mike and co >>> >>> Is there a strong driver behind the choice of Google Protocol Buffers >>> for payment request encoding in BIP-0070? >>> >>> Performance doesn't feel that relevant when you think that: >>> 1. Payment requests are not broadcast, this is a request / response >>> flow, much more akin to a web request. >>> 2. One would be cramming this data into a binary format just so you can >>> then attach it to a no-so-binary format such as HTTP. >>> >>> Some great things about protocols/encodings such as HTTP/JSON/XML are: >>> 1. They are human readable on-the-wire. No Wireshark plugin required, >>> tcpdump or ngrep will do. >>> 2. There are tons of great open source libraries and API for parsing / >>> manipulating / generating. >>> 3. It's really easy to hand-craft a test message for debugging. >>> 4. The standards are much easier to read and write. They don't need to >>> contain code like BIP-0070 currently does and they can contain examples, >>> which BIP70 does not. >>> 5. They are thoroughly specified by independent standards bodies such as >>> the IETF. Gotta love a bit of MUST / SHOULD / MAY in a standard. >>> 6. They're a family ;-) >>> >>> Keen to hear your thoughts on this and very keen to watch the payment >>> protocol grow regardless of encoding choice! My background is SIP / VoIP >>> and I think that could be a fascinating use case for this protocol which >>> I'm hoping to do some work on. >>> >>> Best, >>> Richard >>> >>>
The engineers at Google were well aware that ASN.1 existed. I can assure you of that, because I was one of them.
The protobuf FAQ has a very polite take on the matter:
https://developers.google.com/protocol-buffers/docs/faq
This email thread gives more enlightenment:
https://groups.google.com/forum/#!topic/protobuf/eNAZlnPKVW4
Anyone who has actually had to work with both ASN.1 and protocol buffers will be able to explain why ASN.1 should not be chosen for any modern formats. A lot of it boils down to simplicty and quality of implementations, especially open source implementations.
With respect to the specific concerns Richard raises:
Performance doesn't feel that relevant when you think that:
Quoted text (1 line)
>
Performance wasn't a concern.
Quoted text (3 lines)
> 2. One would be cramming this data into a binary format just so you can > then attach it to a no-so-binary format such as HTTP. >
HTTP transmits files as binary on the wire. So it's binary-clean and, moreover, HTTP/2 aka SPDY is fully binary and doesn't use text anywhere except the gzip dictionary.
Quoted text (3 lines)
> 2. There are tons of great open source libraries and API for parsing / > manipulating / generating. >
Luckily, this is also true of protocol buffers. Language support is pretty good these days.
Quoted text (4 lines)
> 4. The standards are much easier to read and write. They don't need to > contain code like BIP-0070 currently does and they can contain examples, > which BIP70 does not. >
BIP 70 doesn't contain any code, as far as I know. The protobuf schema might look like code, but it's not - it's just a description of what fields a message can contain and their types. This is very relevant for a specification!
JSON in particular is pretty awful and I don't like it much. It suffers complexities with things as basic as encoding numbers and strings. It's very much unsuited to applications where correctness matters and where you're dealing with binary structures.
ASN.1 is not nearly as flexible when it comes to well-supported libraries, generators, and the ecosystem that surrounds the actual encoding. You don't see ASN.1 compilers + language support packages for [all popular programming languages], as you do with protobufs.
Google engineers were well aware it existed I'm sure. There are wider considerations beyond the low-level specified format.
Protobufs have their problems and aren't perfect, but ASN.1 ecosystem is far less developed in the programming ecosystem, far less approachable for programmers. BIP70 wouldn't have been as easily and widely adopted if ASN.1 had been chosen.
On Mon, Jan 19, 2015 at 2:19 PM, Matt Whitlock <bip@mattwhitlock•name> wrote:
Quoted text (57 lines)
> Even if a compact binary encoding is a high priority, there are more > "standard" choices than Google Protocol Buffers. For example, ASN.1 is a > very rigorously defined standard that has been around for decades, and > ASN.1 even has an XML encoding (XER) that is directly convertible to/from > the binary encoding (BER/DER), given the schema. In practice, I'm mostly > agnostic about what encoding is actually used in BIP70, and I wouldn't > fault BIP70 for choosing Google Protocol Buffers, but the very existence of > Protobuf perplexes me, as it apparently re-solves a problem that was solved > 40 years ago by ASN.1. It's as though the engineers at Google weren't aware > that ASN.1 existed. > > > On Monday, 19 January 2015, at 7:07 pm, Richard Brady wrote: > > Hi Gavin, Mike and co > > > > Is there a strong driver behind the choice of Google Protocol Buffers for > > payment request encoding in BIP-0070? > > > > Performance doesn't feel that relevant when you think that: > > 1. Payment requests are not broadcast, this is a request / response flow, > > much more akin to a web request. > > 2. One would be cramming this data into a binary format just so you can > > then attach it to a no-so-binary format such as HTTP. > > > > Some great things about protocols/encodings such as HTTP/JSON/XML are: > > 1. They are human readable on-the-wire. No Wireshark plugin required, > > tcpdump or ngrep will do. > > 2. There are tons of great open source libraries and API for parsing / > > manipulating / generating. > > 3. It's really easy to hand-craft a test message for debugging. > > 4. The standards are much easier to read and write. They don't need to > > contain code like BIP-0070 currently does and they can contain examples, > > which BIP70 does not. > > 5. They are thoroughly specified by independent standards bodies such as > > the IETF. Gotta love a bit of MUST / SHOULD / MAY in a standard. > > 6. They're a family ;-) > > > > Keen to hear your thoughts on this and very keen to watch the payment > > protocol grow regardless of encoding choice! My background is SIP / VoIP > > and I think that could be a fascinating use case for this protocol which > > I'm hoping to do some work on. > > > > Best, > > Richard > > > ------------------------------------------------------------------------------ > New Year. New Location. New Benefits. New Data Center in Ashburn, VA. > GigeNET is offering a free month of service with a new server in Ashburn. > Choose from 2 high performing configs, both with 100TB of bandwidth. > Higher redundancy.Lower latency.Increased capacity.Completely compliant. > http://p.sf.net/sfu/gigenet > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development >
On 19 January 2015 12:09:13 GMT-07:00, Jeff Garzik <jgarzik@bitpay•com> wrote:
Quoted text (4 lines)
>Text formats such as XML or JSON are far less deterministic, are more >loosely specified, have wide variance in parsing, are not very >hash-able, >the list goes on.
Protocol buffers isn't any more hashable than XML or json - round trips aren't deterministic with standard protobuf libraries. To make it deterministic you end up creating a new standard.
I have this problem for an asset representation standard for one of my clients, and I've reluctantly had to roll my own format.
Thanks guys, great answers.
The design choice certainly makes a lot more sense now regardless of whether one agrees with it or not.
Regards, Richard
I'm a bit confused. It's been a long time since I looked at protobuf (and will have to dig into it soon), but I seem to recall it doesn't have any of the determinism properties you guys just said. It is intended to allow you to skip details of the on-the-wire representations and just send a bunch of named fields between systems. I thought there was no guarantee that two identical protobuf structures will get serialized identically...?
On 01/19/2015 02:57 PM, Richard Brady wrote:
Quoted text (22 lines)
> Thanks guys, great answers. > > The design choice certainly makes a lot more sense now regardless of > whether one agrees with it or not. > > Regards, > Richard > > > > ------------------------------------------------------------------------------ > New Year. New Location. New Benefits. New Data Center in Ashburn, VA. > GigeNET is offering a free month of service with a new server in Ashburn. > Choose from 2 high performing configs, both with 100TB of bandwidth. > Higher redundancy.Lower latency.Increased capacity.Completely compliant. > http://p.sf.net/sfu/gigenet > > > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development
That's 100% true: BIP70 passes around serialized protobuf data that it signs directly for this reason; it could just as easily be a byte array with json in it. (not that json/XML/etc. doesn't have other flaws)
On 19 January 2015 13:03:32 GMT-07:00, Alan Reiner <etotheipi@gmail•com> wrote:
Quoted text (58 lines)
>I'm a bit confused. It's been a long time since I looked at protobuf >(and will have to dig into it soon), but I seem to recall it doesn't >have any of the determinism properties you guys just said. It is >intended to allow you to skip details of the on-the-wire >representations >and just send a bunch of named fields between systems. I thought there >was no guarantee that two identical protobuf structures will get >serialized identically...? > > > > >On 01/19/2015 02:57 PM, Richard Brady wrote: >> Thanks guys, great answers. >> >> The design choice certainly makes a lot more sense now regardless of >> whether one agrees with it or not. >> >> Regards, >> Richard >> >> >> >> >------------------------------------------------------------------------------ >> New Year. New Location. New Benefits. New Data Center in Ashburn, VA. >> GigeNET is offering a free month of service with a new server in >Ashburn. >> Choose from 2 high performing configs, both with 100TB of bandwidth. >> Higher redundancy.Lower latency.Increased capacity.Completely >compliant. >> http://p.sf.net/sfu/gigenet >> >> >> _______________________________________________ >> Bitcoin-development mailing list >> Bitcoin-development@lists•sourceforge•net >> https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > > >------------------------------------------------------------------------ > >------------------------------------------------------------------------------ >New Year. New Location. New Benefits. New Data Center in Ashburn, VA. >GigeNET is offering a free month of service with a new server in >Ashburn. >Choose from 2 high performing configs, both with 100TB of bandwidth. >Higher redundancy.Lower latency.Increased capacity.Completely >compliant. >http://p.sf.net/sfu/gigenet > >------------------------------------------------------------------------ > >_______________________________________________ >Bitcoin-development mailing list >Bitcoin-development@lists•sourceforge•net >https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Quoted text (5 lines)
> > I'm a bit confused. It's been a long time since I looked at protobuf (and > will have to dig into it soon), but I seem to recall it doesn't have any of > the determinism properties you guys just said. >
It's not guaranteed no, which is why we store signed sub-messages as byte arrays instead of typed submessages. In practice though, most implementations do seem to serialise things the same way. I recall Python used to be an odd one out, unsure if it still is.
OK, I guess we can boil this down more simply. BIP 70 uses protocol buffers because I designed it and implemented the original prototype (with lots of input from Gavin and an earlier proposal by sipa). I used protocol buffers because, beyond all their nice properties, I used to work at Google and so was very familiar with them.
On Mon, Jan 19, 2015 at 3:40 PM, Mike Hearn <mike@plan99•net> wrote:
Quoted text (7 lines)
> OK, I guess we can boil this down more simply. BIP 70 uses protocol >> buffers because I designed it and implemented the original prototype (with >> lots of input from Gavin and an earlier proposal by sipa). I used protocol >> buffers because, beyond all their nice properties, I used to work at Google >> and so was very familiar with them. >> >
What Mike said. Runner-up for encoding was JSON.
XML+ASN.1 was Right Out, because lots of us hate XML and ASN.1 with a burning passion. Complexity is the Enemy of Security, and both XML and ASN.1 are too complex.
For what it's worth, there was consideration of replacing protocol buffers when modifying BIP70 to function with the altcoin I work on (changes were required anyway in eliminate any risk that payment requests could not be accidentally applied to the wrong blockchain). The eventual conclusion was that while we might have used JSON or XML if we were starting from scratch, there's no choice that's clearly better. While deployed infrastructure for payment protocol is still quite limited, it seems that the cost to replace at this point is higher than not.
If there's ever a major reworking of the standard, for example to handle recurring payments, it's probably worth thinking about then, but protocol buffers result in a compact data format which is supported by most major languages (and size is a concern if dealing with Bluetooth or NFC), and has no major drawbacks I am aware of.
Ross
On 19/01/2015 20:40, Mike Hearn wrote:
Quoted text (29 lines)
>> I'm a bit confused. It's been a long time since I looked at protobuf (and >> will have to dig into it soon), but I seem to recall it doesn't have any of >> the determinism properties you guys just said. >> > It's not guaranteed no, which is why we store signed sub-messages as byte > arrays instead of typed submessages. In practice though, most > implementations do seem to serialise things the same way. I recall Python > used to be an odd one out, unsure if it still is. > > OK, I guess we can boil this down more simply. BIP 70 uses protocol buffers > because I designed it and implemented the original prototype (with lots of > input from Gavin and an earlier proposal by sipa). I used protocol buffers > because, beyond all their nice properties, I used to work at Google and so > was very familiar with them. > > > > ------------------------------------------------------------------------------ > New Year. New Location. New Benefits. New Data Center in Ashburn, VA. > GigeNET is offering a free month of service with a new server in Ashburn. > Choose from 2 high performing configs, both with 100TB of bandwidth. > Higher redundancy.Lower latency.Increased capacity.Completely compliant. > http://p.sf.net/sfu/gigenet > > > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Correct. I should have said "more likely to be deterministic" Bitcoin Core does not *rely* on determinism in BIP70; I was referring to recent upstream efforts to make protobufs usable in a deterministic fashion by default.
On Mon, Jan 19, 2015 at 3:03 PM, Alan Reiner <etotheipi@gmail•com> wrote:
Quoted text (49 lines)
> I'm a bit confused. It's been a long time since I looked at protobuf > (and will have to dig into it soon), but I seem to recall it doesn't have > any of the determinism properties you guys just said. It is intended to > allow you to skip details of the on-the-wire representations and just send > a bunch of named fields between systems. I thought there was no guarantee > that two identical protobuf structures will get serialized identically...? > > > > > > On 01/19/2015 02:57 PM, Richard Brady wrote: > > Thanks guys, great answers. > > The design choice certainly makes a lot more sense now regardless of > whether one agrees with it or not. > > Regards, > Richard > > > > ------------------------------------------------------------------------------ > New Year. New Location. New Benefits. New Data Center in Ashburn, VA. > GigeNET is offering a free month of service with a new server in Ashburn. > Choose from 2 high performing configs, both with 100TB of bandwidth. > Higher redundancy.Lower latency.Increased capacity.Completely compliant.http://p.sf.net/sfu/gigenet > > > > _______________________________________________ > Bitcoin-development mailing listBitcoin-development@lists•sourceforge•nethttps://lists.sourceforge.net/lists/listinfo/bitcoin-development > > > > > ------------------------------------------------------------------------------ > New Year. New Location. New Benefits. New Data Center in Ashburn, VA. > GigeNET is offering a free month of service with a new server in Ashburn. > Choose from 2 high performing configs, both with 100TB of bandwidth. > Higher redundancy.Lower latency.Increased capacity.Completely compliant. > http://p.sf.net/sfu/gigenet > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > >
Damn if SPKI had won out we would be parsing S-expressions instead of X.509 certificates. ASN.1 is not fun IMHO.
Quoted text (27 lines)
> On Jan 19, 2015, at 3:56 PM, Gavin Andresen <gavin@bitcoinfoundation•org> wrote: > > On Mon, Jan 19, 2015 at 3:40 PM, Mike Hearn <mike@plan99•net> wrote: >>> OK, I guess we can boil this down more simply. BIP 70 uses protocol buffers because I designed it and implemented the original prototype (with lots of input from Gavin and an earlier proposal by sipa). I used protocol buffers because, beyond all their nice properties, I used to work at Google and so was very familiar with them. > > > What Mike said. Runner-up for encoding was JSON. > > XML+ASN.1 was Right Out, because lots of us hate XML and ASN.1 with a burning passion. Complexity is the Enemy of Security, and both XML and ASN.1 are too complex. > > > -- > -- > Gavin Andresen > Chief Scientist, Bitcoin Foundation > https://www.bitcoinfoundation.org/ > > ------------------------------------------------------------------------------ > New Year. New Location. New Benefits. New Data Center in Ashburn, VA. > GigeNET is offering a free month of service with a new server in Ashburn. > Choose from 2 high performing configs, both with 100TB of bandwidth. > Higher redundancy.Lower latency.Increased capacity.Completely compliant. > http://p.sf.net/sfu/gigenet > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Quoted text (4 lines)
> For what it's worth, there was consideration of replacing protocol > buffers when modifying BIP70 to function with the altcoin I work on > (changes were required anyway in eliminate any risk that payment > requests could not be accidentally applied to the wrong blockchain).
Why not serialize some kind of blockchain identifier with the messages? Arbitrarily deviating from a given design choice just for the sake of doing it differently may serve the goal of creating more overall code diversity, but would not necessarily serve the quality of the blockchain network where it is done for.
Best regards,
Isidor
That was essentially what we did in the end, we replaced the network identifier ("main"/"test") with the genesis block hash. The result is never going to accidentally work with Bitcoin Core (nor vice-versa), but is readily extensible to any other altcoins that want to use the specification without requiring any sort of central registry.
Ross
On 24/01/15 13:19, Isidor Zeuner wrote:
Quoted text (14 lines)
>> For what it's worth, there was consideration of replacing protocol >> buffers when modifying BIP70 to function with the altcoin I work on >> (changes were required anyway in eliminate any risk that payment >> requests could not be accidentally applied to the wrong blockchain). > > Why not serialize some kind of blockchain identifier with the > messages? Arbitrarily deviating from a given design choice just for > the sake of doing it differently may serve the goal of creating more > overall code diversity, but would not necessarily serve the quality of > the blockchain network where it is done for. > > Best regards, > > Isidor
I agree that the use protocol buffer and x509 by BIP70 is a poor choice.
The choice should have been done to maximize portability, not to maximize efficiency and flexibility.
What I ended up doing for having a similar codebase on all plateform is to parse a BIP70 messages with the help of a web service that convert it to JSON. I don't like this solution since it had a trust dependency, and the certificate verification become handled by the web service, not the device. But even if I solved google buffer problem, I would stumble upon having headache to validate the x509 certificate chain on every plateforms.
A simple BIP70 using JSON + HTTPS would have make things more easy. I agree that it requires that the merchant own the domain name of the BIP70 endpoint, but I don't consider such a big of a deal, since this is how e-commerce works.
On Wed, 28 Jan 2015, Nicolas DORIER wrote:
Quoted text (1 line)
> I agree that the use protocol buffer and x509 by BIP70 is a poor choice.
Well x509 is an international standard in common use, you can't do much better with regard to portability. Your suggestion about HTTPS makes little sense, you do know what TLS uses x509 internally as well?
Re: protocol buffers, I don't know if it's the best possible one, but one serialization method had to be picked. If it weren't, we could still have still been discussing which one to use by now. Just like for JSON there are bindings for many languages.
Though JSON parsers are much more diverse, which people using Bitcoin Core's RPC have bumped into e.g. some have some problems handling large numbers. Something you wouldn't expect using a straightforward binary format. There's no obvious best choice.
Wladimir
Sure I know that x509 is international standard. And that HTTPS uses TLS. This is not my point, my point is that when we use HTTPS the developer delegates certificates verification to the plateform he is running on, so developer don't have to bother about it, making the implementation safer and easier.
On the other hand, if you charge the developer (and not the plateform) to check certificate validity, it means that you have to develop a different codebase for all plateform you are targeting, because each plateform store trusted root certificate in a different manner with different APIs, and also have different types representing a X509 Certificate.
So, let's say I want to target IOS + WP + Android + WinRT + desktop win, I need to develop 4 times chain verification and certificate parsing. (Because I can't verify a certificate if it is not in the specific type of the underlying plateform)
And since it would take too much time to do that, I end up delegating parsing and trust verification to a third party service.
2015-01-28 14:32 GMT+01:00 Wladimir <laanwj@gmail•com>:
Quoted text (22 lines)
> > On Wed, 28 Jan 2015, Nicolas DORIER wrote: > > I agree that the use protocol buffer and x509 by BIP70 is a poor choice. >> > > Well x509 is an international standard in common use, you can't do much > better with regard to portability. Your suggestion about HTTPS makes little > sense, you do know what TLS uses x509 internally as well? > > Re: protocol buffers, I don't know if it's the best possible one, but one > serialization method had to be picked. If it weren't, we could still have > still been discussing which one to use by now. Just like for JSON there are > bindings for many languages. > > Though JSON parsers are much more diverse, which people using Bitcoin > Core's RPC have bumped into e.g. some have some problems handling large > numbers. Something you wouldn't expect using a straightforward binary > format. There's no obvious best choice. > > Wladimir >
Quoted text (7 lines)
> > On the other hand, if you charge the developer (and not the plateform) to > check certificate validity, it means that you have to develop a different > codebase for all plateform you are targeting, because each plateform store > trusted root certificate in a different manner with different APIs, and > also have different types representing a X509 Certificate. >
That's what cross-platform abstraction libraries are for. Both Java and Qt provide a key store library that can load from either the OS root store or a custom one. If your chosen app platform doesn't, OK, then you'll have to make or find one yourself. Perhaps contribute it upstream or make it a library. But that's not a limitation of BIP70.
Just as a reminder, there is no obligation to use the OS root store. You can (and quite possibly should) take a snapshot of the Mozilla/Apple/MSFT etc stores and load it in your app. We do this in bitcoinj by default to avoid cases where BIP70 requests work on some platforms and not others, although the developer can easily override this and use the OS root store instead.
Of all possible solutions, using a third party service to convert things to JSON is one of the least obvious and highest effort. I don't know anyone else who arrived at such a conclusion and respectfully disagree that this is a problem with the design choices in BIP70. It sounds like a bizarre hack around lack of features in whatever runtime you're using.
Not to mention the tiresome and error-prone task of writing your own JSON-to-schema marshalling code -- or something equivalent to the protobufs compiler and libs for JSON.
protobufs -- and its modern competitors such as msgpack -- natively provide type support in a way that must be hacked into JSON or XML.
The protobuf/msgpack design is engineered to avoid bugs routinely found in JSON parsing code; due to the amount of code & effort involved in JSON input sanity checking, bugs and inconsistencies inevitable arise. We have seen this in bitcoind with JSON-RPC.
On Wed, Jan 28, 2015 at 10:42 AM, Mike Hearn <mike@plan99•net> wrote:
Quoted text (41 lines)
> On the other hand, if you charge the developer (and not the plateform) to >> check certificate validity, it means that you have to develop a different >> codebase for all plateform you are targeting, because each plateform store >> trusted root certificate in a different manner with different APIs, and >> also have different types representing a X509 Certificate. >> > > That's what cross-platform abstraction libraries are for. Both Java and Qt > provide a key store library that can load from either the OS root store or > a custom one. If your chosen app platform doesn't, OK, then you'll have to > make or find one yourself. Perhaps contribute it upstream or make it a > library. But that's not a limitation of BIP70. > > Just as a reminder, there is no obligation to use the OS root store. You > can (and quite possibly should) take a snapshot of the Mozilla/Apple/MSFT > etc stores and load it in your app. We do this in bitcoinj by default to > avoid cases where BIP70 requests work on some platforms and not others, > although the developer can easily override this and use the OS root store > instead. > > Of all possible solutions, using a third party service to convert things > to JSON is one of the least obvious and highest effort. I don't know anyone > else who arrived at such a conclusion and respectfully disagree that this > is a problem with the design choices in BIP70. It sounds like a bizarre > hack around lack of features in whatever runtime you're using. > > > > ------------------------------------------------------------------------------ > Dive into the World of Parallel Programming. The Go Parallel Website, > sponsored by Intel and developed in partnership with Slashdot Media, is > your > hub for all things parallel software development, from weekly thought > leadership blogs to news, videos, case studies, tutorials and more. Take a > look and join the conversation now. http://goparallel.sourceforge.net/ > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > >
On 28-01-15 16:42, Mike Hearn wrote:
Quoted text (7 lines)
> Just as a reminder, there is no obligation to use the OS root > store. You can (and quite possibly should) take a snapshot of the > Mozilla/Apple/MSFT etc stores and load it in your app. We do this > in bitcoinj by default to avoid cases where BIP70 requests work on > some platforms and not others, although the developer can easily > override this and use the OS root store instead. >
Except that Mozilla/Apple/MSFT will update these certificate stores - second their policies - and your snapshot/collection might get outdated at a different pace than the OS-provided certificates, depending on how you (or the package maintainer) are rolling out updates.
I am not saying that OS-provided certificate stores are a holy grail, as they comes with their quirks&headaches (for example try to install your CA certificate on Mac), but generally I consider shipping your own snapshot a bad practice, as it makes the system less secure by default for the casual user and harder to audit for the (eventual) admins.
If you are a developer, that's a whole different story.
- --
Giuseppe Mazzotta
/- Bitonic _/My point is not that there is a limitation in BIP70. My point is that you put the burden of certificate verification on developer's shoulder when we can just leverage built in HTTPS support of the platform. This make cross plateform dev a nightmare.
Sure I can use a snapshot of moz/apple/msft store. I depends on BouncyCastle, as bitcoinj, so I theorically can use that way. However, if you want to use your plateform's store, then you are toasted, and the code for converting from BC X509 Certificate to one of each plateform is not obvious and is a headache. Thing that could be just left to the HTTPS support of your plateform.
Have you tried to do that on windows RT and IOS ? I tried, and I quickly stopped doing that since it is not worth the effort. (Frankly I am not even sure you can on win rt, since the API is a stripped down version of windows)
Why have you not heard about the problem ? (until now, because I have this problem because I need to have the same codebase on winrt/win/android/ios/tablets) Because bitcoinj just rely either java's own abstraction of certificate or on BC one. But I highly doubt they are using the plateform store, and even if you theorically can, dealing with X509 is very prone to error... for something that the plateform should just do for you. Also, you bundle mozilla's store in bitcoinj, what happen when the store change and your customer have not intent to use bitcoinj new version ? by leveraging the plateform you benefit from automatic updates. Also, does java stores deals with certificate revocations ? sure you can theorically code that too... or just let the plateform deals with it.
BIP70 does not limit to anything but it is a gigantic pain in the ass for easy cross development because of protobuff and embedded certificates. BIP70 is a client side technology, not a performance and storage critical data structure.
The only valid point of having embedded certificates is to allow the owner of the website to be different from the merchant. But since merchants often have their own website, a protocol without having to reinvent x509 would have been better suited to current needs.
2015-01-28 16:42 GMT+01:00 Mike Hearn <mike@plan99•net>:
Quoted text (27 lines)
> On the other hand, if you charge the developer (and not the plateform) to >> check certificate validity, it means that you have to develop a different >> codebase for all plateform you are targeting, because each plateform store >> trusted root certificate in a different manner with different APIs, and >> also have different types representing a X509 Certificate. >> > > That's what cross-platform abstraction libraries are for. Both Java and Qt > provide a key store library that can load from either the OS root store or > a custom one. If your chosen app platform doesn't, OK, then you'll have to > make or find one yourself. Perhaps contribute it upstream or make it a > library. But that's not a limitation of BIP70. > > Just as a reminder, there is no obligation to use the OS root store. You > can (and quite possibly should) take a snapshot of the Mozilla/Apple/MSFT > etc stores and load it in your app. We do this in bitcoinj by default to > avoid cases where BIP70 requests work on some platforms and not others, > although the developer can easily override this and use the OS root store > instead. > > Of all possible solutions, using a third party service to convert things > to JSON is one of the least obvious and highest effort. I don't know anyone > else who arrived at such a conclusion and respectfully disagree that this > is a problem with the design choices in BIP70. It sounds like a bizarre > hack around lack of features in whatever runtime you're using. > >
On Wednesday, 28 January 2015, at 5:19 pm, Giuseppe Mazzotta wrote:
Quoted text (12 lines)
> On 28-01-15 16:42, Mike Hearn wrote: > > Just as a reminder, there is no obligation to use the OS root > > store. You can (and quite possibly should) take a snapshot of the > > Mozilla/Apple/MSFT etc stores and load it in your app. We do this > > in bitcoinj by default to avoid cases where BIP70 requests work on > > some platforms and not others, although the developer can easily > > override this and use the OS root store instead. > > > Except that Mozilla/Apple/MSFT will update these certificate stores - > second their policies - and your snapshot/collection might get > outdated at a different pace than the OS-provided certificates, > depending on how you (or the package maintainer) are rolling out updates.
I'm frankly _horrified_ to learn that BitcoinJ ships its own root CA certificates bundle. This means that, if a root CA gets breached and a certificate gets revoked, all BitcoinJ-using software will be vulnerable until BitcoinJ ships an update *and* the software in question pulls in the new BitcoinJ update and releases its own update. That might never happen.
For the number of field there is in the spec, I don't consider having a JSON to schama really worthwhile. If you fear it is error prone, then we should provide some testing data for the BIP70. (Which I already did for protobuf, but was rejected, because deemed no useful thanks to the code generator... But such code generator gave me inconsistencies with gavin's implementation for example)
Why do you think type support is very useful in our case ? we have 3 types, and dealing only with bytes, int, and string. It cost me more time to find a suitable cross plateform lib for protobuf (in c#, that works in ios and winrt) than I would by just coding the json wrapper classes by hand. (JSON libs are more wildspread and supported than protobuf)
2015-01-28 17:04 GMT+01:00 Jeff Garzik <jgarzik@bitpay•com>:
Quoted text (64 lines)
> Not to mention the tiresome and error-prone task of writing your own > JSON-to-schema marshalling code -- or something equivalent to the protobufs > compiler and libs for JSON. > > protobufs -- and its modern competitors such as msgpack -- natively > provide type support in a way that must be hacked into JSON or XML. > > The protobuf/msgpack design is engineered to avoid bugs routinely found in > JSON parsing code; due to the amount of code & effort involved in JSON > input sanity checking, bugs and inconsistencies inevitable arise. We have > seen this in bitcoind with JSON-RPC. > > > > On Wed, Jan 28, 2015 at 10:42 AM, Mike Hearn <mike@plan99•net> wrote: > >> On the other hand, if you charge the developer (and not the plateform) to >>> check certificate validity, it means that you have to develop a different >>> codebase for all plateform you are targeting, because each plateform store >>> trusted root certificate in a different manner with different APIs, and >>> also have different types representing a X509 Certificate. >>> >> >> That's what cross-platform abstraction libraries are for. Both Java and >> Qt provide a key store library that can load from either the OS root store >> or a custom one. If your chosen app platform doesn't, OK, then you'll have >> to make or find one yourself. Perhaps contribute it upstream or make it a >> library. But that's not a limitation of BIP70. >> >> Just as a reminder, there is no obligation to use the OS root store. You >> can (and quite possibly should) take a snapshot of the Mozilla/Apple/MSFT >> etc stores and load it in your app. We do this in bitcoinj by default to >> avoid cases where BIP70 requests work on some platforms and not others, >> although the developer can easily override this and use the OS root store >> instead. >> >> Of all possible solutions, using a third party service to convert things >> to JSON is one of the least obvious and highest effort. I don't know anyone >> else who arrived at such a conclusion and respectfully disagree that this >> is a problem with the design choices in BIP70. It sounds like a bizarre >> hack around lack of features in whatever runtime you're using. >> >> >> >> ------------------------------------------------------------------------------ >> Dive into the World of Parallel Programming. The Go Parallel Website, >> sponsored by Intel and developed in partnership with Slashdot Media, is >> your >> hub for all things parallel software development, from weekly thought >> leadership blogs to news, videos, case studies, tutorials and more. Take a >> look and join the conversation now. http://goparallel.sourceforge.net/ >> _______________________________________________ >> Bitcoin-development mailing list >> Bitcoin-development@lists•sourceforge•net >> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >> >> > > > -- > Jeff Garzik > Bitcoin core developer and open source evangelist > BitPay, Inc. https://bitpay.com/ >
Quoted text (5 lines)
> > My point is not that there is a limitation in BIP70. My point is that you > put the burden of certificate verification on developer's shoulder when we > can just leverage built in HTTPS support of the platform. >
Platforms that support HTTPS but not certificate handling are rare - I know HTML5 is such a platform but such apps are inherently dependent on the server anyway and the server can just do the parsing and validation work itself. If WinRT is such a platform, OK, too bad.
The embedding of the certificates is not arbitrary or pointless, by the way. It's there for a very good reason - it makes the signed payment request verifiable by third parties. Effectively you can store the signed message and present it later to someone else, it's undeniable. Combined with the transactions and merkle branches linking them to the block chain, what you have is a form of digital receipt ... a proof of purchase that can be automatically verified as legitimate. This has all kinds of use cases.
Because of how HTTPS works, you can't easily prove to a third party that a server gave you a piece of data. Doing so requires staggeringly complex hacks (see tls notary) and when we designed BIP70, those hacks didn't even exist. So we'd lose the benefit of having a digitally signed request.
Additionally, doing things this way means BIP70 requests can be signed by things which are not HTTPS servers. For example you can sign with an email address cert, an EV certificate i.e. a company, a certificate issued by some user forum, whatever else we end up wanting. Not every payment recipient can be identified by a domain name + dynamic session.
Quoted text (2 lines)
> However, if you want to use your plateform's store, then you are toasted >
That's a bit melodramatic. BitcoinJ is able to use the Android, JRE, Windows and Mac certificate stores all using the same code or very minor variants on it (e.g. on Mac you have to specify you want the system store but it's a one-liner).
Yes, that's not *every* platform. Some will require custom binding glue and it depends what abstractions and languages you are using.
Quoted text (4 lines)
> Have you tried to do that on windows RT and IOS ? I tried, and I quickly > stopped doing that since it is not worth the effort. (Frankly I am not even > sure you can on win rt, since the API is a stripped down version of windows) >
There is code to do iOS using the Apple APIs here:
https://github.com/voisine/breadwallet/blob/master/BreadWallet/BRPaymentProtocol.m#L391
Quoted text (4 lines)
> Why have you not heard about the problem ? (until now, because I have this > problem because I need to have the same codebase on > winrt/win/android/ios/tablets) >
WinRT is a minority platform in the extreme, and all the other platforms you mentioned have the necessary APIs. Java abstracts you from them. So I think you are encountering this problem because you desire to target WinRT and other platforms with a single codebase. That's an unusual constraint.
AFAIK the only other people who encountered this are BitPay, because they want to do everything in Javascript which doesn't really provide any major APIs.
Quoted text (4 lines)
> Also, you bundle mozilla's store in bitcoinj, what happen when the store > change and your customer have not intent to use bitcoinj new version ? by > leveraging the plateform you benefit from automatic updates. >
Yes, there are pros and cons to bundling a custom root store.
Quoted text (3 lines)
> Also, does java stores deals with certificate revocations ? sure you can > theorically code that too... or just let the plateform deals with it. >
It can do OCSP checks, yes, although I believe no wallets currently do so. A better solution would be to implement an OCSP stapling extension to BIP70 though.
Quoted text (6 lines)
> > I'm frankly _horrified_ to learn that BitcoinJ ships its own root CA > certificates bundle. This means that, if a root CA gets breached and a > certificate gets revoked, all BitcoinJ-using software will be vulnerable > until BitcoinJ ships an update *and* the software in question pulls in the > new BitcoinJ update and releases its own update. That might never happen.
If your wallet is unmaintained, you have other problems beyond (extremely rare) root CA revocations.
As far as I know the only time a CA in wide usage has been revoked entirely is DigiNotar.
One advantage of doing it this way is if, for example, a widely used piece of community infrastructure (e.g. bitcointalk, reddit, whatever) decides to become a CA, the Bitcoin community can decide to have different inclusion rules vs the OS/browser root CA programs. For example we'd probably relax the constraint to use an HSM and just ensure that the rendering of the asserted identity isn't confusible with other kinds of more strongly protected identities. For example no forum usernames like "foo.com" but rendering it in the UI as "Reddit forum user foo.com" would be OK.
Also you don't get problems due to old operating systems not including new certs.
Finally, Linux doesn't have any kind of standardised cert/keystore API. There are a few places where popular distros put certs but AFAIK they aren't standardised and there's no standard code to load them. So that's another reason why there's a built in store.
But yes, this is a debatable topic on which reasonable people can disagree. The API makes it easy to use the platform OS store for wallet devs that want to do that, and I think using the platform store on Android is the default. It's only on the desktop where we fall back to a different store.
Mike, I am not denying it is impossible to do all of that. Just that it is not a trivial stuff to do to make it works everywhere, and I think that it is not a good thing for a client side technology. BIP70 has its use, and I understand why there is case where it is good to ship the certs in the message and not depends on the transport.
But a standard that just use JSON and HTTPS, even if less flexible that BIP70, would make it easier and sufficient for today's use case.
On Wed, Jan 28, 2015 at 5:55 PM, Mike Hearn <mike@plan99•net> wrote:
Quoted text (84 lines)
> My point is not that there is a limitation in BIP70. My point is that you >> put the burden of certificate verification on developer's shoulder when we >> can just leverage built in HTTPS support of the platform. >> > > Platforms that support HTTPS but not certificate handling are rare - I > know HTML5 is such a platform but such apps are inherently dependent on the > server anyway and the server can just do the parsing and validation work > itself. If WinRT is such a platform, OK, too bad. > > The embedding of the certificates is not arbitrary or pointless, by the > way. It's there for a very good reason - it makes the signed payment > request verifiable by third parties. Effectively you can store the signed > message and present it later to someone else, it's undeniable. Combined > with the transactions and merkle branches linking them to the block chain, > what you have is a form of digital receipt ... a proof of purchase that can > be automatically verified as legitimate. This has all kinds of use cases. > > Because of how HTTPS works, you can't easily prove to a third party that a > server gave you a piece of data. Doing so requires staggeringly complex > hacks (see tls notary) and when we designed BIP70, those hacks didn't even > exist. So we'd lose the benefit of having a digitally signed request. > > Additionally, doing things this way means BIP70 requests can be signed by > things which are not HTTPS servers. For example you can sign with an email > address cert, an EV certificate i.e. a company, a certificate issued by > some user forum, whatever else we end up wanting. Not every payment > recipient can be identified by a domain name + dynamic session. > > >> However, if you want to use your plateform's store, then you are toasted >> > > That's a bit melodramatic. BitcoinJ is able to use the Android, JRE, > Windows and Mac certificate stores all using the same code or very minor > variants on it (e.g. on Mac you have to specify you want the system store > but it's a one-liner). > > Yes, that's not *every* platform. Some will require custom binding glue > and it depends what abstractions and languages you are using. > > >> Have you tried to do that on windows RT and IOS ? I tried, and I quickly >> stopped doing that since it is not worth the effort. (Frankly I am not even >> sure you can on win rt, since the API is a stripped down version of windows) >> > > There is code to do iOS using the Apple APIs here: > > > https://github.com/voisine/breadwallet/blob/master/BreadWallet/BRPaymentProtocol.m#L391 > > >> Why have you not heard about the problem ? (until now, because I have >> this problem because I need to have the same codebase on >> winrt/win/android/ios/tablets) >> > > WinRT is a minority platform in the extreme, and all the other platforms > you mentioned have the necessary APIs. Java abstracts you from them. So I > think you are encountering this problem because you desire to target WinRT > and other platforms with a single codebase. That's an unusual constraint. > > AFAIK the only other people who encountered this are BitPay, because they > want to do everything in Javascript which doesn't really provide any major > APIs. > > >> Also, you bundle mozilla's store in bitcoinj, what happen when the store >> change and your customer have not intent to use bitcoinj new version ? by >> leveraging the plateform you benefit from automatic updates. >> > > Yes, there are pros and cons to bundling a custom root store. > > >> Also, does java stores deals with certificate revocations ? sure you can >> theorically code that too... or just let the plateform deals with it. >> > > It can do OCSP checks, yes, although I believe no wallets currently do so. > A better solution would be to implement an OCSP stapling extension to BIP70 > though. >
I think we'll just have to agree to disagree on this one. I've implemented BIP70 a couple of times now and didn't find it to be difficult. I know you had odd problems with the C# protobuf implementation you were using but library bugs can happen for any kind of programming.
I forgot to mention the other reason it's done this way. One of the driving goals of BIP70 was to support the TREZOR and similar devices. For hardware wallets, it's critical to keep the amount of code they need to run as small as possible. Any bugs in the code there can cause security holes and lead to the device being hacked.
Doing it the way you suggest would mean the secure code would have to contain complex and bug-prone text parsing logic as well as a full blown HTTP and SSL stack, that requires not only X.509 handling but also lots of other stuff on top. It'd increase cost, complexity and decrease security quite a bit.
Whilst I appreciate if your platform provides a scripting-like API and nothing low level it might seem easier to use JSON+HTTPS, that isn't the case for one of the primary design targets.
On Wed, Jan 28, 2015 at 6:04 PM, Nicolas Dorier <nicolas.dorier@gmail•com> wrote:
Quoted text (97 lines)
> Mike, I am not denying it is impossible to do all of that. > Just that it is not a trivial stuff to do to make it works everywhere, and > I think that it is not a good thing for a client side technology. > BIP70 has its use, and I understand why there is case where it is good to > ship the certs in the message and not depends on the transport. > > But a standard that just use JSON and HTTPS, even if less flexible that > BIP70, would make it easier and sufficient for today's use case. > > On Wed, Jan 28, 2015 at 5:55 PM, Mike Hearn <mike@plan99•net> wrote: > >> My point is not that there is a limitation in BIP70. My point is that you >>> put the burden of certificate verification on developer's shoulder when we >>> can just leverage built in HTTPS support of the platform. >>> >> >> Platforms that support HTTPS but not certificate handling are rare - I >> know HTML5 is such a platform but such apps are inherently dependent on the >> server anyway and the server can just do the parsing and validation work >> itself. If WinRT is such a platform, OK, too bad. >> >> The embedding of the certificates is not arbitrary or pointless, by the >> way. It's there for a very good reason - it makes the signed payment >> request verifiable by third parties. Effectively you can store the signed >> message and present it later to someone else, it's undeniable. Combined >> with the transactions and merkle branches linking them to the block chain, >> what you have is a form of digital receipt ... a proof of purchase that can >> be automatically verified as legitimate. This has all kinds of use cases. >> >> Because of how HTTPS works, you can't easily prove to a third party that >> a server gave you a piece of data. Doing so requires staggeringly complex >> hacks (see tls notary) and when we designed BIP70, those hacks didn't even >> exist. So we'd lose the benefit of having a digitally signed request. >> >> Additionally, doing things this way means BIP70 requests can be signed by >> things which are not HTTPS servers. For example you can sign with an email >> address cert, an EV certificate i.e. a company, a certificate issued by >> some user forum, whatever else we end up wanting. Not every payment >> recipient can be identified by a domain name + dynamic session. >> >> >>> However, if you want to use your plateform's store, then you are toasted >>> >> >> That's a bit melodramatic. BitcoinJ is able to use the Android, JRE, >> Windows and Mac certificate stores all using the same code or very minor >> variants on it (e.g. on Mac you have to specify you want the system store >> but it's a one-liner). >> >> Yes, that's not *every* platform. Some will require custom binding glue >> and it depends what abstractions and languages you are using. >> >> >>> Have you tried to do that on windows RT and IOS ? I tried, and I quickly >>> stopped doing that since it is not worth the effort. (Frankly I am not even >>> sure you can on win rt, since the API is a stripped down version of windows) >>> >> >> There is code to do iOS using the Apple APIs here: >> >> >> https://github.com/voisine/breadwallet/blob/master/BreadWallet/BRPaymentProtocol.m#L391 >> >> >>> Why have you not heard about the problem ? (until now, because I have >>> this problem because I need to have the same codebase on >>> winrt/win/android/ios/tablets) >>> >> >> WinRT is a minority platform in the extreme, and all the other platforms >> you mentioned have the necessary APIs. Java abstracts you from them. So I >> think you are encountering this problem because you desire to target WinRT >> and other platforms with a single codebase. That's an unusual constraint. >> >> AFAIK the only other people who encountered this are BitPay, because they >> want to do everything in Javascript which doesn't really provide any major >> APIs. >> >> >>> Also, you bundle mozilla's store in bitcoinj, what happen when the store >>> change and your customer have not intent to use bitcoinj new version ? by >>> leveraging the plateform you benefit from automatic updates. >>> >> >> Yes, there are pros and cons to bundling a custom root store. >> >> >>> Also, does java stores deals with certificate revocations ? sure you can >>> theorically code that too... or just let the plateform deals with it. >>> >> >> It can do OCSP checks, yes, although I believe no wallets currently do >> so. A better solution would be to implement an OCSP stapling extension to >> BIP70 though. >> > >
why not allow both serializations and keep serialization format a parameter, keep everyone happy.
On Wed, Jan 28, 2015 at 12:14 PM, Mike Hearn <mike@plan99•net> wrote:
Quoted text (139 lines)
> I think we'll just have to agree to disagree on this one. I've implemented > BIP70 a couple of times now and didn't find it to be difficult. I know you > had odd problems with the C# protobuf implementation you were using but > library bugs can happen for any kind of programming. > > I forgot to mention the other reason it's done this way. One of the > driving goals of BIP70 was to support the TREZOR and similar devices. For > hardware wallets, it's critical to keep the amount of code they need to run > as small as possible. Any bugs in the code there can cause security holes > and lead to the device being hacked. > > Doing it the way you suggest would mean the secure code would have to > contain complex and bug-prone text parsing logic as well as a full blown > HTTP and SSL stack, that requires not only X.509 handling but also lots of > other stuff on top. It'd increase cost, complexity and decrease security > quite a bit. > > Whilst I appreciate if your platform provides a scripting-like API and > nothing low level it might seem easier to use JSON+HTTPS, that isn't the > case for one of the primary design targets. > > > > On Wed, Jan 28, 2015 at 6:04 PM, Nicolas Dorier <nicolas.dorier@gmail•com> > wrote: > >> Mike, I am not denying it is impossible to do all of that. >> Just that it is not a trivial stuff to do to make it works everywhere, >> and I think that it is not a good thing for a client side technology. >> BIP70 has its use, and I understand why there is case where it is good to >> ship the certs in the message and not depends on the transport. >> >> But a standard that just use JSON and HTTPS, even if less flexible that >> BIP70, would make it easier and sufficient for today's use case. >> >> On Wed, Jan 28, 2015 at 5:55 PM, Mike Hearn <mike@plan99•net> wrote: >> >>> My point is not that there is a limitation in BIP70. My point is that >>>> you put the burden of certificate verification on developer's shoulder when >>>> we can just leverage built in HTTPS support of the platform. >>>> >>> >>> Platforms that support HTTPS but not certificate handling are rare - I >>> know HTML5 is such a platform but such apps are inherently dependent on the >>> server anyway and the server can just do the parsing and validation work >>> itself. If WinRT is such a platform, OK, too bad. >>> >>> The embedding of the certificates is not arbitrary or pointless, by the >>> way. It's there for a very good reason - it makes the signed payment >>> request verifiable by third parties. Effectively you can store the signed >>> message and present it later to someone else, it's undeniable. Combined >>> with the transactions and merkle branches linking them to the block chain, >>> what you have is a form of digital receipt ... a proof of purchase that can >>> be automatically verified as legitimate. This has all kinds of use cases. >>> >>> Because of how HTTPS works, you can't easily prove to a third party that >>> a server gave you a piece of data. Doing so requires staggeringly complex >>> hacks (see tls notary) and when we designed BIP70, those hacks didn't even >>> exist. So we'd lose the benefit of having a digitally signed request. >>> >>> Additionally, doing things this way means BIP70 requests can be signed >>> by things which are not HTTPS servers. For example you can sign with an >>> email address cert, an EV certificate i.e. a company, a certificate issued >>> by some user forum, whatever else we end up wanting. Not every payment >>> recipient can be identified by a domain name + dynamic session. >>> >>> >>>> However, if you want to use your plateform's store, then you are toasted >>>> >>> >>> That's a bit melodramatic. BitcoinJ is able to use the Android, JRE, >>> Windows and Mac certificate stores all using the same code or very minor >>> variants on it (e.g. on Mac you have to specify you want the system store >>> but it's a one-liner). >>> >>> Yes, that's not *every* platform. Some will require custom binding glue >>> and it depends what abstractions and languages you are using. >>> >>> >>>> Have you tried to do that on windows RT and IOS ? I tried, and I >>>> quickly stopped doing that since it is not worth the effort. (Frankly I am >>>> not even sure you can on win rt, since the API is a stripped down version >>>> of windows) >>>> >>> >>> There is code to do iOS using the Apple APIs here: >>> >>> >>> https://github.com/voisine/breadwallet/blob/master/BreadWallet/BRPaymentProtocol.m#L391 >>> >>> >>>> Why have you not heard about the problem ? (until now, because I have >>>> this problem because I need to have the same codebase on >>>> winrt/win/android/ios/tablets) >>>> >>> >>> WinRT is a minority platform in the extreme, and all the other platforms >>> you mentioned have the necessary APIs. Java abstracts you from them. So I >>> think you are encountering this problem because you desire to target WinRT >>> and other platforms with a single codebase. That's an unusual constraint. >>> >>> AFAIK the only other people who encountered this are BitPay, because >>> they want to do everything in Javascript which doesn't really provide any >>> major APIs. >>> >>> >>>> Also, you bundle mozilla's store in bitcoinj, what happen when the >>>> store change and your customer have not intent to use bitcoinj new version >>>> ? by leveraging the plateform you benefit from automatic updates. >>>> >>> >>> Yes, there are pros and cons to bundling a custom root store. >>> >>> >>>> Also, does java stores deals with certificate revocations ? sure you >>>> can theorically code that too... or just let the plateform deals with it. >>>> >>> >>> It can do OCSP checks, yes, although I believe no wallets currently do >>> so. A better solution would be to implement an OCSP stapling extension to >>> BIP70 though. >>> >> >> > > > ------------------------------------------------------------------------------ > Dive into the World of Parallel Programming. The Go Parallel Website, > sponsored by Intel and developed in partnership with Slashdot Media, is > your > hub for all things parallel software development, from weekly thought > leadership blogs to news, videos, case studies, tutorials and more. Take a > look and join the conversation now. http://goparallel.sourceforge.net/ > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > >
Sure,
But the mobile targets, it is still easier to use Json + HTTPS, especially when you want one code base for everything. And as you said, developers need to think about fetching mozilla store time to time, and check revocations themselves. This is not obvious thing to do, and hard to test correctly.
If your use case was the primary utility of BIP70, then I'd say it fit the bill. But for cross plateform client development an atlernative would be easier.
Quoted text (1 line)
> why not allow both serializations and keep serialization format a
parameter, keep everyone happy.
It would be another BIP, because if we use JSON with HTTPS, the difference is also in the semantic (no embedded certificates)
I will likely provide this option for a product I am developing. I will only use another Content Type. We'll see then how it goes.
2015-01-28 18:14 GMT+01:00 Mike Hearn <mike@plan99•net>:
Quoted text (125 lines)
> I think we'll just have to agree to disagree on this one. I've implemented > BIP70 a couple of times now and didn't find it to be difficult. I know you > had odd problems with the C# protobuf implementation you were using but > library bugs can happen for any kind of programming. > > I forgot to mention the other reason it's done this way. One of the > driving goals of BIP70 was to support the TREZOR and similar devices. For > hardware wallets, it's critical to keep the amount of code they need to run > as small as possible. Any bugs in the code there can cause security holes > and lead to the device being hacked. > > Doing it the way you suggest would mean the secure code would have to > contain complex and bug-prone text parsing logic as well as a full blown > HTTP and SSL stack, that requires not only X.509 handling but also lots of > other stuff on top. It'd increase cost, complexity and decrease security > quite a bit. > > Whilst I appreciate if your platform provides a scripting-like API and > nothing low level it might seem easier to use JSON+HTTPS, that isn't the > case for one of the primary design targets. > > > > On Wed, Jan 28, 2015 at 6:04 PM, Nicolas Dorier <nicolas.dorier@gmail•com> > wrote: > >> Mike, I am not denying it is impossible to do all of that. >> Just that it is not a trivial stuff to do to make it works everywhere, >> and I think that it is not a good thing for a client side technology. >> BIP70 has its use, and I understand why there is case where it is good to >> ship the certs in the message and not depends on the transport. >> >> But a standard that just use JSON and HTTPS, even if less flexible that >> BIP70, would make it easier and sufficient for today's use case. >> >> On Wed, Jan 28, 2015 at 5:55 PM, Mike Hearn <mike@plan99•net> wrote: >> >>> My point is not that there is a limitation in BIP70. My point is that >>>> you put the burden of certificate verification on developer's shoulder when >>>> we can just leverage built in HTTPS support of the platform. >>>> >>> >>> Platforms that support HTTPS but not certificate handling are rare - I >>> know HTML5 is such a platform but such apps are inherently dependent on the >>> server anyway and the server can just do the parsing and validation work >>> itself. If WinRT is such a platform, OK, too bad. >>> >>> The embedding of the certificates is not arbitrary or pointless, by the >>> way. It's there for a very good reason - it makes the signed payment >>> request verifiable by third parties. Effectively you can store the signed >>> message and present it later to someone else, it's undeniable. Combined >>> with the transactions and merkle branches linking them to the block chain, >>> what you have is a form of digital receipt ... a proof of purchase that can >>> be automatically verified as legitimate. This has all kinds of use cases. >>> >>> Because of how HTTPS works, you can't easily prove to a third party that >>> a server gave you a piece of data. Doing so requires staggeringly complex >>> hacks (see tls notary) and when we designed BIP70, those hacks didn't even >>> exist. So we'd lose the benefit of having a digitally signed request. >>> >>> Additionally, doing things this way means BIP70 requests can be signed >>> by things which are not HTTPS servers. For example you can sign with an >>> email address cert, an EV certificate i.e. a company, a certificate issued >>> by some user forum, whatever else we end up wanting. Not every payment >>> recipient can be identified by a domain name + dynamic session. >>> >>> >>>> However, if you want to use your plateform's store, then you are toasted >>>> >>> >>> That's a bit melodramatic. BitcoinJ is able to use the Android, JRE, >>> Windows and Mac certificate stores all using the same code or very minor >>> variants on it (e.g. on Mac you have to specify you want the system store >>> but it's a one-liner). >>> >>> Yes, that's not *every* platform. Some will require custom binding glue >>> and it depends what abstractions and languages you are using. >>> >>> >>>> Have you tried to do that on windows RT and IOS ? I tried, and I >>>> quickly stopped doing that since it is not worth the effort. (Frankly I am >>>> not even sure you can on win rt, since the API is a stripped down version >>>> of windows) >>>> >>> >>> There is code to do iOS using the Apple APIs here: >>> >>> >>> https://github.com/voisine/breadwallet/blob/master/BreadWallet/BRPaymentProtocol.m#L391 >>> >>> >>>> Why have you not heard about the problem ? (until now, because I have >>>> this problem because I need to have the same codebase on >>>> winrt/win/android/ios/tablets) >>>> >>> >>> WinRT is a minority platform in the extreme, and all the other platforms >>> you mentioned have the necessary APIs. Java abstracts you from them. So I >>> think you are encountering this problem because you desire to target WinRT >>> and other platforms with a single codebase. That's an unusual constraint. >>> >>> AFAIK the only other people who encountered this are BitPay, because >>> they want to do everything in Javascript which doesn't really provide any >>> major APIs. >>> >>> >>>> Also, you bundle mozilla's store in bitcoinj, what happen when the >>>> store change and your customer have not intent to use bitcoinj new version >>>> ? by leveraging the plateform you benefit from automatic updates. >>>> >>> >>> Yes, there are pros and cons to bundling a custom root store. >>> >>> >>>> Also, does java stores deals with certificate revocations ? sure you >>>> can theorically code that too... or just let the plateform deals with it. >>>> >>> >>> It can do OCSP checks, yes, although I believe no wallets currently do >>> so. A better solution would be to implement an OCSP stapling extension to >>> BIP70 though. >>> >> >> >
It is not "fear", it is field experience.
JSON has proven to be a bug generator for the reasons already stated.
JSON does not include type marshalling and input validation. Protobufs/msgpack/etc. engineered those to occur automatically, because that is an area shown by field experience to be a constant source of bugs and inconsistent parsing/validation behavior.
On Wed, Jan 28, 2015 at 11:52 AM, Nicolas DORIER <nicolas.dorier@gmail•com> wrote:
Quoted text (84 lines)
> For the number of field there is in the spec, I don't consider having a > JSON to schama really worthwhile. > If you fear it is error prone, then we should provide some testing data > for the BIP70. (Which I already did for protobuf, but was rejected, because > deemed no useful thanks to the code generator... But such code generator > gave me inconsistencies with gavin's implementation for example) > > Why do you think type support is very useful in our case ? we have 3 > types, and dealing only with bytes, int, and string. > It cost me more time to find a suitable cross plateform lib for protobuf > (in c#, that works in ios and winrt) than I would by just coding the json > wrapper classes by hand. (JSON libs are more wildspread and supported than > protobuf) > > 2015-01-28 17:04 GMT+01:00 Jeff Garzik <jgarzik@bitpay•com>: > >> Not to mention the tiresome and error-prone task of writing your own >> JSON-to-schema marshalling code -- or something equivalent to the protobufs >> compiler and libs for JSON. >> >> protobufs -- and its modern competitors such as msgpack -- natively >> provide type support in a way that must be hacked into JSON or XML. >> >> The protobuf/msgpack design is engineered to avoid bugs routinely found >> in JSON parsing code; due to the amount of code & effort involved in JSON >> input sanity checking, bugs and inconsistencies inevitable arise. We have >> seen this in bitcoind with JSON-RPC. >> >> >> >> On Wed, Jan 28, 2015 at 10:42 AM, Mike Hearn <mike@plan99•net> wrote: >> >>> On the other hand, if you charge the developer (and not the plateform) >>>> to check certificate validity, it means that you have to develop a >>>> different codebase for all plateform you are targeting, because each >>>> plateform store trusted root certificate in a different manner with >>>> different APIs, and also have different types representing a X509 >>>> Certificate. >>>> >>> >>> That's what cross-platform abstraction libraries are for. Both Java and >>> Qt provide a key store library that can load from either the OS root store >>> or a custom one. If your chosen app platform doesn't, OK, then you'll have >>> to make or find one yourself. Perhaps contribute it upstream or make it a >>> library. But that's not a limitation of BIP70. >>> >>> Just as a reminder, there is no obligation to use the OS root store. You >>> can (and quite possibly should) take a snapshot of the Mozilla/Apple/MSFT >>> etc stores and load it in your app. We do this in bitcoinj by default to >>> avoid cases where BIP70 requests work on some platforms and not others, >>> although the developer can easily override this and use the OS root store >>> instead. >>> >>> Of all possible solutions, using a third party service to convert things >>> to JSON is one of the least obvious and highest effort. I don't know anyone >>> else who arrived at such a conclusion and respectfully disagree that this >>> is a problem with the design choices in BIP70. It sounds like a bizarre >>> hack around lack of features in whatever runtime you're using. >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> Dive into the World of Parallel Programming. The Go Parallel Website, >>> sponsored by Intel and developed in partnership with Slashdot Media, is >>> your >>> hub for all things parallel software development, from weekly thought >>> leadership blogs to news, videos, case studies, tutorials and more. Take >>> a >>> look and join the conversation now. http://goparallel.sourceforge.net/ >>> _______________________________________________ >>> Bitcoin-development mailing list >>> Bitcoin-development@lists•sourceforge•net >>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >>> >>> >> >> >> -- >> Jeff Garzik >> Bitcoin core developer and open source evangelist >> BitPay, Inc. https://bitpay.com/ >> > >
Quoted text (5 lines)
> > It is not "fear", it is field experience. > > JSON has proven to be a bug generator for the reasons already stated. >
To back Jeff up on this point, today we see this story:
http://www.theregister.co.uk/2015/01/27/trivial_hole_left_black_phones_open_to_plunder/
The maker of BlackPhone – a mobile marketed as offering unusually high levels of security – has patched *a critical vulnerability that allows hackers to run malicious code on the handsets*. Attackers need little more than a phone number to send a message that can compromise the devices via the Silent Text application.
"The SCIMP protocol encodes messages as JSON objects, which are then transmitted to the remote party over XMPP," Dowd explained to *The Register*. "*The flaw I discovered occurs during the deserialization of these JSON objects*. It is *a type confusion vulnerability*, which when exploited allows an attacker to overwrite a pointer in memory, either partially or in full. This pointer is later manipulated by the program and also the system allocator, allowing you to do things such as pass arbitrary pointers to free()."
The C++/Java/Python protocol buffer implementations are used by Google for all internal inter-server communication. Any similar exploit in them would result in total bypass of their entire internal security and auditing system by allowing you to run code as any user. The Google security team is very good, the protobuf code is carefully reviewed and the format is relatively constrained. The chances of there being any security problems in the parsing code generated by the protobuf compilers is drastically smaller. As BIP70 requests are parsed by security sensitive code, this matters.
The vision for BIP70 has always been to be a foundation for many features. We haven't really done much with it so far because there have always been higher priorities. But I hope that if Bitcoin continues to be successful and grows, one day payment requests will have many different features in them and those will likely include many complex data structures.
Quoted text (6 lines)
> That was essentially what we did in the end, we replaced the network
> identifier ("main"/"test") with the genesis block hash. The result is
> never going to accidentally work with Bitcoin Core (nor vice-versa), but
> is readily extensible to any other altcoins that want to use the
> specification without requiring any sort of central registry.
>Interesting approach, and I also think that requiring a central registry would be potentially harmful.
However, I think it might not be adequate to think of the network identifier as being congruent with the genesis block hash. In the theoretical case of the blockchain being continued on two forked chains (with two communities which prefer one of the chains each), clients would not be prevented from interpreting messages on the wrong chain.
Best regards,
Isidor
That case is very unlikely IMO, but still you can solve it while keeping hash of the genesis block as the chain id. If a community decides to accept a forking chain with new rules from block N (let's call it bitcoinB), the original chain can maintain the original genesis block and the new community can define N (which is not accepted by bitcoin due to the new rules) as the genesis block for bitcoinB for the purposes of chain ID. As said forking bitcoins and bitcoinsB with the same owners doesn't make much sense to me. If you're creating a new currency you can just as well define a new chain. If you want to start an initial utxo giving the new coins to bitcoin holders...I still don't see the point, but you can also do that in a new chain.
In summary, your example is not a good reason not to adopt a hash of the genesis block as chain ID. On Mar 14, 2015 5:22 PM, "Isidor Zeuner" <cryptocurrencies@quidecco•de> wrote:
Quoted text (36 lines)
> > That was essentially what we did in the end, we replaced the network
> > identifier ("main"/"test") with the genesis block hash. The result is
> > never going to accidentally work with Bitcoin Core (nor vice-versa), but
> > is readily extensible to any other altcoins that want to use the
> > specification without requiring any sort of central registry.
> >
>
> Interesting approach, and I also think that requiring a central
> registry would be potentially harmful.
>
> However, I think it might not be adequate to think of the network
> identifier as being congruent with the genesis block hash. In the
> theoretical case of the blockchain being continued on two forked
> chains (with two communities which prefer one of the chains each),
> clients would not be prevented from interpreting messages on the wrong
> chain.
>
> Best regards,
>
> Isidor
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website,
> sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for
> all
> things parallel software development, from weekly thought leadership blogs
> to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>