New "sendheaders" p2p message
6 messages, 24 Sep 2015. Started by Suhas Daftuar.
Suhas Daftuar (3), Peter Todd (2), Tier Nolan
Hi,
I'm proposing the addition of a new, optional p2p message to help improve the way blocks are announced on the network. The draft BIP is available here and pasted below: https://gist.github.com/sdaftuar/465bf008f0a4768c0def
The goal of this p2p message is to facilitate nodes being able to optionally announce blocks with headers messages rather than with inv's, which is particularly beneficial since the introduction of headers-first download in Bitcoin Core 0.10. In particular, this allows for more efficient propagation of reorgs as it would eliminate a round trip in network communication.
The implementation of this BIP (which includes code to directly fetch blocks based on announced headers) is in https://github.com/bitcoin/bitcoin/pull/6494. For additional background, please also see https://github.com/bitcoin/bitcoin/issues/5982.
Note that this new p2p message is optional; nodes can feel free to ignore and continue to use inv messages to announce new blocks.
Thanks to Pieter Wuille for suggesting this idea.
Draft BIP text:
<pre> BIP: <unassigned> Title: sendheaders message Author: Suhas Daftuar <sdaftuar@chaincode•com> Status: Draft Type: Standards Track Created: 2015-05-08 </pre>
==Abstract==
Add a new message, "sendheaders", which indicates that a node prefers to receive new block announcements via a "headers" message rather than an "inv".
==Motivation==
Since the introduction of "headers-first" downloading of blocks in 0.10, blocks will not be processed unless they are able to connect to a (valid) headers chain. Consequently, block relay generally works as follows: # A node (N) announces the new tip with an "inv" message, containing the block hash # A peer (P) responds to the "inv" with a "getheaders" message (to request headers up to the new tip) and a "getdata" message for the new tip itself # N responds with a "headers" message (with the header for the new block along with any preceding headers unknown to P) and a "block" message containing the new block
However, in the case where a new block is being announced that builds on the tip, it would be generally more efficient if the node N just announced the block header for the new block, rather than just the block hash, and saved the peer from generating and transmitting the getheaders message (and the required block locator).
In the case of a reorg, where 1 or more blocks are disconnected, nodes currently just send an "inv" for the new tip. Peers currently are able to request the new tip immediately, but wait until the headers for the intermediate blocks are delivered before requesting those blocks. By announcing headers from the last fork point leading up to the new tip in the block announcement, peers are able to request all the intermediate blocks immediately.
==Specification==
# The sendheaders message is defined as an empty message where pchCommand == "sendheaders" # Upon receipt of a "sendheaders" message, the node will be permitted, but not required, to announce new blocks by sending the header of the new block (along with any other blocks that a node believes a peer might need in order for the block to connect). # Feature discovery is enabled by checking protocol version >= 70012
==Backward compatibility==
Older clients remain fully compatible and interoperable after this change.
==Implementation==
On Thu, Sep 24, 2015 at 7:02 PM, Suhas Daftuar via bitcoin-dev < bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (14 lines)
> Hi, > > I'm proposing the addition of a new, optional p2p message to help improve > the way blocks are announced on the network. The draft BIP is available > here and pasted below: > https://gist.github.com/sdaftuar/465bf008f0a4768c0def > > The goal of this p2p message is to facilitate nodes being able to > optionally announce blocks with headers messages rather than with inv's, > which is particularly beneficial since the introduction of headers-first > download in Bitcoin Core 0.10. In particular, this allows for more > efficient propagation of reorgs as it would eliminate a round trip in > network communication. >
Is there actually a requirement for the new message? New nodes could just unilaterally switch to sending headers and current nodes would be compatible.
It looks like the only DOS misbehaving penalty is if the header is invalid or if the headers don't form a chain.
On Thu, Sep 24, 2015 at 2:17 PM, Tier Nolan via bitcoin-dev < bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (5 lines)
> > Is there actually a requirement for the new message? New nodes could just > unilaterally switch to sending headers and current nodes would be > compatible. >
I don't believe that unilaterally switching to headers announcements would work for all network participants -- both for users running older Bitcoin Core versions (anything before 0.10, which I believe all ignore headers messages) and for non-Bitcoin Core software that participates on the network (which may ignore headers messages too, I'm not sure what all is out there).
Even for Bitcoin Core versions 0.10 and 0.11, which process headers and use them to determine what blocks to download, the block fetching logic is not optimized for new block announcements via headers messages. Part of what is implemented in the pull request is direct fetching of blocks upon receiving a headers message; nodes that don't implement block downloading in response to headers announcements should continue to receive inv's, I think -- hence this p2p message to opt-in to the new behavior.
On 24 September 2015 14:37:40 GMT-04:00, Suhas Daftuar via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (19 lines)
>On Thu, Sep 24, 2015 at 2:17 PM, Tier Nolan via bitcoin-dev < >bitcoin-dev@lists•linuxfoundation•org> wrote: > >> >> Is there actually a requirement for the new message? New nodes could >just >> unilaterally switch to sending headers and current nodes would be >> compatible. >> > >I don't believe that unilaterally switching to headers announcements >would >work for all network participants -- both for users running older >Bitcoin >Core versions (anything before 0.10, which I believe all ignore headers >messages) and for non-Bitcoin Core software that participates on the >network (which may ignore headers messages too, I'm not sure what all >is >out there).
You can enable the behaviour based on advertised p2p network version.
I considered that as well, but it seemed to me that other software on the network (say, different wallet implementations) might prefer the option of being able to bump up their protocol version in the future to pick up some other change, without having to also opt-in to receiving headers-announcements for blocks.
In particular, inv-based block announcements aren't going away (even in my implementation of headers announcements, there are some edge cases where the code would need to fall back to an inv announcement), so forcing all software on the network to upgrade to supporting headers announcements, whether now or in the future, seems too drastic -- I could imagine some software not being very concerned about optimizing block relay in this particular way.
On Thu, Sep 24, 2015 at 2:41 PM, Peter Todd <pete@petertodd•org> wrote:
Quoted text (41 lines)
> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > > > On 24 September 2015 14:37:40 GMT-04:00, Suhas Daftuar via bitcoin-dev < > bitcoin-dev@lists•linuxfoundation•org> wrote: > >On Thu, Sep 24, 2015 at 2:17 PM, Tier Nolan via bitcoin-dev < > >bitcoin-dev@lists•linuxfoundation•org> wrote: > > > >> > >> Is there actually a requirement for the new message? New nodes could > >just > >> unilaterally switch to sending headers and current nodes would be > >> compatible. > >> > > > >I don't believe that unilaterally switching to headers announcements > >would > >work for all network participants -- both for users running older > >Bitcoin > >Core versions (anything before 0.10, which I believe all ignore headers > >messages) and for non-Bitcoin Core software that participates on the > >network (which may ignore headers messages too, I'm not sure what all > >is > >out there). > > You can enable the behaviour based on advertised p2p network version. > -----BEGIN PGP SIGNATURE----- > > iQE9BAEBCgAnIBxQZXRlciBUb2RkIDxwZXRlQHBldGVydG9kZC5vcmc+BQJWBEO5 > AAoJEMCF8hzn9Lncz4MH/3ztGWdFvMWWcwQsjIRH+eP6PH57WaEru1smmFYOmKrj > djdiRVdxfChxRqP3adO21RUKKchjl8DNjrFJHPFz75FSM0cDcD0QAGAHilVdnICE > LEIlTEoiIc0f1z9f/EJHSHPhiUXMnjpl/l7PYJFZV3Lt2Bl30yLsNnrp9qxjR30n > 3nykZjyRad4JSavdTP6Evd3qaqwGXNUWsdObXNI+WPKlrw6hczlhFDKQ7RC1FPQU > Rbgb21pavtqLUTwbBZGUisAAc94e2Gama1p3ioUFklbVtLTdw+FtxPgV/0ZS75OR > V9pCXIbg9VM6QY4+9gYnP635+qCkqAJ4tBsYGmsT8yA= > =cF4B > -----END PGP SIGNATURE----- > >
On 24 September 2015 14:56:23 GMT-04:00, Suhas Daftuar <sdaftuar@gmail•com> wrote:
Quoted text (19 lines)
>I considered that as well, but it seemed to me that other software on >the >network (say, different wallet implementations) might prefer the option >of >being able to bump up their protocol version in the future to pick up >some >other change, without having to also opt-in to receiving >headers-announcements for blocks. > >In particular, inv-based block announcements aren't going away (even in >my >implementation of headers announcements, there are some edge cases >where >the code would need to fall back to an inv announcement), so forcing >all >software on the network to upgrade to supporting headers announcements, >whether now or in the future, seems too drastic -- I could imagine some >software not being very concerned about optimizing block relay in this >particular way.
Block headers are so small - 80 bytes - that it may be reasonable to just stop using the inv mechanism for them in favor of always sending headers. IIRC a inv is 32 bytes of digest and another four bytes or something of the inv string itself - that's already nearly half of the header.
Meanwhile reducing the amount of state in the protocol does have some value, and decreasing overall latency for headers to get around the network certainely isnt a bad thing.