A bitcoin UDP P2P protocol extension

Here is a rough draft implementation of a UDP P2P protocol extension for bitcoin:

     https://github.com/jgarzik/bitcoin/tree/udp
     http://yyz.us/bitcoin/udp-v0.patch

Protocol specification (such that it is):

- UDP, bound to same port as TCP P2P (normally 8333) - Active, simultaneous TCP P2P connection required (useful against DoS and other attacks) - Same message format as TCP P2P, same pchMessageStart conventions, etc. (my CNetMessage pull req would be helpful here) - Multiple P2P messages per UDP packet permitted - Max UDP packet size 100*1024 bytes - Advertises NODE_UDP in nServices - New "getudpcook" TCP P2P command returns a "udpcook" message, containing a yummy cookie - UDP P2P commands "inv", "tx" and "addr" are handled as if received via TCP - UDP P2P command "udpsub" sets a mask, that subscribes to one or more data broadcasts. - When USM_INV_BCAST mask bit is set, receive "inv" messages via UDP rather than TCP.

Project and design goals (or, why do this?):

- It is theorized that UDP may be useful for some messages we broadcast throughout the network - A productive discussion STARTS with code, otherwise discussion continues forever. - Investigate improving "inv" relay speed - Investigate unconditional "tx" broadcasting via UDP, as "inv" alternative, for small tx's. - Investigate improving block relay speed (or perhaps block header relay speed) - Open up new design avenues, for P2P patterns more suited to UDP than TCP in general.

Not looked at code yet, some thoughts:

1) In IPv4 max UDP data size is 65507 bytes.
2) What about big messages (block)?
3) I think relay speed can be increased only by reducing network diameter.

Introducing super-nodes with thousands of connected peers can greatly help here. 4) The only (IMO) UDP advantage: hole punching. But requirement of active

TCP connection defies it. And hole punching will need protocol extension. 5) unsolicited tx broadcast will increase traffic of nodes (especially well-connected) and less people will run them.

---- Original Message -----

From: Jeff Garzik <jgarzik@exmulti•com> To: Bitcoin Development <bitcoin-development@lists•sourceforge•net> Cc: Sent: Saturday, March 23, 2013 7:17 AM Subject: [Bitcoin-development] A bitcoin UDP P2P protocol extension

Here is a rough draft implementation of a UDP P2P protocol extension for bitcoin:

    https://github.com/jgarzik/bitcoin/tree/udp
     http://yyz.us/bitcoin/udp-v0.patch

Protocol specification (such that it is):

- UDP, bound to same port as TCP P2P (normally 8333) - Active, simultaneous TCP P2P connection required (useful against DoS and other attacks) - Same message format as TCP P2P, same pchMessageStart conventions, etc. (my CNetMessage pull req would be helpful here) - Multiple P2P messages per UDP packet permitted - Max UDP packet size 100*1024 bytes - Advertises NODE_UDP in nServices - New "getudpcook" TCP P2P command returns a "udpcook" message, containing a yummy cookie - UDP P2P commands "inv", "tx" and "addr" are handled as if received via TCP - UDP P2P command "udpsub" sets a mask, that subscribes to one or more data broadcasts. - When USM_INV_BCAST mask bit is set, receive "inv" messages via UDP rather than TCP.

Project and design goals (or, why do this?):

- It is theorized that UDP may be useful for some messages we broadcast throughout the network - A productive discussion STARTS with code, otherwise discussion continues forever. - Investigate improving "inv" relay speed - Investigate unconditional "tx" broadcasting via UDP, as "inv" alternative, for small tx's. - Investigate improving block relay speed (or perhaps block header relay speed) - Open up new design avenues, for P2P patterns more suited to UDP than TCP in general.

On Saturday, March 23, 2013 10:42:26 AM Randy Willis wrote:

> Introducing super-nodes with thousands of connected peers can greatly help
> here.

UDP is connectionless. I would hope any UDP bitcoin protocol doesn't try to emulate a connection. :/

Luke

On Sat, Mar 23, 2013 at 10:52 AM, Luke-Jr <luke@dashjr•org> wrote:

> On Saturday, March 23, 2013 10:42:26 AM Randy Willis wrote:
>> Introducing super-nodes with thousands of connected peers can greatly help
>> here.
>
> UDP is connectionless.
> I would hope any UDP bitcoin protocol doesn't try to emulate a connection. :/

It depends on the usage. Simply broadcasting a TX or INV to a remote peer does not require a connection, clearly... but you probably want to signal acceptance of those messages somehow.

But other uses, like subscribing to a broadcast, does require some notion of an association.

In the rough draft, a parallel TCP connection with version/verack sequence is required, and you may make use of it if a connection is needed.

But that is just one approach. A more robust, heavyweight UDP P2P might be a hole-punching TCP alternative. It's up to the community and results of experimentation.

Bittorrent has evolved a full transfer protocol over UDP, to get around firewalls and the like.

On 03/23/2013 11:24 AM, Jeff Garzik wrote:

> On Sat, Mar 23, 2013 at 10:52 AM, Luke-Jr <luke@dashjr•org> wrote:
>> On Saturday, March 23, 2013 10:42:26 AM Randy Willis wrote:
>>> Introducing super-nodes with thousands of connected peers can greatly help
>>> here.
>>
>> UDP is connectionless.
>> I would hope any UDP bitcoin protocol doesn't try to emulate a connection. :/
>
> It depends on the usage.  Simply broadcasting a TX or INV to a remote
> peer does not require a connection, clearly...  but you probably want
> to signal acceptance of those messages somehow.
>
> But other uses, like subscribing to a broadcast, does require some
> notion of an association.
>
> In the rough draft, a parallel TCP connection with version/verack
> sequence is required, and you may make use of it if a connection is
> needed.
>
> But that is just one approach.  A more robust, heavyweight UDP P2P
> might be a hole-punching TCP alternative.  It's up to the community
> and results of experimentation.
>
> Bittorrent has evolved a full transfer protocol over UDP, to get
> around firewalls and the like.
>

Bittorrent uses UDP in 2 ways and for different reasons.

The tracker protocol is now UDP because large trackers are under such enormous strain from short lived HTTP connections (40Gb/s) that there have been instances of upstream routers becoming overloaded from the storm of SYN, ACK and FIN messages. UDP helps solve that.

The inter-peer protocol is now UDP because TCP does not play nice in the context of bufferbloat and Bittorrent needs lots of active TCP connections to work, exacerbating the problem. In this instance Bittorrent uses a full userspace TCP stack which just sends w/ UDP.

+1 for experimenting with UDP, we might learn some interesting things.

It's worthwhile to actually speed test UDP v. TCP because the time to send an INV on an established TCP connection with Nagle disabled may not be significantly longer than that for sending with UDP.

Also +1 for experimentation with sending a small transaction instead of an INV, if INVs are not being grouped because we want the fastest possible network propagation, they are mostly overhead anyway. If b/w is more important than propagation speed then of course TCP/Nagle is the way to go.

Thanks, Caleb

If you're considering a datagram protocol, you might be interested in some more modern alternatives to UDP:

UDT: Breaking the Data Transfer Bottleneck http://udt.sourceforge.net/

Stream Control Transmission Protocol http://en.wikipedia.org/wiki/Stream_Control_Transmission_Protocol

On Sat, Mar 23, 2013 at 12:17 AM, Jeff Garzik <jgarzik@exmulti•com> wrote:

> Here is a rough draft implementation of a UDP P2P protocol extension
> for bitcoin:
>
>      https://github.com/jgarzik/bitcoin/tree/udp
>      http://yyz.us/bitcoin/udp-v0.patch
>
> Protocol specification (such that it is):
>
> - UDP, bound to same port as TCP P2P (normally 8333)
> - Active, simultaneous TCP P2P connection required (useful against DoS
> and other attacks)
> - Same message format as TCP P2P, same pchMessageStart conventions,
> etc. (my CNetMessage pull req would be helpful here)
> - Multiple P2P messages per UDP packet permitted
> - Max UDP packet size 100*1024 bytes
> - Advertises NODE_UDP in nServices
> - New "getudpcook" TCP P2P command returns a "udpcook" message,
> containing a yummy cookie
> - UDP P2P commands "inv", "tx" and "addr" are handled as if received via
> TCP
> - UDP P2P command "udpsub" sets a mask, that subscribes to one or more
> data broadcasts.
> - When USM_INV_BCAST mask bit is set, receive "inv" messages via UDP
> rather than TCP.
>
> Project and design goals (or, why do this?):
>
> - It is theorized that UDP may be useful for some messages we
> broadcast throughout the network
> - A productive discussion STARTS with code, otherwise discussion
> continues forever.
> - Investigate improving "inv" relay speed
> - Investigate unconditional "tx" broadcasting via UDP, as "inv"
> alternative, for small tx's.
> - Investigate improving block relay speed (or perhaps block header relay
> speed)
> - Open up new design avenues, for P2P patterns more suited to UDP than
> TCP in general.
>
> --
> Jeff Garzik
> exMULTI, Inc.
> jgarzik@exmulti•com
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_mar
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

My first concern was that I and about everyone else only has TCP/UDP port forwarding, but at least for the first:

UDT uses UDP to transfer bulk data with its own reliability control and congestion control mechanisms. Multiple UDT flows can share a single UDP port, thus a firewall can open only one UDP port for all UDT connections.

The latter appears not so friendly to NAT.

On 3/23/2013 3:30 PM, Mark Friedenbach wrote:

> If you're considering a datagram protocol, you might be interested in
> some more modern alternatives to UDP:
>
> UDT: Breaking the Data Transfer Bottleneck
> http://udt.sourceforge.net/
>
> Stream Control Transmission Protocol
> http://en.wikipedia.org/wiki/Stream_Control_Transmission_Protocol
>
>
>

On Sat, Mar 23, 2013 at 5:57 PM, Jay F <jayf@outlook•com> wrote:

> My first concern was that I and about everyone else only has TCP/UDP
> port forwarding,

You tunnel it! http://tools.ietf.org/html/draft-tuexen-tsvwg-sctp-dtls-encaps-00

You could do worse to have a data stream that looks like WEBRTC traffic…

In some ways SCTP is a pretty optimal transport for Bitcoin like messaging.

On Sat, Mar 23, 2013 at 9:22 PM, Gregory Maxwell <gmaxwell@gmail•com> wrote:

> In some ways SCTP is a pretty optimal transport for Bitcoin like messaging.

Darn near everything can be shoehorned into a "message" So absolutely agreed... in theory. Been an SCTP fan for years.

Firewall practices tend to put a damper on cool new protocols like that, though.

Nearly all of these new(er) user-mode transports run over UDP, so you can hole-punch and port forward just the same. Some which don't can nevertheless be tunneled, to the same effect.

Ultimately I don't have any skin in this game though. Just trying to save someone from reinventing a perfectly good wheel ;)

On Sat, Mar 23, 2013 at 5:57 PM, Jay F <jayf@outlook•com> wrote:

> My first concern was that I and about everyone else only has TCP/UDP
> port forwarding, but at least for the first:
>
> UDT uses UDP to transfer bulk data with its own reliability control and
> congestion control mechanisms. Multiple UDT flows can share a single UDP
> port, thus a firewall can open only one UDP port for all UDT connections.
>
> The latter appears not so friendly to NAT.
>
>
> On 3/23/2013 3:30 PM, Mark Friedenbach wrote:
> > If you're considering a datagram protocol, you might be interested in
> > some more modern alternatives to UDP:
> >
> > UDT: Breaking the Data Transfer Bottleneck
> > http://udt.sourceforge.net/
> >
> > Stream Control Transmission Protocol
> > http://en.wikipedia.org/wiki/Stream_Control_Transmission_Protocol
> >
> >
> >
>
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_mar
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

> Been an SCTP fan for years.

I have never seen SCTP out there. Any example where it is used?

> Firewall practices tend to put a damper on cool new protocols like that, though.

We are a bit old fashioned, indeed.

rm sec-guy