Bootstrapping via BitTorrent trackers
10 messages, 13 Jun 2011. Started by Christian Decker.
Jeff Garzik (4), Christian Decker (3), Vladimir Marchenko (2), Gavin Andresen
Hi all.
Just wanted to carry the discussion from the Forum over to the dev-list.
We have quite a few bootstrapping mechanisms, starting with the overly complex (IMHO) IRC bootstrapping, which is often suspected as bot-activity. Then we have a few hardcoded nodes and some fallback nodes. I was wondering why we didn't adopt BitTorrent tracker bootstrapping until now? It's basically all it does. Given a hash (SHA1 hash of the genesis bloc would be nice ^^) it gives you a list of other nodes with the same hash.
Given that there are quite a few open trackers (accepting and tracking any hash you throw at them) we could just decide to use 2-3 of those to bootstrap.
The downside would be that they return bencoded data, which has to be interpreted first, but it's easier than implementing the IRC stuff, I think.
Any comments?
Regards, Chris
On Mon, Jun 13, 2011 at 4:55 AM, Christian Decker <decker.christian@gmail•com> wrote:
Quoted text (6 lines)
> We have quite a few bootstrapping mechanisms, starting with the overly > complex (IMHO) IRC bootstrapping, which is often suspected as bot-activity. > Then we have a few hardcoded nodes and some fallback nodes. I was wondering > why we didn't adopt BitTorrent tracker bootstrapping until now? It's > basically all it does. Given a hash (SHA1 hash of the genesis bloc would be > nice ^^) it gives you a list of other nodes with the same hash.
It seems to offer few benefits over DNS seeding, while potentially potentially creating a vulnerable hot spot in the DHT. Sybil attacks on DHTs are well documented.
Don't get me wrong, DNS Seeding is an excellent way to bootstrap via trusted nodes, I'm not trying to replace it. What I'm trying to get rid of is the IRC bootstrapping and the hardcoded nodes in the client, they're easy targets.
BitTorrent trackers are used to handle several thousands of requests, so they would probably scale well enough. I'm not even talking about using the DHT trackers, but using old fashioned HTTP based trackers. The fact that each bitcoin client would contact the tracker would make it very hard for an attacker to get bootstrapping clients to exclusively connect to his compromised clients. I would say that using a tracker such as OpenBittorrent provides the same advantages as using an IRC channel.
On Mon, Jun 13, 2011 at 11:09 AM, Jeff Garzik <jgarzik@exmulti•com> wrote:
Quoted text (21 lines)
> On Mon, Jun 13, 2011 at 4:55 AM, Christian Decker > <decker.christian@gmail•com> wrote: > > We have quite a few bootstrapping mechanisms, starting with the overly > > complex (IMHO) IRC bootstrapping, which is often suspected as > bot-activity. > > Then we have a few hardcoded nodes and some fallback nodes. I was > wondering > > why we didn't adopt BitTorrent tracker bootstrapping until now? It's > > basically all it does. Given a hash (SHA1 hash of the genesis bloc would > be > > nice ^^) it gives you a list of other nodes with the same hash. > > It seems to offer few benefits over DNS seeding, while potentially > potentially creating a vulnerable hot spot in the DHT. Sybil attacks > on DHTs are well documented. > > -- > Jeff Garzik > exMULTI, Inc. > jgarzik@exmulti•com >
On Mon, Jun 13, 2011 at 5:38 AM, Christian Decker <decker.christian@gmail•com> wrote:
Quoted text (7 lines)
> BitTorrent trackers are used to handle several thousands of requests, so > they would probably scale well enough. I'm not even talking about using the > DHT trackers, but using old fashioned HTTP based trackers. The fact that > each bitcoin client would contact the tracker would make it very hard for an > attacker to get bootstrapping clients to exclusively connect to his > compromised clients. I would say that using a tracker such as OpenBittorrent > provides the same advantages as using an IRC channel.
And how does the client discover HTTP trackers? You're either hardcoding -those- into the client, or adding an additional bootstrap step to discover them. Either way, it has the same problems as other current methods.
The history and experience of gnutella's web caches vs. UDP host caches seems highly relevant here.
one possible bootstrap method of last resort,
1. create a convention of bitcoind listening on a specific last octest of IPv4 address, let's say, .14 when possible. Those of us who have access to IP space would use .14's.
2. if no other bootstrap method works, client could start scanning x.x.x.14 addresses, perhaps in some semi-intelligent order (starting from more pobable /8's and /16's), if enough people place bitcoind on x.x.x.14 than after a 10-100 thousand checks it bound to find a bitcoind peer.
It's messy, with all the excessive scanning etc... but it does not depend on anything except a bunch of bitcoind by convention preferring listening on x.x.x.14's.
Given that this is a method of last resort in bootrap chain it whould hopefully not lead to DDOS on those unlucky to own *.14 and not running bitcoind there. Also the more people are running bitcoind on .14, the quicker it would find a peer, the less scanning to do. It is kind of self-regualting.
For whatever it worth...
On 13 June 2011 10:56, Jeff Garzik <jgarzik@exmulti•com> wrote:
Quoted text (33 lines)
> On Mon, Jun 13, 2011 at 5:38 AM, Christian Decker > <decker.christian@gmail•com> wrote: >> BitTorrent trackers are used to handle several thousands of requests, so >> they would probably scale well enough. I'm not even talking about using the >> DHT trackers, but using old fashioned HTTP based trackers. The fact that >> each bitcoin client would contact the tracker would make it very hard for an >> attacker to get bootstrapping clients to exclusively connect to his >> compromised clients. I would say that using a tracker such as OpenBittorrent >> provides the same advantages as using an IRC channel. > > And how does the client discover HTTP trackers? You're either > hardcoding -those- into the client, or adding an additional bootstrap > step to discover them. Either way, it has the same problems as other > current methods. > > The history and experience of gnutella's web caches vs. UDP host > caches seems highly relevant here. > > -- > Jeff Garzik > exMULTI, Inc. > jgarzik@exmulti•com > > ------------------------------------------------------------------------------ > EditLive Enterprise is the world's most technically advanced content > authoring tool. Experience the power of Track Changes, Inline Image > Editing and ensure content is compliant with Accessibility Checking. > http://p.sf.net/sfu/ephox-dev2dev > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development >
Yes, those trackers would be hard coded, just like the IRC servers and channels are hardcoded right now.
The advantages over IRC and DNS Seeds are: - sporadic HTTP requests to a tracker, as opposed to keeping an IRC connection open at all times - no virus/botnet like behaviour (automatically join IRC channel with cryptic name), ISPs tend to bother network admins (like myself) with alerts when they see this... - adapts faster than DNS Seeds which require configuration changes on seed should the nodes become unreachable - we already use HTTP to determine our external IP, so it would be a consolidation of transports - more peers than DNS Seeds (better load balancing)
As for Vladimirs proposal, seems like an extreme measure, that is not really practical. Also it leads to network partitions since nodes will prefer their own /8 and /16 networks. IPv6 will also soon be a problem for this method.
On Mon, Jun 13, 2011 at 12:54 PM, Vladimir Marchenko < vladimir@marchenko•co•uk> wrote:
Quoted text (75 lines)
> one possible bootstrap method of last resort, > > 1. create a convention of bitcoind listening on a specific last octest > of IPv4 address, let's say, .14 when possible. Those of us who have > access to IP space would use .14's. > > 2. if no other bootstrap method works, client could start scanning > x.x.x.14 addresses, perhaps in some semi-intelligent order (starting > from more pobable /8's and /16's), if enough people place bitcoind on > x.x.x.14 than after a 10-100 thousand checks it bound to find a > bitcoind peer. > > It's messy, with all the excessive scanning etc... but it does not > depend on anything except a bunch of bitcoind by convention preferring > listening on x.x.x.14's. > > Given that this is a method of last resort in bootrap chain it whould > hopefully not lead to DDOS on those unlucky to own *.14 and not > running bitcoind there. Also the more people are running bitcoind on > .14, the quicker it would find a peer, the less scanning to do. It is > kind of self-regualting. > > For whatever it worth... > > > On 13 June 2011 10:56, Jeff Garzik <jgarzik@exmulti•com> wrote: > > On Mon, Jun 13, 2011 at 5:38 AM, Christian Decker > > <decker.christian@gmail•com> wrote: > >> BitTorrent trackers are used to handle several thousands of requests, so > >> they would probably scale well enough. I'm not even talking about using > the > >> DHT trackers, but using old fashioned HTTP based trackers. The fact that > >> each bitcoin client would contact the tracker would make it very hard > for an > >> attacker to get bootstrapping clients to exclusively connect to his > >> compromised clients. I would say that using a tracker such as > OpenBittorrent > >> provides the same advantages as using an IRC channel. > > > > And how does the client discover HTTP trackers? You're either > > hardcoding -those- into the client, or adding an additional bootstrap > > step to discover them. Either way, it has the same problems as other > > current methods. > > > > The history and experience of gnutella's web caches vs. UDP host > > caches seems highly relevant here. > > > > -- > > Jeff Garzik > > exMULTI, Inc. > > jgarzik@exmulti•com > > > > > ------------------------------------------------------------------------------ > > EditLive Enterprise is the world's most technically advanced content > > authoring tool. Experience the power of Track Changes, Inline Image > > Editing and ensure content is compliant with Accessibility Checking. > > http://p.sf.net/sfu/ephox-dev2dev > > _______________________________________________ > > Bitcoin-development mailing list > > Bitcoin-development@lists•sourceforge•net > > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > > > > ------------------------------------------------------------------------------ > EditLive Enterprise is the world's most technically advanced content > authoring tool. Experience the power of Track Changes, Inline Image > Editing and ensure content is compliant with Accessibility Checking. > http://p.sf.net/sfu/ephox-dev2dev > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development >
On Mon, Jun 13, 2011 at 7:48 AM, Christian Decker <decker.christian@gmail•com> wrote:
Quoted text (3 lines)
> The advantages over IRC and DNS Seeds are: > - sporadic HTTP requests to a tracker, as opposed to keeping an IRC > connection open at all times
No advantage over DNS (you're talking about IRC)
Quoted text (3 lines)
> - no virus/botnet like behaviour (automatically join IRC channel with > cryptic name), ISPs tend to bother network admins (like myself) with alerts > when they see this...
No advantage over DNS (you're talking about IRC)
Quoted text (2 lines)
> - adapts faster than DNS Seeds which require configuration changes on seed > should the nodes become unreachable
False. A proper DNS seed actively samples the bitcoin network for live nodes.
Quoted text (2 lines)
> - we already use HTTP to determine our external IP, so it would be a > consolidation of transports
Irrelevant -- it's still a bunch of additional code. We "already use" DNS lookups as well, so I could apply similar logic to DNS seeding.
Quoted text (1 line)
> - more peers than DNS Seeds (better load balancing)
This is an invalid assumption, based on an overly simplistic sampling. DNS seeds are perfectly capable of returning new and useful peers on each query.
Jeff is absolutely correct, stating that DNS bootstrapping can potentially be very robust.
Consider, for example, that seeds can be hosted by services like zerigo, who provide decent management API's, as well as by thousands of small VPS operated by people all over the world. Moreover, if namecoin lives up to it's promises than some seeds can be hosted using it's tlds, and as such we would get around centralised ICANN/registrar weakness.
The most practical way I see how to improve DNS bootstrapping is publishing build instructions for a simple DNS server build and a script which feeds it with node data. Something simple like djbdns plus a perl script, or similar zerigo API script, would do. Than anyone can host it. All left to do than is to hardcode a bunch of such DNS seeds into the client.
Hardcoding seeds is not good enough? Get a convention that anyone owning a bitcoin.* domain should point seed.bitcoin.* to a decent DNS seed. Than clients simply scan 100-200 tld's to find working seed.bitcoin.*
I am not quite sure, though, is if this would enable some attacks by poisoned seeds.
Vladimir.
Quoted text (6 lines)
> The most practical way I see how to improve DNS bootstrapping is > publishing build instructions for a simple DNS server build and a > script which feeds it with node data. Something simple like djbdns plus > a perl script, or similar zerigo API script, would do. Than anyone can > host it. All left to do than is to hardcode a bunch of such DNS seeds > into the client.
Adding that code to bitcoin in a contrib/dns folder seems like a good idea. I'd be happy to run a dns seed.
On Mon, Jun 13, 2011 at 2:41 PM, Gavin <gavinandresen@gmail•com> wrote:
Quoted text (8 lines)
>> The most practical way I see how to improve DNS bootstrapping is >> publishing build instructions for a simple DNS server build and a >> script which feeds it with node data. Something simple like djbdns plus >> a perl script, or similar zerigo API script, would do. Than anyone can >> host it. All left to do than is to hardcode a bunch of such DNS seeds >> into the client. > > Adding that code to bitcoin in a contrib/dns folder seems like a good idea.
Agreed. I'll see if I can put together a couple different solutions, and stick them in there. ETA 3-4 weeks though, given current bugfix workload + personal issues (moving to a new house).
I had a thought related to this, but readily admit I haven't thought this through: create an RPC in bitcoin that returns a random selection of fresh P2P node addresses, from addr.dat. One problem I was running into was accessing this data from outside bitcoin.
Using ArtForz' half-a-node as a base was a possibility, but creating "getnodes" RPC seemed far easier.
djbdns (tinydns) is great for this: it is highly scalable and stable, and its input is a -very- simple text format, easier to generate than BIND zone files.
Quoted text (1 line)
> I'd be happy to run a dns seed.
Great! A big missing piece is simply trusted people running DNS seeds on stable connectivity, once the DNS software is there.