DNS seeds returning gone peers
15 messages, 3 Aug 2011. Started by Mike Hearn.
Mike Hearn (6), Matt Corallo (2), Rick Wesson (2), Johannes Henninger, Caleb James DeLisle, Douglas Huff, Christian Decker, Greg Maxwell
This is expected to happen from time to time of course as it's inherently racy, but there are a *lot* of bad nodes appearing in the DNS seeds.
$ nmap -oG /tmp/x -p 8333 `dig +short bitseed.bitcoin.org.uk dnsseed.bluematt.me bitseed.xf2.org` ... Nmap done: 48 IP addresses (25 hosts up) scanned in 9.80 seconds
$ grep -c 'closed' /tmp/x 6
So of 48 IPs returned only 19 are actually usable. This is slowing down peer bringup for the Android apps, which don't currently save the addresses of last-used peers (yes, I know we should fix this).
I was talking to a friend a few days ago about Bitcoin, he seemed interested. I'm hoping he might take on DNS seeding as a project. A custom DNS server that watches the network to find long-lived peers that run the latest version would be helpful for resolving this kind of thing.
On Wed, 2011-08-03 at 12:04 +0200, Mike Hearn wrote:
Quoted text (12 lines)
> This is expected to happen from time to time of course as it's > inherently racy, but there are a lot of bad nodes appearing in the DNS > seeds. > $ nmap -oG /tmp/x -p 8333 `dig +short bitseed.bitcoin.org.uk > dnsseed.bluematt.me bitseed.xf2.org` > ... > Nmap done: 48 IP addresses (25 hosts up) scanned in 9.80 seconds > $ grep -c 'closed' /tmp/x > 6 > So of 48 IPs returned only 19 are actually usable. This is slowing > down peer bringup for the Android apps, which don't currently save the > addresses of last-used peers (yes, I know we should fix this).
Its actually much, much less. You forgot to grep for filtered, which are also worthless and you didn't make an actual connection to the node, meaning there is no way to tell if the node has its connection slots full (a node which has the maximum connection count will ack a syn, but will drop the connection after the first message, so nmap thinks the port is open). I just tested and I show 0 accepting from bitseed.xf2.org and 0 from bitcoin.bitcoin.co.uk. dnsseed.bluematt.me rotates every 2 minutes to the most recently checked so it tends to be pretty good if you get it right after a rotate, if you wait to long, those slots fill up quick.
Quoted text (6 lines)
> > I was talking to a friend a few days ago about Bitcoin, he seemed > interested. I'm hoping he might take on DNS seeding as a project. A > custom DNS server that watches the network to find long-lived peers > that run the latest version would be helpful for resolving this kind > of thing.
Point him to https://github.com/TheBlueMatt/dnsseed it could use a bit of cleanup, but it works. If a different DNS Server were used to could pull directly from the database in a more dynamic way it would probably work better too (it was originally set up on MySQL and PowerDNS, but that is quite a resource hog compared to SQLite and BIND, but the original backend is still there and could work if you have a beefy enough server).
Matt
Yeah, the limited number of slots doesn't help either. I wonder if the current settings are too conservative.
Your seed returns quite a few IPs that don't respond to ICMP pings - not sure what is happening there.
03:01:53 scott:~$ for ip in `dig +short dnsseed.bluematt.me`; do echo -n -e "Testing $ip: "; ping -c 1 $ip|grep 'packet loss'; done Testing 24.7.158.162: 1 packets transmitted, 1 received, 0% packet loss, time 0ms *Testing 50.19.225.254: 1 packets transmitted, 0 received, 100% packet loss, time 0ms Testing 67.242.10.199: 1 packets transmitted, 0 received, 100% packet loss, time 0ms Testing 72.223.56.138: 1 packets transmitted, 0 received, 100% packet loss, time 0ms Testing 76.92.171.255: 1 packets transmitted, 0 received, 100% packet loss, time 0ms Testing 76.123.10.117: 1 packets transmitted, 0 received, 100% packet loss, time 0ms Testing 80.3.173.28: 1 packets transmitted, 0 received, 100% packet loss, time 0ms *Testing 80.101.109.52: 1 packets transmitted, 1 received, 0% packet loss, time 0ms Testing 93.186.32.117: 1 packets transmitted, 1 received, 0% packet loss, time 0ms *Testing 94.19.17.167: 1 packets transmitted, 0 received, 100% packet loss, time 0ms *Testing 97.86.39.50: 1 packets transmitted, 1 received, 0% packet loss, time 0ms Testing 113.255.177.241: 1 packets transmitted, 1 received, 0% packet loss, time 0ms Testing 118.208.226.66: 1 packets transmitted, 1 received, 0% packet loss, time 0ms Testing 173.180.141.86: 1 packets transmitted, 1 received, 0% packet loss, time 0ms *Testing 174.119.14.66: 1 packets transmitted, 0 received, 100% packet loss, time 0ms *Testing 178.18.129.133: 1 packets transmitted, 1 received, 0% packet loss, time 0ms *Testing 193.86.163.77: 1 packets transmitted, 0 received, 100% packet loss, time 0ms Testing 206.255.99.164: 1 packets transmitted, 0 received, 100% packet loss, time 0ms Testing 216.8.180.85: 1 packets transmitted, 0 received, 100% packet loss, time 0ms Testing 24.1.117.3: 1 packets transmitted, 0 received, 100% packet loss, time 0ms*
On 08/03/2011 02:00 PM, Mike Hearn wrote:
Quoted text (5 lines)
> Yeah, the limited number of slots doesn't help either. I wonder if the > current settings are too conservative. > > Your seed returns quite a few IPs that don't respond to ICMP pings - > not sure what is happening there.
Many home routers ignore ICMP requests (mine does) and as far as I know Windows Vista/7 only responds to pings that originate from a LAN address/private address range in order to stay more "hidden".
OK, but, <50% show as port open .... maybe a bug in the seed?
05:16:52 scott:~$ nmap -p 8333 `dig +short dnsseed.bluematt.me`
Starting Nmap 5.00 ( http://nmap.org ) at 2011-08-03 14:17 CEST Interesting ports on 83.220.45.22: PORT STATE SERVICE 8333/tcp open unknown
Interesting ports on c-6de0e055.27-2-64736c11.cust.bredbandsbolaget.se(85.224.224.109): PORT STATE SERVICE 8333/tcp open unknown
Interesting ports on mackila.com (88.168.105.251): PORT STATE SERVICE 8333/tcp open unknown
Interesting ports on 93-81-112-85.broadband.corbina.ru (93.81.112.85): PORT STATE SERVICE 8333/tcp open unknown
Interesting ports on kons-5f710a2a.pool.mediaWays.net (95.113.10.42): PORT STATE SERVICE 8333/tcp open unknown
Interesting ports on 173-218-216-132.atw.suddenlink.net (173.218.216.132): PORT STATE SERVICE 8333/tcp open unknown
Interesting ports on dynamicip-188-232-23-24.pppoe.omsk.ertelecom.ru(188.232.23.24): PORT STATE SERVICE 8333/tcp open unknown
Interesting ports on c-71-229-116-166.hsd1.fl.comcast.net (71.229.116.166): PORT STATE SERVICE 8333/tcp filtered unknown
Interesting ports on c-76-25-209-23.hsd1.co.comcast.net (76.25.209.23): PORT STATE SERVICE 8333/tcp open unknown
Nmap done: 20 IP addresses (9 hosts up) scanned in 3.50 seconds
On 08/03/2011 07:38 AM, Matt Corallo wrote:
Quoted text (26 lines)
> On Wed, 2011-08-03 at 12:04 +0200, Mike Hearn wrote: >> This is expected to happen from time to time of course as it's >> inherently racy, but there are a lot of bad nodes appearing in the DNS >> seeds. > >> $ nmap -oG /tmp/x -p 8333 `dig +short bitseed.bitcoin.org.uk >> dnsseed.bluematt.me bitseed.xf2.org` >> ... >> Nmap done: 48 IP addresses (25 hosts up) scanned in 9.80 seconds > >> $ grep -c 'closed' /tmp/x >> 6 > >> So of 48 IPs returned only 19 are actually usable. This is slowing >> down peer bringup for the Android apps, which don't currently save the >> addresses of last-used peers (yes, I know we should fix this). > Its actually much, much less. You forgot to grep for filtered, which > are also worthless and you didn't make an actual connection to the node, > meaning there is no way to tell if the node has its connection slots > full (a node which has the maximum connection count will ack a syn, but > will drop the connection after the first message, so nmap thinks the > port is open). > I just tested and I show 0 accepting from bitseed.xf2.org and 0 from > bitcoin.bitcoin.co.uk. dnsseed.bluematt.me rotates every 2 minutes to > the most recently checked so it tends to be pretty good if you get it > right after a rotate, if you wait to long, those slots fill up quick.
Someone I know who runs a moderately large website told me that some ISPs cache DNS for as long as a week without regard to TTL. If your DNS seeds are not pointing to your own dedicated boxen then you might want to do a lookup on a random cookie as a subdomain.
Caleb
Quoted text (31 lines)
>> >> I was talking to a friend a few days ago about Bitcoin, he seemed >> interested. I'm hoping he might take on DNS seeding as a project. A >> custom DNS server that watches the network to find long-lived peers >> that run the latest version would be helpful for resolving this kind >> of thing. > Point him to https://github.com/TheBlueMatt/dnsseed it could use a bit > of cleanup, but it works. > If a different DNS Server were used to could pull directly from the > database in a more dynamic way it would probably work better too (it was > originally set up on MySQL and PowerDNS, but that is quite a resource > hog compared to SQLite and BIND, but the original backend is still there > and could work if you have a beefy enough server). > > Matt > > > > ------------------------------------------------------------------------------ > BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA > The must-attend event for mobile developers. Connect with experts. > Get tools for creating Super Apps. See the latest technologies. > Sessions, hands-on labs, demos & much more. Register early & save! > http://p.sf.net/sfu/rim-blackberry-1 > > > > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Quoted text (4 lines)
> > Someone I know who runs a moderately large website told me that some ISPs > cache DNS for as long as a week without regard to TTL. >
We track DNS disobeyers at Google, as we use it for load balancing (along with many other large sites).
I'd be kind of surprised if any large/professional ISP disobeyed the TTL that badly, because it would cause frequent problems reaching popular sites like anything hosted on Google or Akamai. But randomizing the DNS request isn't a bad idea.
Mike,
I think I can contribute to your DNS seeding project. Could you help define long-lived peers?
-rick
On Wed, Aug 3, 2011 at 3:04 AM, Mike Hearn <mike@plan99•net> wrote:
Quoted text (33 lines)
> This is expected to happen from time to time of course as it's inherently > racy, but there are a *lot* of bad nodes appearing in the DNS seeds. > > $ nmap -oG /tmp/x -p 8333 `dig +short bitseed.bitcoin.org.uk > dnsseed.bluematt.me bitseed.xf2.org` > ... > Nmap done: 48 IP addresses (25 hosts up) scanned in 9.80 seconds > > $ grep -c 'closed' /tmp/x > 6 > > So of 48 IPs returned only 19 are actually usable. This is slowing down > peer bringup for the Android apps, which don't currently save the addresses > of last-used peers (yes, I know we should fix this). > > I was talking to a friend a few days ago about Bitcoin, he seemed > interested. I'm hoping he might take on DNS seeding as a project. A custom > DNS server that watches the network to find long-lived peers that run the > latest version would be helpful for resolving this kind of thing. > > > ------------------------------------------------------------------------------ > BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA > The must-attend event for mobile developers. Connect with experts. > Get tools for creating Super Apps. See the latest technologies. > Sessions, hands-on labs, demos & much more. Register early & save! > http://p.sf.net/sfu/rim-blackberry-1 > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > >
There's no project currently :-)
Starting from Matts code is probably the way to go. It's written in PHP. Alternatively, you could write a Java app for it, as there are drop-in DNS serving libraries you could link with BitCoinJ+sqlite. It probably wouldn't be that hard. You'd want to sort nodes by version, how long they've been observed to exist, the last polling time, etc.
On Wed, Aug 3, 2011 at 4:00 PM, Rick Wesson <rick@support-intelligence•com>wrote:
Quoted text (44 lines)
> Mike, > > I think I can contribute to your DNS seeding project. Could you help define > long-lived peers? > > -rick > > > On Wed, Aug 3, 2011 at 3:04 AM, Mike Hearn <mike@plan99•net> wrote: > >> This is expected to happen from time to time of course as it's inherently >> racy, but there are a *lot* of bad nodes appearing in the DNS seeds. >> >> $ nmap -oG /tmp/x -p 8333 `dig +short bitseed.bitcoin.org.uk >> dnsseed.bluematt.me bitseed.xf2.org` >> ... >> Nmap done: 48 IP addresses (25 hosts up) scanned in 9.80 seconds >> >> $ grep -c 'closed' /tmp/x >> 6 >> >> So of 48 IPs returned only 19 are actually usable. This is slowing down >> peer bringup for the Android apps, which don't currently save the addresses >> of last-used peers (yes, I know we should fix this). >> >> I was talking to a friend a few days ago about Bitcoin, he seemed >> interested. I'm hoping he might take on DNS seeding as a project. A custom >> DNS server that watches the network to find long-lived peers that run the >> latest version would be helpful for resolving this kind of thing. >> >> >> ------------------------------------------------------------------------------ >> BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA >> The must-attend event for mobile developers. Connect with experts. >> Get tools for creating Super Apps. See the latest technologies. >> Sessions, hands-on labs, demos & much more. Register early & save! >> http://p.sf.net/sfu/rim-blackberry-1 >> _______________________________________________ >> Bitcoin-development mailing list >> Bitcoin-development@lists•sourceforge•net >> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >> >> >
Starting from bitcoinj, I have plenty of ways to publish DNS. Why sort them by version? Ordering from highest to lowest?
how about publishing addresses under version.example.com if you version has a perfrence?
-rick
On Wed, Aug 3, 2011 at 7:10 AM, Mike Hearn <mike@plan99•net> wrote:
Quoted text (56 lines)
> There's no project currently :-) > > Starting from Matts code is probably the way to go. It's written in PHP. > Alternatively, you could write a Java app for it, as there are drop-in DNS > serving libraries you could link with BitCoinJ+sqlite. It probably wouldn't > be that hard. You'd want to sort nodes by version, how long they've been > observed to exist, the last polling time, etc. > > On Wed, Aug 3, 2011 at 4:00 PM, Rick Wesson <rick@support-intelligence•com > > wrote: > >> Mike, >> >> I think I can contribute to your DNS seeding project. Could you help >> define long-lived peers? >> >> -rick >> >> >> On Wed, Aug 3, 2011 at 3:04 AM, Mike Hearn <mike@plan99•net> wrote: >> >>> This is expected to happen from time to time of course as it's inherently >>> racy, but there are a *lot* of bad nodes appearing in the DNS seeds. >>> >>> $ nmap -oG /tmp/x -p 8333 `dig +short bitseed.bitcoin.org.uk >>> dnsseed.bluematt.me bitseed.xf2.org` >>> ... >>> Nmap done: 48 IP addresses (25 hosts up) scanned in 9.80 seconds >>> >>> $ grep -c 'closed' /tmp/x >>> 6 >>> >>> So of 48 IPs returned only 19 are actually usable. This is slowing down >>> peer bringup for the Android apps, which don't currently save the addresses >>> of last-used peers (yes, I know we should fix this). >>> >>> I was talking to a friend a few days ago about Bitcoin, he seemed >>> interested. I'm hoping he might take on DNS seeding as a project. A custom >>> DNS server that watches the network to find long-lived peers that run the >>> latest version would be helpful for resolving this kind of thing. >>> >>> >>> ------------------------------------------------------------------------------ >>> BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA >>> The must-attend event for mobile developers. Connect with experts. >>> Get tools for creating Super Apps. See the latest technologies. >>> Sessions, hands-on labs, demos & much more. Register early & save! >>> http://p.sf.net/sfu/rim-blackberry-1 >>> _______________________________________________ >>> Bitcoin-development mailing list >>> Bitcoin-development@lists•sourceforge•net >>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >>> >>> >> >
There's a bigger problem here honestly. The p2p network is just starved for connectable slots.
You can start up a bitcoind, on a host with sufficient connectivity, with -maxconnections=512 and they will fill in <15 minutes.
As to why sort by version: <=3.23 has problems serving the blocks from the last 2 months. It can take days to weeks to get the whole chain if you're connecting to those nodes.
I'm curious whether my Non-Blocking network stack (BitDroid) isn't better suited for detecting and tracking available peers. I have implemented several benchmarks, including a simple peer counter listener, which would have to be adapted to fit the DNS needs (open and check if a real peer is listening). Being non-blocking it can open several hundreds of connections to check reachability of the peers and at the same time keep a pool of peers connected to listen for address broadcasts, with minimal overhead (single thread, close to no context switches).
Just an idea :-)
Regards, Chris
On Wed, Aug 3, 2011 at 4:18 PM, Rick Wesson <rick@support-intelligence•com>wrote:
Quoted text (82 lines)
> Starting from bitcoinj, I have plenty of ways to publish DNS. Why sort them > by version? Ordering from highest to lowest? > > how about publishing addresses under version.example.com if you version > has a perfrence? > > -rick > > > On Wed, Aug 3, 2011 at 7:10 AM, Mike Hearn <mike@plan99•net> wrote: > >> There's no project currently :-) >> >> Starting from Matts code is probably the way to go. It's written in PHP. >> Alternatively, you could write a Java app for it, as there are drop-in DNS >> serving libraries you could link with BitCoinJ+sqlite. It probably wouldn't >> be that hard. You'd want to sort nodes by version, how long they've been >> observed to exist, the last polling time, etc. >> >> On Wed, Aug 3, 2011 at 4:00 PM, Rick Wesson < >> rick@support-intelligence•com> wrote: >> >>> Mike, >>> >>> I think I can contribute to your DNS seeding project. Could you help >>> define long-lived peers? >>> >>> -rick >>> >>> >>> On Wed, Aug 3, 2011 at 3:04 AM, Mike Hearn <mike@plan99•net> wrote: >>> >>>> This is expected to happen from time to time of course as it's >>>> inherently racy, but there are a *lot* of bad nodes appearing in the >>>> DNS seeds. >>>> >>>> $ nmap -oG /tmp/x -p 8333 `dig +short bitseed.bitcoin.org.uk >>>> dnsseed.bluematt.me bitseed.xf2.org` >>>> ... >>>> Nmap done: 48 IP addresses (25 hosts up) scanned in 9.80 seconds >>>> >>>> $ grep -c 'closed' /tmp/x >>>> 6 >>>> >>>> So of 48 IPs returned only 19 are actually usable. This is slowing down >>>> peer bringup for the Android apps, which don't currently save the addresses >>>> of last-used peers (yes, I know we should fix this). >>>> >>>> I was talking to a friend a few days ago about Bitcoin, he seemed >>>> interested. I'm hoping he might take on DNS seeding as a project. A custom >>>> DNS server that watches the network to find long-lived peers that run the >>>> latest version would be helpful for resolving this kind of thing. >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA >>>> The must-attend event for mobile developers. Connect with experts. >>>> Get tools for creating Super Apps. See the latest technologies. >>>> Sessions, hands-on labs, demos & much more. Register early & save! >>>> http://p.sf.net/sfu/rim-blackberry-1 >>>> _______________________________________________ >>>> Bitcoin-development mailing list >>>> Bitcoin-development@lists•sourceforge•net >>>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >>>> >>>> >>> >> > > > ------------------------------------------------------------------------------ > BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA > The must-attend event for mobile developers. Connect with experts. > Get tools for creating Super Apps. See the latest technologies. > Sessions, hands-on labs, demos & much more. Register early & save! > http://p.sf.net/sfu/rim-blackberry-1 > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > >
Quoted text (4 lines)
> > There's a bigger problem here honestly. The p2p network is just starved for > connectable slots. >
Suggestions:
- massively increasing all the anti-DoS limits in 0.4, so far they've caused a lot more damage than they solved.
- broadcasting an "upgrade now" type announcement. The alert system is intended for informing users when there's a threat to system stability and that's exactly what has been happening recently.
On Wed, Aug 3, 2011 at 10:39 AM, Mike Hearn <mike@plan99•net> wrote:
Quoted text (6 lines)
>> There's a bigger problem here honestly. The p2p network is just starved >> for connectable slots. > > Suggestions: > - massively increasing all the anti-DoS limits in 0.4, so far they've caused > a lot more damage than they solved.
You can't "massively increase" the number of available connection slots without risking running nodes on lower memory systems (e.g. VMs) out of memory.
Moreover, 125 slots should be more than enough. We need to figure out why it isn't.
On Wed, 2011-08-03 at 10:48 -0400, Gregory Maxwell wrote:
Quoted text (7 lines)
> On Wed, Aug 3, 2011 at 10:39 AM, Mike Hearn <mike@plan99•net> wrote: > >> There's a bigger problem here honestly. The p2p network is just starved > >> for connectable slots. > > > > Suggestions: > > - massively increasing all the anti-DoS limits in 0.4, so far they've caused > > a lot more damage than they solved.
They were redone in .24 to the point that they should not cause any such issues in the future.
Quoted text (7 lines)
> > You can't "massively increase" the number of available connection > slots without risking running nodes on lower memory systems (e.g. VMs) > out of memory. > > Moreover, 125 slots should be more than enough. We need to figure out > why it isn't.
Agreed.