Building a node crawler to map network

>
> While I'm asking questions I'll add one more regarding the getaddr
> message.

> **
>
>
> While I'm asking questions I'll add one more regarding the getaddr message.
>
>
>
> Talking to myself here.  I just sent this message then found this brilliant
> set of articles in the Dev & Tech forum which answers the question very
> nicely: *https://bitcointalk.org/index.php?topic=41722.0<https://bitcointalk.org/index.php?board=6.0>
> *
> Anyway just as an FYI I've been running v0.0.0.0.0.0.0.0.1 for about an
> hour.  It's only running 10 concurrent connections due to girlfriend
> complaining she couldn't watch youtube but here's some early results.
>
> New nodes: 19319 // node address discovered but no contact attempt made yet
> Contacted nodes: 754
> Uncontactable nodes: 3253
> Limbo nodes: 9 //not as exciting as it sounds, just nodes with connect in
> progress
> Total nodes: 23335 // about 5000 from initial IRC discover, the rest are
> from getaddr
>
> Versions: {
> 300=1,
> 31900=7,
> 31902=1,
> 32000=2,
> 32001=7,
> 32002=22,
> 32100=100,
> 32200=24,
> 32300=277,
> 32400=317,
> 32500=2}
>
> Fails: {
> ConnectException: Connection refused=377,
> IOException: Socket is disconnected=87,
> SocketException: Network is unreachable=2,
> ProtocolException: Error deserializing message =1,
> NoRouteToHostException: No route to host=115,
> SocketException: Connection reset=149,
> SocketTimeoutException: connect timed out=2521}
>
>
>
>
> ------------------------------------------------------------------------------
> Special Offer -- Download ArcSight Logger for FREE!
> Finally, a world-class log management solution at an even better
> price-free! And you'll get a free "Love Thy Logs" t-shirt when you
> download Logger. Secure your free ArcSight Logger TODAY!
> http://p.sf.net/sfu/arcsisghtdev2dev
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

> Actually Steve, take a look at the bitcoinj mailing list today.
> Somebody has already built this and has it running. It's accumulating
> data at the moment, they'll announce it more widely soon. But I think
> there's no need to duplicate work.

>
> I've looked but can't find a post like you're talking about.  Can you point
> me to it?
>

> If so then bollocks... I'm looking for something useful to do atm.
>  PoolServerJ is in a holding pattern atm as I've stabilisied all the bugs I
> know about and am waiting for several pools to finish testing and move into
> production so I'm twiddling thumbs trying to figure out how to spend my
> time.
>

>
>     I've looked but can't find a post like you're talking about.  Can
>     you point me to it?
>
> https://groups.google.com/forum/?pli=1#!topic/bitcoinj/LSlZdUWcCdk
> <https://groups.google.com/forum/?pli=1#%21topic/bitcoinj/LSlZdUWcCdk>
>
>     If so then bollocks... I'm looking for something useful to do atm.
>      PoolServerJ is in a holding pattern atm as I've stabilisied all
>     the bugs I know about and am waiting for several pools to finish
>     testing and move into production so I'm twiddling thumbs trying to
>     figure out how to spend my time.
>
>
> Patches to BitCoinJ are always welcome :-)
>
> If you'd rather do your own thing, you could experiment with writing a
> proxy that sits in front of bitcoind and multiplexes connections.
> Gavin is concerned about socket exhaustion as users move to
> lightweight clients. Multiplexing proxies are a battle-tested
> technique for reducing the strain of this type of thing. BitCoinJ uses
> thread-per-connection so wouldn't do a good job of that right now, but
> allowing it to use a mix of async io and multi-threading would be a
> nice improvement. It'd need some changes to bitcoind as well for a
> really good effort, to allow for IPs to be forwarded. I'm happy to
> discuss it more with you over on the bitcoinj list if wanted.

> Hi All,
>
> I started messing around today with building a node crawler to try and
> map out the bitcoin network and hopefully provide some useful
> statistics.  It's very basic so far using a mutilated bitcoinj to
> connect (due me being java developer and not having a clue with c/c++).
>  If it's worthwhile I'll hack bitcoinj some more to run on top Netty to
> take advantage of it's NIO architecture (netty's been shown to handle
> 1/2 million concurrent connections so would be ideal for the purpose).
>
> Hoping to a get a bit of input into what would be useful as well as
> strategy for getting max possible connections without distorted data.  I
> seem to recall Gavin talking about the need for some kind of network
> health monitoring so I assume there's a need for something like this...
>
> Firstly at the moment basically I'm just storing version message and the
> results of getaddr for each node that I can connect to.  Is there any
> other useful info that can be extracted from a node that's worth
> collecting?
>
> Second and main issue is how to connect.  From my first very basic
> probing it seems the very vast majority of nodes don't accept incoming
> connections no doubt due to lack of upnp.  So it seems the active crawl
> approach is not really ideal for the purpose.  Even if it was used the
> resultant data would be hopelessly distorted.
>
> A honeypot approach would probably be better if there was some way to
> make a node 'attractive' to other nodes to connect to.  That way it
> could capture non-listening nodes as well.  If there is some way to
> influence other nodes to connect to the crawler node that solves the
> problem.  If there isn't which I suspect is the case then perhaps
> another approach is to build an easy to deploy crawler node that many
> volunteers could run and that could then upload collected data to a
> central repository.
>
> While I'm asking questions I'll add one more regarding the getaddr
> message.  It seems most nodes return about 1000 addresses in response to
> this message.  Obviously most of these nodes haven't actually talked to
> all 1000 on the list so where does this list come from?  Is it mixture
> of addresses obtained from other nodes somehow sorted by timestamp?
> Does it include some nodes discovered by IRC/DNS? Or are those only used
> to find the first nodes to connect to?
>
> Thanks for any input... Hopefully I can build something that's useful
> for the network...
>
>
> ------------------------------------------------------------------------------
> Special Offer -- Download ArcSight Logger for FREE!
> Finally, a world-class log management solution at an even better
> price-free! And you'll get a free "Love Thy Logs" t-shirt when you
> download Logger. Secure your free ArcSight Logger TODAY!
> http://p.sf.net/sfu/arcsisghtdev2dev
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

> **
> I'm not really understanding the use case though.  I believe most
> bitcoind's have a default max connections of 8.  Is the goal to increase
> this without fundamentally altering the bitcoind concurrency model?
>

> On Tue, Sep 6, 2011 at 4:17 PM, Steve <shadders.del@gmail•com
> <mailto:shadders.del@gmail.com>> wrote:
>
>     I'm not really understanding the use case though.  I believe most
>     bitcoind's have a default max connections of 8.  Is the goal to
>     increase this without fundamentally altering the bitcoind
>     concurrency model?
>
>
> bitcoind already uses asynchronous IO. That's not the problem.
>
> The issue came up in a conversation about scalability. If Bitcoins
> popularity continues to grow, users are very likely to migrate away
> from running full verifying nodes to lightweight clients, either a
> different mode of the Satoshi client or different implementations like
> the Android Wallet or MultiBit.
>
> Lightweight clients cannot verify thus should not relay. And they'll
> be run by users who just want to send/receive coins from time to time,
> so don't leave the programs running 24/7. The result could be running
> out of sockets (like we have had problems with recently). It's
> especially true because lightweight clients cannot check transactions
> for themselves. If they want to show transactions appearing
> immediately (and they do), they have to use "heard from lots of nodes"
> as a proxy for validity. So lightweight clients are likely to be
> socket intensive.
>
> We could solve this by just hoping that lots of people run full nodes.
> The problem is that a full node is quite an intensive thing already,
> it uses lots of CPU and disk seeks, and will just get more expensive
> in future. And as transaction traffic increases, that leaves less CPU
> time available to service thousands of connected clients. The ROI of
> bringing up a new node decreases at the same time as the userbase
> increases.
>
> One traditional approach to solving this is frontend proxies.
> Jabber.com/org used this technique many years ago, and Google has also
> used it to scale up the lockservice
> <http://static.googleusercontent.com/external_content/untrusted_dlcp/labs.google.com/en/us/papers/chubby-osdi06.pdf> (see
> section 3.1). It's effective because often maintaining connections to
> thousands of clients doesn't involve much brainwork, just shifting
> bytes around. This is especially true of Bitcoin. So if somebody is
> running a full node already they could increase their client capacity
> by just bringing up a frontend proxy and having it handle things like
> outbound tx broadcasts/deduping inbound broadcasts, connection setup,
> relaying recently found blocks etc. A well written proxy could
> probably support tens of thousands of simultaneous clients which frees
> up the bitcoinds time for verification and wallet manipulation.