Announcing the Statoshi fork
19 messages, 7 to 8 May 2014. Started by Jameson Lopp.
Wladimir J. van der Laan (6), Jameson Lopp (4), Mike Hearn (4), Charlie 'Charles' Shrem (2), Pavol Rusnak, Nelson Castillo, Greg Maxwell
In order to gain more insight into what messages and requests a node is processing, I've created a Bitcoin Core fork that outputs statistics to StatsD. I hope that some of you will find this interesting and potentially useful.
http://coinchomp.com/2014/05/07/announcing-statoshi-realtime-bitcoin-node-statistics/
https://jlopp.github.io/statoshi/
Feedback is appreciated!
- - Jameson
On Wed, May 7, 2014 at 9:12 PM, Jameson Lopp <jameson.lopp@gmail•com> wrote:
Quoted text (10 lines)
> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > In order to gain more insight into what messages and requests a node is processing, I've created a Bitcoin Core fork that outputs statistics to StatsD. I hope that some of you will find this interesting and potentially useful. > > http://coinchomp.com/2014/05/07/announcing-statoshi-realtime-bitcoin-node-statistics/ > > https://jlopp.github.io/statoshi/ > > Feedback is appreciated!
Ooh nice graphs!
We were coincidentally talking about showing stats from a node on a local web site on the #bitcoin-dev IRC a few days ago.
At some point, if we're going to offer Bitcoin Core node-only installers, it'd be nice to include something like this so the user can keep an eye on their node(s).
Wladimir
Really nice! We definitely need to put together a team who really cares about the operations side of the network and this is a fantastic start.
It'd be nice if you didn't assume knowledge of what statsd is out of the box. Given the name I'd assumed it was a small UNIX daemon but it seems it's actually a Javascript thingy?
It looks like putting together a monitored bitcoind setup can be quite a lot of work. I wonder if there are ways to simplify it. For example, would it make sense for someone to run a community statsd and graphite instance, so we can get aggregate statistics across many nodes and the node operators don't have to set everything up themselves? Does that make any sense?
On 05/07/2014 09:12 PM, Jameson Lopp wrote:
Quoted text (1 line)
> I've created a Bitcoin Core fork that outputs statistics to StatsD.
How complex is the patchset? Would it be possible to merge it into the mainline and enable compilation of this feature conditionally by some build-time option?
The next logical step may be for us to offer a public instance of these graphs; I'd be happy to work with you to set one up.
I agree that it would be awesome to offer these types of stats with the installer; unfortunately the route I've taken has dependencies on several other other pieces of software to do all the heavy lifting of stats aggregation and chart rendering. I'm assuming that you would not want to build any of that processing into Bitcoin Core itself; would you be opposed to packaging other software along with the installer?
- - Jameson
On 05/07/2014 03:46 PM, Wladimir wrote:
Quoted text (23 lines)
> On Wed, May 7, 2014 at 9:12 PM, Jameson Lopp <jameson.lopp@gmail•com> wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> In order to gain more insight into what messages and requests a node is processing, I've created a Bitcoin Core fork that outputs statistics to StatsD. I hope that some of you will find this interesting and potentially useful. >> >> http://coinchomp.com/2014/05/07/announcing-statoshi-realtime-bitcoin-node-statistics/ >> >> https://jlopp.github.io/statoshi/ >> >> Feedback is appreciated! > > Ooh nice graphs! > > We were coincidentally talking about showing stats from a node on a > local web site on the #bitcoin-dev IRC a few days ago. > > At some point, if we're going to offer Bitcoin Core node-only > installers, it'd be nice to include something like this so the user > can keep an eye on their node(s). > > Wladimir >
I completely agree that this setup is far too difficult to reasonably expect anyone to implement it. You're correct that we could run a single StatsD daemon and have quite a few nodes sending statistics to it - this is really what StatsD was designed for - sampling small amounts of stats from high volume systems. There would be an issue of trust, however - StatsD was also only really designed to be run inside of highly secure infrastructure where you trust all of the machines that are talking to it.
- - Jameson
On 05/07/2014 03:50 PM, Mike Hearn wrote:
Quoted text (13 lines)
> Really nice! We definitely need to put together a team who really cares > about the operations side of the network and this is a fantastic start. > > It'd be nice if you didn't assume knowledge of what statsd is out of the > box. Given the name I'd assumed it was a small UNIX daemon but it seems > it's actually a Javascript thingy? > > It looks like putting together a monitored bitcoind setup can be quite a > lot of work. I wonder if there are ways to simplify it. For example, would > it make sense for someone to run a community statsd and graphite instance, > so we can get aggregate statistics across many nodes and the node operators > don't have to set everything up themselves? Does that make any sense? >
It looks like the packet format statsd expects is rather simple - it should be easy to experiment with.
Perhaps a good next step would be to improve your patch so that someone can get a feed of the stats packets via TCP by e.g. ssh tunnelling to their host. Once it's easy to get a feed of simple stats packets, people can easily experiment with different kinds of aggregation and monitoring software from their desktop.
On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp <jameson.lopp@gmail•com> wrote:
Quoted text (4 lines)
> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > I agree that it would be awesome to offer these types of stats with the installer; unfortunately the route I've taken has dependencies on several other other pieces of software to do all the heavy lifting of stats aggregation and chart rendering. I'm assuming that you would not want to build any of that processing into Bitcoin Core itself; would you be opposed to packaging other software along with the installer?
Depends on just how much stuff it is. The idea is primarily to have an installer for running a (wallet-less) node as an OS background service.
Having some statistics available would be worth some extra download size, otherwise it would be pretty much invisible.
We'd already decided that we would need something like Python for the stats service. Implementing things like web services in C++ is just not realistic given the time constraints and the great already-written code that is out there. As an optional tool it should be external, not part of bitcoind itself.
I suppose the chart rendering happens client-side? In that case the web service just has to collect and provide the data, and serve static html/js files.
Wladimir
I think there a few different possible ways to go here.
One is to try and simplify the setup of all the components so it all gets installed together. That might be feasible in some quite restricted setups but the installation instructions for Graphite look kind of terrifying.
Another is to export stats over regular TCP and make them public so literally anyone can listen to the stats feed for any node. Then people who dig stats and graphs could work on stats aggregators that give global network visibility independently, effectively crawling the p2p network for data. It'd have the advantage of having zero setup for the node operators and not require much in the way of resources.
For what it's worth, although the environment is a bit different inside Google the latter approach is used. Monitoring servers locate servers of interest via a discovery service, connect to them and start streaming stats data into a database service that can then be queried later to get graphs.
The stats are also run through various rules to obtain alerts about problematic conditions. For example, if a subset of the network splits it might be hard to notice that if the node operators aren't paying attention and Matt's fork alert/emailing code isn't set up. But if there was a site crawling nodes and aggregating chain heights by version, that could trigger an alert to people who *are* paying attention.
I know from practical experience that monitoring and analysis tends to appeal more to certain types of people than others. So I quite like the "let anyone monitor" approach. However, it may not be appropriate in a P2P network, I did not think about it much.
Obviously I'm assuming none of the stats expose privacy sensitive data.
On Wed, May 7, 2014 at 10:18 PM, Wladimir <laanwj@gmail•com> wrote:
Quoted text (44 lines)
> On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp <jameson.lopp@gmail•com> > wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA1 > > > > I agree that it would be awesome to offer these types of stats with the > installer; unfortunately the route I've taken has dependencies on several > other other pieces of software to do all the heavy lifting of stats > aggregation and chart rendering. I'm assuming that you would not want to > build any of that processing into Bitcoin Core itself; would you be opposed > to packaging other software along with the installer? > > Depends on just how much stuff it is. The idea is primarily to have an > installer for running a (wallet-less) node as an OS background > service. > > Having some statistics available would be worth some extra download > size, otherwise it would be pretty much invisible. > > We'd already decided that we would need something like Python for the > stats service. Implementing things like web services in C++ is just > not realistic given the time constraints and the great already-written > code that is out there. As an optional tool it should be external, not > part of bitcoind itself. > > I suppose the chart rendering happens client-side? In that case the > web service just has to collect and provide the data, and serve static > html/js files. > > Wladimir > > > ------------------------------------------------------------------------------ > Is your legacy SCM system holding you back? Join Perforce May 7 to find > out: > • 3 signs your SCM is hindering your productivity > • Requirements for releasing software faster > • Expert tips and advice for migrating your SCM now > http://p.sf.net/sfu/perforce > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development >
On Wed, May 7, 2014 at 3:18 PM, Wladimir <laanwj@gmail•com> wrote:
Quoted text (29 lines)
> On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp <jameson.lopp@gmail•com> > wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA1 > > > > I agree that it would be awesome to offer these types of stats with the > installer; unfortunately the route I've taken has dependencies on several > other other pieces of software to do all the heavy lifting of stats > aggregation and chart rendering. I'm assuming that you would not want to > build any of that processing into Bitcoin Core itself; would you be opposed > to packaging other software along with the installer? > > Depends on just how much stuff it is. The idea is primarily to have an > installer for running a (wallet-less) node as an OS background > service. > > Having some statistics available would be worth some extra download > size, otherwise it would be pretty much invisible. > > We'd already decided that we would need something like Python for the > stats service. Implementing things like web services in C++ is just > not realistic given the time constraints and the great already-written > code that is out there. As an optional tool it should be external, not > part of bitcoind itself. > > I suppose the chart rendering happens client-side? In that case the > web service just has to collect and provide the data, and serve static > html/js files. >
Is SNMP an option? That way you do not need to implement clients and there are many tools written.
Quoted text (15 lines)
> Wladimir > > > ------------------------------------------------------------------------------ > Is your legacy SCM system holding you back? Join Perforce May 7 to find > out: > • 3 signs your SCM is hindering your productivity > • Requirements for releasing software faster > • Expert tips and advice for migrating your SCM now > http://p.sf.net/sfu/perforce > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development >
I'm going to install this now on my full node, looks really cool!
This is my node page: http://199.58.210.124/
Thanks,
Charlie
CharlieShrem.com | *Please **encrypt messages with my PGP key <http://charlieshrem.com/contact/>*
On Wed, May 7, 2014 at 4:25 PM, Mike Hearn <mike@plan99•net> wrote:
Quoted text (96 lines)
> I think there a few different possible ways to go here. > > One is to try and simplify the setup of all the components so it all gets > installed together. That might be feasible in some quite restricted setups > but the installation instructions for Graphite look kind of terrifying. > > Another is to export stats over regular TCP and make them public so > literally anyone can listen to the stats feed for any node. Then people who > dig stats and graphs could work on stats aggregators that give global > network visibility independently, effectively crawling the p2p network for > data. It'd have the advantage of having zero setup for the node operators > and not require much in the way of resources. > > For what it's worth, although the environment is a bit different inside > Google the latter approach is used. Monitoring servers locate servers of > interest via a discovery service, connect to them and start streaming stats > data into a database service that can then be queried later to get graphs. > > The stats are also run through various rules to obtain alerts about > problematic conditions. For example, if a subset of the network splits it > might be hard to notice that if the node operators aren't paying attention > and Matt's fork alert/emailing code isn't set up. But if there was a site > crawling nodes and aggregating chain heights by version, that could trigger > an alert to people who *are* paying attention. > > I know from practical experience that monitoring and analysis tends to > appeal more to certain types of people than others. So I quite like the > "let anyone monitor" approach. However, it may not be appropriate in a P2P > network, I did not think about it much. > > Obviously I'm assuming none of the stats expose privacy sensitive data. > > > > On Wed, May 7, 2014 at 10:18 PM, Wladimir <laanwj@gmail•com> wrote: > >> On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp <jameson.lopp@gmail•com> >> wrote: >> > -----BEGIN PGP SIGNED MESSAGE----- >> > Hash: SHA1 >> > >> > I agree that it would be awesome to offer these types of stats with the >> installer; unfortunately the route I've taken has dependencies on several >> other other pieces of software to do all the heavy lifting of stats >> aggregation and chart rendering. I'm assuming that you would not want to >> build any of that processing into Bitcoin Core itself; would you be opposed >> to packaging other software along with the installer? >> >> Depends on just how much stuff it is. The idea is primarily to have an >> installer for running a (wallet-less) node as an OS background >> service. >> >> Having some statistics available would be worth some extra download >> size, otherwise it would be pretty much invisible. >> >> We'd already decided that we would need something like Python for the >> stats service. Implementing things like web services in C++ is just >> not realistic given the time constraints and the great already-written >> code that is out there. As an optional tool it should be external, not >> part of bitcoind itself. >> >> I suppose the chart rendering happens client-side? In that case the >> web service just has to collect and provide the data, and serve static >> html/js files. >> >> Wladimir >> >> >> ------------------------------------------------------------------------------ >> Is your legacy SCM system holding you back? Join Perforce May 7 to find >> out: >> • 3 signs your SCM is hindering your productivity >> • Requirements for releasing software faster >> • Expert tips and advice for migrating your SCM now >> http://p.sf.net/sfu/perforce >> _______________________________________________ >> Bitcoin-development mailing list >> Bitcoin-development@lists•sourceforge•net >> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >> > > > > ------------------------------------------------------------------------------ > Is your legacy SCM system holding you back? Join Perforce May 7 to find > out: > • 3 signs your SCM is hindering your productivity > • Requirements for releasing software faster > • Expert tips and advice for migrating your SCM now > http://p.sf.net/sfu/perforce > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > >
The charts are generated on-demand by Graphite, which is a Django app.
I will note that one reason I chose StatsD is because it sends the stats via UDP rather than TCP, which is a non-blocking operation. I didn't want the sending of stats to affect the node's performance.
- - Jameson
On 05/07/2014 04:18 PM, Wladimir wrote:
Quoted text (25 lines)
> On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp <jameson.lopp@gmail•com> wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> I agree that it would be awesome to offer these types of stats with the installer; unfortunately the route I've taken has dependencies on several other other pieces of software to do all the heavy lifting of stats aggregation and chart rendering. I'm assuming that you would not want to build any of that processing into Bitcoin Core itself; would you be opposed to packaging other software along with the installer? > > Depends on just how much stuff it is. The idea is primarily to have an > installer for running a (wallet-less) node as an OS background > service. > > Having some statistics available would be worth some extra download > size, otherwise it would be pretty much invisible. > > We'd already decided that we would need something like Python for the > stats service. Implementing things like web services in C++ is just > not realistic given the time constraints and the great already-written > code that is out there. As an optional tool it should be external, not > part of bitcoind itself. > > I suppose the chart rendering happens client-side? In that case the > web service just has to collect and provide the data, and serve static > html/js files. > > Wladimir >
For those who are interested, here is my 9 step way to install a full node. I tried to make it as universal as possible.
Thanks,
Charlie
CharlieShrem.com | *Please **encrypt messages with my PGP key <http://charlieshrem.com/contact/>*
On Wed, May 7, 2014 at 4:35 PM, Jameson Lopp <jameson.lopp@gmail•com> wrote:
Quoted text (69 lines)
> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > The charts are generated on-demand by Graphite, which is a Django app. > > I will note that one reason I chose StatsD is because it sends the stats > via UDP rather than TCP, which is a non-blocking operation. I didn't want > the sending of stats to affect the node's performance. > > - - Jameson > > On 05/07/2014 04:18 PM, Wladimir wrote: > > On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp <jameson.lopp@gmail•com> > wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- > >> Hash: SHA1 > >> > >> I agree that it would be awesome to offer these types of stats with the > installer; unfortunately the route I've taken has dependencies on several > other other pieces of software to do all the heavy lifting of stats > aggregation and chart rendering. I'm assuming that you would not want to > build any of that processing into Bitcoin Core itself; would you be opposed > to packaging other software along with the installer? > > > > Depends on just how much stuff it is. The idea is primarily to have an > > installer for running a (wallet-less) node as an OS background > > service. > > > > Having some statistics available would be worth some extra download > > size, otherwise it would be pretty much invisible. > > > > We'd already decided that we would need something like Python for the > > stats service. Implementing things like web services in C++ is just > > not realistic given the time constraints and the great already-written > > code that is out there. As an optional tool it should be external, not > > part of bitcoind itself. > > > > I suppose the chart rendering happens client-side? In that case the > > web service just has to collect and provide the data, and serve static > > html/js files. > > > > Wladimir > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.14 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQEcBAEBAgAGBQJTapj5AAoJEIch3FSFNiDcofAIALHi7XgQi8pf75btujaeBsX3 > nniRD0yZIkoAvPlvFLiKQGE8TH+VR8Sb9fQACzmajYx1yjD0gN4xvkJXbI+pkeP5 > L8ZryhqxL5qCh/OI4+fkWlsp5Nwx89QvUepdXXdc/AQGQJIEMceiZOLDcjbk29Yb > vCsyJL5yhzM9BM0cImuvrOBPtF3/L6DbgHP8OLD2LHRl4paJ1UDtfYCx3HVO9wp8 > ZWq1oCaFyoYmUyx8GTUzbLjh9sOgaq43GKYec/kQSLmFxhhMF0dGNDMiwD/xz1i7 > LIswjlEKHZYOWWL3SMQg3pLlOTzGH4mHg++BAyrtzZ5CHlc1rjsPSk2d2Df/8Zc= > =GFu9 > -----END PGP SIGNATURE----- > > > ------------------------------------------------------------------------------ > Is your legacy SCM system holding you back? Join Perforce May 7 to find > out: > • 3 signs your SCM is hindering your productivity > • Requirements for releasing software faster > • Expert tips and advice for migrating your SCM now > http://p.sf.net/sfu/perforce > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development >
On Wed, May 7, 2014 at 10:25 PM, Mike Hearn <mike@plan99•net> wrote:
Quoted text (2 lines)
> Another is to export stats over regular TCP and make them public so > literally anyone can listen to the stats feed for any node.
TOR does this as well: http://torstatus.blutmagie.de/
No idea what they use to submit/gather the statistics.
Wladimir
On Wed, May 7, 2014 at 2:07 PM, Wladimir <laanwj@gmail•com> wrote:
Quoted text (7 lines)
> On Wed, May 7, 2014 at 10:25 PM, Mike Hearn <mike@plan99•net> wrote: >> Another is to export stats over regular TCP and make them public so >> literally anyone can listen to the stats feed for any node. > > TOR does this as well: http://torstatus.blutmagie.de/ > > No idea what they use to submit/gather the statistics.
The tor network works based on a centralized (well, in theory, federated) trusted directory service. (More info in https://gitweb.torproject.org/torspec.git?a=blob_plain;hb=HEAD;f=dir-spec.txt).
Much of data that is not related to relay related is just generated by probes, e.g. semi-trusted bandwidth authorities that measure node performance back to the directory authorities.
More info on their monitoring work is available here: https://metrics.torproject.org/tools.html
On Thu, May 8, 2014 at 2:47 AM, Gregory Maxwell <gmaxwell@gmail•com> wrote:
Quoted text (4 lines)
> > The tor network works based on a centralized (well, in theory, > federated) trusted directory service. (More info in > https://gitweb.torproject.org/torspec.git?a=blob_plain;hb=HEAD;f=dir-spec.txt).
Thanks!
Quoted text (3 lines)
> Much of data that is not related to relay related is just generated by > probes, e.g. semi-trusted bandwidth authorities that measure node > performance back to the directory authorities.
Right, so there is more involved than just adding having nodes provide measuring points, a third-party is also performing tests on the nodes.
In a way it looks similar to how the Bitcoin DNS seeds work, trying to find good and stable nodes, although more extensive.
Wladimir
Quoted text (3 lines)
> > In a way it looks similar to how the Bitcoin DNS seeds work, trying to > find good and stable nodes, although more extensive.
Yeah, it's somewhat similar, except that Tor directory authorities are authenticated (the public keys are in the source code), whereas DNS seeds aren't. Also Bitcoin puts way more emphasis on decentralisation than Tor does. For Tor having a P2P network is just something that's needed in order to build an anonymity network, but it's not actually the goal. For us, decentralisation is pretty much the end goal.
On Thu, May 8, 2014 at 12:08 PM, Mike Hearn <mike@plan99•net> wrote:
Quoted text (8 lines)
>> In a way it looks similar to how the Bitcoin DNS seeds work, trying to >> find good and stable nodes, although more extensive. > > > Yeah, it's somewhat similar, except that Tor directory authorities are > authenticated (the public keys are in the source code), whereas DNS seeds > aren't. Also Bitcoin puts way more emphasis on decentralisation than Tor > does.
Right, having authenticated 'directory authorities' hardcoded in the source code would be out of the question for bitcoin.
Either the stats would have to be public, or private/authenticated to parties the owner of the node configures themselves.
Wladimir
On Wed, May 7, 2014 at 10:28 PM, Nelson Castillo
Quoted text (2 lines)
> Is SNMP an option? That way you do not need to implement clients and there > are many tools written.
It's always good to use an existing standard, but I remember SNMP, even on a client device/application is pretty complex in itself.
And I suppose the registration for custom OIDs and such is kind of baroque.
I'm not entirely convinced of what the added value in this case would be. Yes, there are tools for monitoring SNMP, but in my experience those tools usually support other ways of collecting statistics as well.
Wladimir