About the small number of bitcoin nodes

About the small number of bitcoin nodes: Hi, I read the message that Mike Hearn sent to this mailing list some days ago (2014-04-07 11:34:43) related to the number of bitcoin full nodes.

As an owner of two Bitcoin Nodes, one in my home computer and one in a dedicated server, I believe I can contribute with some of my thoughts and ideas:

- Allow users to view the bandwith used by Bitcoin Core: This is available in the Bitcoin Core GUI (btw, when the computer is restarted the data gets reseted) but I cant find it in the bitcoind commandline, people that run nodes want to see the amount of GB that they have "donated" to the network.

- Educate users about the correct setup of a bitcoin node: Add a page in the bitcoin.org website with a tutorial about running Bitcoin Core with the ports opened, about runing bitcoind, etc. This guide shoud not be for regular users but for advanced ones.

- bitcoind and Bitcoin Core should create a bitcoin.conf file on the first start: The first time the software should create a default config file with a random RCP password and username (user can change it later) and the config file should be commented so the user can know how to change configurations. This is very useful in setups without GUI, for example in Ubuntu Server.

- bitcoind and Bitcoin Core should be in Linux repos: People want to type "yum install bitcoind" or "apt-get install bitcoind" and install bitcoin. No one wants to follow a tutorial made by somewho saying that you have to add external repos to install bitcoin in your server. For example Electrum has been added to Ubuntu software center recently. Bitcoin Core an bitcoind should be on CentOS, Debian, Ubuntu and Ubuntu Server repos.

- Create a "grafical interface" for bitcoind on Linux servers: Create a command, for example "bitcoind show" that shows a nice summary in your Terminal (Console) with all the data that a node administrator wants to know. When I say "grafical interface" I mean like "top" command, an interface made out of characters in ASCII.

- Split Bitcoin Wallet from Bitcoin Node: I believe that this is planned, some people want to help the network and others want to keep a wallet, someones want both. With bitcoind you can use the option "disablewallet=1" that allows to save some memory.

- Inform users if 8333 port is closed: That should be more visible, I dont mean an alert or warning but some icon.

- Keep connections if bitcoind is restarted: I noticed that if I restart bitcoind (to apply new config) my reset to 0 and take some hours to rise up to ~40. I believe that my peers should notice that I am down for less than ~15 minutes and try to connect again faster.

About the small number of bitcoin nodes: Hi, I read the message that Mike Hearn sent to this mailing list some days ago (2014-04-07 11:34:43) related to the number of bitcoin full nodes.

As an owner of two Bitcoin Nodes, one in my home computer and one in a dedicated server, I believe I can contribute with some of my thoughts and ideas:

- Allow users to view the bandwith used by Bitcoin Core: This is available in the Bitcoin Core GUI (btw, when the computer is restarted the data gets reseted) but I cant find it in the bitcoind commandline, people that run nodes want to see the amount of GB that they have "donated" to the network.

- Educate users about the correct setup of a bitcoin node: Add a page in the bitcoin.org website with a tutorial about running Bitcoin Core with the ports opened, about runing bitcoind, etc. This guide shoud not be for regular users but for advanced ones.

- bitcoind and Bitcoin Core should create a bitcoin.conf file on the first start: The first time the software should create a default config file with a random RCP password and username (user can change it later) and the config file should be commented so the user can know how to change configurations. This is very useful in setups without GUI, for example in Ubuntu Server.

- bitcoind and Bitcoin Core should be in Linux repos: People want to type "yum install bitcoind" or "apt-get install bitcoind" and install bitcoin. No one wants to follow a tutorial made by somewho saying that you have to add external repos to install bitcoin in your server. For example Electrum has been added to Ubuntu software center recently. Bitcoin Core an bitcoind should be on CentOS, Debian, Ubuntu and Ubuntu Server repos.

- Create a "grafical interface" for bitcoind on Linux servers: Create a command, for example "bitcoind show" that shows a nice summary in your Terminal (Console) with all the data that a node administrator wants to know. When I say "grafical interface" I mean like "top" command, an interface made out of characters in ASCII.

- Split Bitcoin Wallet from Bitcoin Node: I believe that this is planned, some people want to help the network and others want to keep a wallet, someones want both. With bitcoind you can use the option "disablewallet=1" that allows to save some memory.

- Inform users if 8333 port is closed: That should be more visible, I dont mean an alert or warning but some icon.

- Keep connections if bitcoind is restarted: I noticed that if I restart bitcoind (to apply new config) my reset to 0 and take some hours to rise up to ~40. I believe that my peers should notice that I am down for less than ~15 minutes and try to connect again faster.

Thanks for reading

On Sun, May 18, 2014 at 1:43 PM, Raúl Martínez <rme@i-rme•es> wrote:

> - Allow users to view the bandwith used by Bitcoin Core:

+1 for the sake of transparency

HOWEVER, the impact on this feature RE user population is unpredictable. Users may see bigger than expected numbers, and switch off their node.

> - Educate users about the correct setup of a bitcoin node:

+1

> - bitcoind and Bitcoin Core should create a bitcoin.conf file on the first

Meh. I like example configs, perhaps tuned by the distro. If the distro (_not_ Bitcoin Core upstream) chooses to install a bitcoin.conf in the proper location, that's up to them.

> - bitcoind and Bitcoin Core should be in Linux repos:

Agreed with conditions: 1) The distro MUST let bitcoin devs dictate which dependent libs are shipped with / built statically into the bitcoin binaries/libs. 2) The distro MUST permit fresh updates even to older stable distros. 2) The maintainer(s) MUST be active, and follow bitcoin development, release status, etc. on a near-daily basis, be able to respond quickly if security issues arise, etc.

Matt C seems to do a good job of this in Ubuntu PPA, I'm told.

> - Create a "grafical interface" for bitcoind on Linux servers:
> When I say "grafical interface" I mean like "top" command, an interface made
> out of characters in ASCII.

The best path for this is figure out what statistics you want to see, and have bitcoind export those raw numbers to any willing consumer. Then write your bitcoind-top on top of that.

> - Split Bitcoin Wallet from Bitcoin Node:

+1

In progress. Disable-wallet support, at compile time or runtime, was the first step.

> - Inform users if 8333 port is closed:

+1

> - Keep connections if bitcoind is restarted:
> I noticed that if I restart bitcoind (to apply new config) my reset to 0 and
> take some hours to rise up to ~40. I believe that my peers should notice
> that I am down for less than ~15 minutes and try to connect again faster.

No, you don't want this (and it's not possible in many cases anyway).

On 05/19/2014 09:11 AM, Jeff Garzik wrote:

> Meh.  I like example configs, perhaps tuned by the distro.  If the
> distro (_not_ Bitcoin Core upstream) chooses to install a
> bitcoin.conf in the proper location, that's up to them.
>
>
>> - bitcoind and Bitcoin Core should be in Linux repos:
>
> Agreed with conditions: 1) The distro MUST let bitcoin devs dictate
> which dependent libs are shipped with / built statically into the
> bitcoin binaries/libs. 2) The distro MUST permit fresh updates even
> to older stable distros. 2) The maintainer(s) MUST be active, and
> follow bitcoin development, release status, etc. on a near-daily
> basis, be able to respond quickly if security issues arise, etc.

If we're talking about making bitcoind behave like a proper daemon, then syslog support should be on the list.

Also, the option to store things in FHS-compliant directories (not somewhere/.bitcoin)

On Sun, May 18, 2014 at 7:43 PM, Raúl Martínez <rme@i-rme•es> wrote:

> About the small number of bitcoin nodes:
> Hi, I read the message that Mike Hearn sent to this mailing list some days
> ago (2014-04-07 11:34:43) related to the number of bitcoin full nodes.
>
> As an owner of two Bitcoin Nodes, one in my home computer and one in a
> dedicated server, I believe I can contribute with some of my thoughts and
> ideas:
>
> - Allow users to view the bandwith used by Bitcoin Core:
> This is available in the Bitcoin Core GUI (btw, when the computer is
> restarted the data gets reseted) but I cant find it in the bitcoind
> commandline

That's also possible through the RPC. See "getnettotals". You can also get stats per peer in "getpeerinfo".

I also suggest looking at Jameson Lopp's Statoshi work (http://statoshi.info) if you like graphs and more detailed stats.

> - Educate users about the correct setup of a bitcoin node:
> Add a page in the bitcoin.org website with a tutorial about running Bitcoin
> Core with the ports opened, about runing bitcoind, etc. This guide shoud not
> be for regular users but for advanced ones.

Yes, such a document would be very welcome.

Maybe coordinate with Saïvann Carignan or David Harding, it could be part of their bitcoin documentation project.

> - bitcoind and Bitcoin Core should create a bitcoin.conf file on the first
> start:
> The first time the software should create a default config file with a
> random RCP password and username (user can change it later) and the config
> file should be commented so the user can know how to change configurations.
> This is very useful in setups without GUI, for example in Ubuntu Server.

I agree with you that a default configuration file is useful, *however* this does not need to be created by the daemon.

The idea would be to make bitcoind and its data and configuration system-wide. See https://github.com/bitcoin/bitcoin/issues/4124

A daemon should not even have write access to its own configuration files. To follow the example of apache, tor, and such the distribution installs a default configuration file which the user can adapt.

> - bitcoind and Bitcoin Core should be in Linux repos:
> People want to type "yum install bitcoind" or "apt-get install bitcoind" and
> install bitcoin. No one wants to follow a tutorial made by somewho saying
> that you have to add external repos to install bitcoin in your server.
> For example Electrum has been added to Ubuntu software center recently.
> Bitcoin Core an bitcoind should be on CentOS, Debian, Ubuntu and Ubuntu
> Server repos.

This sounds good, but as usual the practice is much uglier.

Bitcoind was part of the Ubuntu default repos for a while, but they don't upgrade versions as we need to. This resulted in Ubuntu 12.04 stable being stuck with 0.3.xx forever. It would be even worse for Debian Stable, which has even older versions of packages.

So right now we need you to add the PPA to get the package for Ubuntu. This is only a small extra step.

This has to be determined per distribution, though. In some distros this may be perfectly possible. This is just another place where the project is completely dependent on volunteers.

> - Create a "grafical interface" for bitcoind on Linux servers:
> Create a command, for example "bitcoind show" that shows a nice summary in
> your Terminal (Console) with all the data that a node administrator wants to
> know.
> When I say "grafical interface" I mean like "top" command, an interface made
> out of characters in ASCII.

Sounds like a fun project for someone in Python. Most of the information is already available through RPC (and if not, request it!).

Some hacking with ncurses could quickly make a decent tool here. It could be packaged with bitcoin itself but that's not necessary. For example Tor has the tool 'arm' which is a separate package.

You may want to talk with Shawn Wilkinson he has some ideas in this direction. See also the issue https://github.com/bitcoin/bitcoin/issues/3122 .

> - Split Bitcoin Wallet from Bitcoin Node:
> I believe that this is planned, some people want to help the network and
> others want to keep a wallet, someones want both.
> With bitcoind you can use the option "disablewallet=1" that allows to save
> some memory.

Running the node without wallet is already possible since 0.9.0 in two ways:

- ./configure --disable-wallet when compiling
- run with -disablewallet  (as you say)

This works both for the GUI and the daemon. You can use the resulting node-only instance ("edge router") with any existing SPV wallet.

There are plans to split off the wallet so that it can run separately, but I wouldn't be holding my breath.

It feels to me that the general direction things are going in is that other wallet projects are advancing much faster than Bitcoin Core's wallet and people will likely switch to other wallet projects for wallet functionality. Bitcoin Core is moving to an edge router role.

I'm happy to be proven wrong here and would like to see someone work on bitcoind's wallet, but with the current development resources we have to focus on a what is most urgent: maintaining and improving the infrastructure.

> - Inform users if 8333 port is closed:
> That should be more visible, I dont mean an alert or warning but some icon.

Yes, it would be great of connectivity and proxy problems were signaled in some way.

Detecting whether your port is closed from the outside is an imprecise art at most, though, as it relies on information from others.

A first step could be showing the number of incoming and outgoing connections separately in getnetworkinfo. If you have no incoming connections after a while you can be fairly sure that there is no outward connectivity.

> - Keep connections if bitcoind is restarted:
> I noticed that if I restart bitcoind (to apply new config) my reset to 0 and
> take some hours to rise up to ~40. I believe that my peers should notice
> that I am down for less than ~15 minutes and try to connect again faster.

What incentive do peers have to reconnect to you specifically? The nature of a P2P network is that nodes are interchangeable. If a node fails or kicks them, they'll just try the next node in the list. Sometimes that will be you, sometimes it will not.

Wladimir

On 18/05/14 19:43, Raúl Martínez wrote:

> About the small number of bitcoin nodes:
> Hi, I read the message that Mike Hearn sent to this mailing list some
> days ago (2014-04-07 11:34:43) related to the number of bitcoin full nodes.
>
> As an owner of two Bitcoin Nodes, one in my home computer and one in a
> dedicated server, I believe I can contribute with some of my thoughts
> and ideas:
>
> <snip some good ideas>

As an interested party not intimately familiar with the bitcoin codebase who also spent some time setting up a node a while ago, I would like to add one thing to the above list - network rate limiting. When I first set up my node, I did not consider this until it started eating all of my upstream bandwith (on an ADSL line that isn't so fun), and I must admit I was a bit disappointed when I found that I would have to set this up separately. An option to limit the upstream and downstream network usage (like what tor or most torrent clients does) would be useful.

On the note of the distro packages, I've so far been pretty happy with the packages provided by Arch Linux - I have no idea if they build it correctly (with the correct static libs and all), but it seems to work fine for the edge router case.

Bjørn Øivind

On Mon, May 19, 2014 at 10:48 AM, Wladimir <laanwj@gmail•com> wrote:

>
> Some hacking with ncurses could quickly make a decent tool here. It
> could be packaged with bitcoin itself but that's not necessary. For
> example Tor has the tool 'arm' which is a separate package.

Regarding tor-arm, here are some screenshots: https://www.atagar.com/arm/screenshots.php

It shows, among other things: - bandwidth up/down graphs - CPU usage - debug logging (in real time) - connected peers+statistics - currently active configuration

Would be nice to have a similar tool for bitcoind.

Wladimir

Is the small number of bitcoin nodes a concern? If yes, why? What kind of attack can the network suffer? And where can we find statistical information about the full nodes running?

I guess the only effective incentive to keep a node running would be financial. A kind of proof of stake would be nice on bitcoin. But I have no idea how to begin with.

Felipe Micaroni Lalli

Walltime: https://walltime.info Bitcoin Paranoid Android developer PGP ID: 0x4c0afccfed5cde14 BTC: 1LipeR1AjHL6gwE7WQECW4a2H4tuqm768N

On 19/05/2014, at 07:39, Wladimir <laanwj@gmail•com> wrote:

> On Mon, May 19, 2014 at 10:48 AM, Wladimir <laanwj@gmail•com> wrote:
>>
>> Some hacking with ncurses could quickly make a decent tool here. It
>> could be packaged with bitcoin itself but that's not necessary. For
>> example Tor has the tool 'arm' which is a separate package.
>
> Regarding tor-arm, here are some screenshots:
> https://www.atagar.com/arm/screenshots.php
>
> It shows, among other things:
> - bandwidth up/down graphs
> - CPU usage
> - debug logging (in real time)
> - connected peers+statistics
> - currently active configuration
>
> Would be nice to have a similar tool for bitcoind.
>
> Wladimir
>
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.
> Get unparalleled scalability from the best Selenium testing platform available
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development

>
> As an interested party not intimately familiar with the bitcoin codebase
> who also spent some time setting up a node a while ago, I would like to
> add one thing to the above list - network rate limiting.

The problem is that this is easier said than done. Bitcoin Core won't notice a remote peer is working but slow and switch to a faster one, and even if it did, it'd just mean throttling your connection would cause all remote nodes to give up and hit the other unthrottled peers even more.

The best way to implement this is to do chain pruning, so your node will still try and shovel bytes as fast as possible, but it's limited by how many bytes it has to shovel. Remote nodes that are pulling down the block chain can then switch between nodes depending on what they have available in order to try and avoid hitting one node too hard. Nodes that were offline for a while and just catching up would prefer nodes that have less of the chain.

It'd be great if someone could experiment with this. The first step is extending the p2p protocol so addr broadcasts and version messages include how much of the chain (counting blocks from the head?) the peer is willing to serve, and then updating the downloading code so it tries to be smarter about peer selection. Unfortunately all this work is sort of backed up waiting for sipa to finish merging in headers-first downloading.

On 19/05/14 14:15, Mike Hearn wrote:

>     As an interested party not intimately familiar with the bitcoin codebase
>     who also spent some time setting up a node a while ago, I would like to
>     add one thing to the above list - network rate limiting.
>
>
> The problem is that this is easier said than done. Bitcoin Core won't
> notice a remote peer is working but slow and switch to a faster one, and
> even if it did, it'd just mean throttling your connection would cause
> all remote nodes to give up and hit the other unthrottled peers even more.
>

Does this mean that you can currently actively hurt the network by adding a node with a very slow upstream / downstream? If so, what is the recommended minimum amount of bandwidth you should allocate for a node? I've already throttled mine with QoS based on the script in the contrib/ folder.

Bjørn Øivind

>
> Does this mean that you can currently actively hurt the network by
> adding a node with a very slow upstream / downstream?

Well, I guess "hurting" the network is perhaps a bit dramatic. There are already lots of ways the download process can go wrong and take days. Using the torrent is much faster. But my understanding is that this will slow down the bootstrap process for some people yes. Remote peers won't try and download in parallel or anything like that.

On Mon, May 19, 2014 at 11:26 AM, Bjørn Øivind Bjørnsen <bo.bjornsen@gmail•com> wrote:

> On 18/05/14 19:43, Raúl Martínez wrote:
>> <snip some good ideas>
>
> As an interested party not intimately familiar with the bitcoin codebase
> who also spent some time setting up a node a while ago, I would like to
> add one thing to the above list - network rate limiting.

There is already an (old) patch that implements that. It won't be merged, though, until headers-first and parallel block download is in. Only when the node can download blocks from multiple peers at once it is really safe to allow limiting rates.

(sure - there are tricks to limit rates anyway, like the script in contrib/qos, but to have it generally available the block download needs to be more robust first)

Wladimir

>
> (sure - there are tricks to limit rates anyway, like the script in
> contrib/qos, but to have it generally available the block download
> needs to be more robust first)

One thing we could consider as a short term solution (if headers first+parallel downloading will take a while, which seems plausible) is to add a service bit that says "I have chain data and am willing to Bloom filter it for you, but I won't serve full block data", and then just exclude all of those from the chain download logic. It should not be a deep change to the code headers first is impacting, and would allow home users who may have no tolerance for block chain uploads at all to still take part and offer useful services to the network.

I know Pieter likes the idea of an "archival node" service bit, or something like that. I'd been thinking that the stored chain height value would be better, but perhaps we need to divorce "I have CPU and can filter" from "I have bandwidth and can serve" more vigorously.

On Mon, May 19, 2014 at 3:11 AM, Jeff Garzik <jgarzik@bitpay•com> wrote:

> On Sun, May 18, 2014 at 1:43 PM, Raúl Martínez <rme@i-rme•es> wrote:
>> - bitcoind and Bitcoin Core should be in Linux repos:
>
> Agreed with conditions:
> 1) The distro MUST let bitcoin devs dictate which dependent libs are
> shipped with / built statically into the bitcoin binaries/libs.
> 2) The distro MUST permit fresh updates even to older stable distros.
> 2) The maintainer(s) MUST be active, and follow bitcoin development,
> release status, etc. on a near-daily basis, be able to respond quickly
> if security issues arise, etc.
>
> Matt C seems to do a good job of this in Ubuntu PPA, I'm told.

Update:

(1) and (3) are doable, however, Debian and Ubuntu policies make (2) very difficult (with the exception of security patches). Micha Bailey and I worked to get bitcoin removed from Debian and Ubuntu stable releases because they would not allow (2). There are other mechanisms that could accomplish (2) (backports, volatile, and updates repositories), however they are not enabled by default and require user intervention.

Debian unstable does allow (2) since there is no release, and there is a package in Debian unstable. That package is blocked from transitioning to a stable release. We've also blacklisted it from Ubuntu so that Ubuntu doesn't just autoimport and release the Debian unstable package in an Ubuntu stable release.

Micha is also working to have all old versions of bitcoin removed from previous released Ubuntu versions.

Matt C's PPA is the best way of getting (1-3) above on Ubuntu, and the Debian unstable package is probably the best way of getting (1-3) above in Debian. Both require users to add a line to their apt sources list; the Debian package would also require apt pinning.

Regards, Scott

> - Create a "grafical interface" for bitcoind on Linux servers:
> Create a command, for example "bitcoind show" that shows a nice summary in
> your Terminal (Console) with all the data that a node administrator wants to
> know.
> When I say "grafical interface" I mean like "top" command, an interface made
> out of characters in ASCII.

FYI someone created this! It's still in the initial stages, I'm sure the author could use some help to grow this into a full-functional node admin tool.

https://github.com/azeteki/bitcoind-ncurses

Wladimir