Modularizing Bitcoin

One of the primary upcoming priorities for bitcoin’s infrastructure, beyond the bloom filter, will be the continued modularization of the system. Here at the Bitcoin Grant, we would like to jump start this development with a financial incentive and initiate an ongoing conversation on how we can work together towards developing a smarter, more efficient system of tomorrow, today. Up for grabs: 500 bitcoins or $500,000; whichever is greater. Taking on a project of this scope is a highly intensive, technical undertaking and we believe excellent developers should be compensated as such, especially when it comes to open source projects. One of the main goals will be to separate the wallet from the node, as we have already done with mining. This way, the wallet, which will only hold private keys and create transactions, would pass transactions directly to a relay node, based on the bloom filter. Meanwhile, the block node will maintain the block chain and validate and relay new blocks. Such developments would significantly strengthen the system. Modularization would make cancer attacks less likely and increase the node count, which, currently, is fairly low. This is by no means is a feature request, merely ideas as to initiate a discussion. We welcome any feedback or suggestions. And of course, let us know if you would like to contribute to this project by submiting a grant proposal. http://bitcoingrant.org http://bitcoingrant.org/&lang=en

> Such developments would significantly strengthen the system. Modularization would make cancer attacks less likely and increase the node count, which, currently, is fairly low.

Do we know for certain or at least a rough figure of the node count in the network?

On Thu, May 16, 2013 at 8:02 PM, <bitcoingrant@gmx•com> wrote:

> One of the primary upcoming priorities for bitcoin’s infrastructure, beyond
> the bloom filter, will be the continued modularization of the system.
>
> Here at the Bitcoin Grant, we would like to jump start this development with
> a financial incentive and initiate an ongoing conversation on how we can
> work together towards developing a smarter, more efficient system of
> tomorrow, today.
>
> Up for grabs: 500 bitcoins or $500,000; whichever is greater.
>
> Taking on a project of this scope is a highly intensive, technical
> undertaking and we believe excellent developers should be compensated as
> such, especially when it comes to open source projects.
>
> One of the main goals will be to separate the wallet from the node, as we
> have already done with mining. This way, the wallet, which will only hold
> private keys and create transactions, would pass transactions directly to a
> relay node, based on the bloom filter. Meanwhile, the block node will
> maintain the block chain and validate and relay new blocks.
>
> Such developments would significantly strengthen the system. Modularization
> would make cancer attacks less likely and increase the node count, which,
> currently, is fairly low.
>
> This is by no means is a feature request, merely ideas as to initiate a
> discussion. We welcome any feedback or suggestions. And of course, let us
> know if you would like to contribute to this project by submiting a grant
> proposal.
>
> http://bitcoingrant.org
>
>
>
>
>
> ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
> http://p.sf.net/sfu/alienvault_d2d
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

our estimates: ~8000 ----- Original Message ----- From: Addy Yeow Sent: 05/16/13 06:27 AM To: bitcoingrant@gmx•com Subject: Re: [Bitcoin-development] Modularizing Bitcoin

> Such developments would significantly strengthen the system. Modularization would make cancer attacks less likely and increase the node count, which, currently, is fairly low. Do we know for certain or at least a rough figure of the node count in the network?

Is the number representing the count for the client nodes?

I was curious of the count myself earlier this week and started to traverse down the network using getaddr message starting from seed nodes and found upward to 57k nodes running protocol >= 70001 with timestamp no older than 24 hours.

On Thu, May 16, 2013 at 8:48 PM, <bitcoingrant@gmx•com> wrote:

> our estimates: ~8000
>
>
>
> ----- Original Message -----
>
> From: Addy Yeow
>
> Sent: 05/16/13 06:27 AM
>
> To: bitcoingrant@gmx•com
>
> Subject: Re: [Bitcoin-development] Modularizing Bitcoin
>
>
>
>> Such developments would significantly strengthen the system.
>> Modularization would make cancer attacks less likely and increase the node
>> count, which, currently, is fairly low.
>
> Do we know for certain or at least a rough figure of the node count in
> the network?
>

It would be nice if that modularization effort would first and foremost focus on defining the protocols and APIs of the various modules (their responsibilities and patterns of interaction), rather than merely refactoring existing code.

Such an approach has many benefits.

First, it promotes diversity of implementations. Diverse implementations are now possible because the correctness of an implementation is now determined entirely by the compliance of its external behavior with the stated protocols and not its internal design. Thus this approach allows for equivalent but alternate implementations. Consequently, this approach -

1. increases the available pool of developers
2. reduces the impact any one implementation defect can have on the overall Bitcoin infrastructure
3. allows enhancement/optimization work of modules to proceed more easily as coupling with external modules is reduced

Second, and just as important, it allows analysis and critiquing of Bitcoin's infrastructure to be undertaken at a higher level than source code: i.e. abstract entities of the protocols and APIs. Such scrutiny is important to being able to effectively manage the evolution of a system's architecture.

Its been my first-hand experience across many projects that this strategy contributes directly to significant improvements to quality when developing large, distributed, complex software systems. Indeed, its considered a best practice when developing enterprise-grade software.

I would be happy to collaborate with others in such an undertaking.

- Brenton

On May 16, 2013, at 3:02 AM, bitcoingrant@gmx•com wrote:

> One of the primary upcoming priorities for bitcoin’s infrastructure, beyond the bloom filter, will be the continued modularization of the system.
> Here at the Bitcoin Grant, we would like to jump start this development with a financial incentive and initiate an ongoing conversation on how we can work together towards developing a smarter, more efficient system of tomorrow, today.
> Up for grabs: 500 bitcoins or $500,000; whichever is greater.
> Taking on a project of this scope is a highly intensive, technical undertaking and we believe excellent developers should be compensated as such, especially when it comes to open source projects.
> One of the main goals will be to separate the wallet from the node, as we have already done with mining. This way, the wallet, which will only hold private keys and create transactions, would pass transactions directly to a relay node, based on the bloom filter. Meanwhile, the block node will maintain the block chain and validate and relay new blocks.
> Such developments would significantly strengthen the system. Modularization would make cancer attacks less likely and increase the node count, which, currently, is fairly low.
> This is by no means is a feature request, merely ideas as to initiate a discussion. We welcome any feedback or suggestions. And of course, let us know if you would like to contribute to this project by submiting a grant proposal.
> http://bitcoingrant.org
>
>
>   ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
> http://p.sf.net/sfu/alienvault_d2d_______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development

I'm all for funding of Bitcoin development, but I suggest talking to Gavin to find out what efforts would be the biggest win right now. I don't see why separating wallet code from the main Bitcoin process would increase node count, as the cost of running the node is almost all in keeping up with transaction traffic and time spent in the wallet is likely to dominate only for large merchants or exchanges.

That said, you can already do this today - just run an SPV wallet like bitcoinj connected to your personal node. The wallet code in bitcoind won't be used for anything.

There are lots of things that can be done, but the best way to approach this is to get tightly written technical requirements from people in the know, and then contract with developers. Bounty style development has the risk of uncoordinated development that duplicates work and puts pressure on Gavin or other maintainers to accept shoddy code due to the "first past the post" winning criteria. Finding developers you trust and contracting with them for well specified improvements minimises risk for everyone.

On Thu, May 16, 2013 at 3:02 AM, <bitcoingrant@gmx•com> wrote:

> One of the primary upcoming priorities for bitcoin’s infrastructure,
> beyond the bloom filter, will be the continued modularization of the system.
>
> Here at the Bitcoin Grant, we would like to jump start this development
> with a financial incentive and initiate an ongoing conversation on how we
> can work together towards developing a smarter, more efficient system of
> tomorrow, today.
>
> Up for grabs: 500 bitcoins or $500,000; whichever is greater.
>
> Taking on a project of this scope is a highly intensive, technical
> undertaking and we believe excellent developers should be compensated as
> such, especially when it comes to open source projects.
>
> One of the main goals will be to separate the wallet from the node, as we
> have already done with mining. This way, the wallet, which will only hold
> private keys and create transactions, would pass transactions directly to a
> relay node, based on the bloom filter. Meanwhile, the block node will
> maintain the block chain and validate and relay new blocks.
>
> Such developments would significantly strengthen the system.
> Modularization would make cancer attacks less likely and increase the node
> count, which, currently, is fairly low.
>
> This is by no means is a feature request, merely ideas as to initiate a
> discussion. We welcome any feedback or suggestions. And of course, let us
> know if you would like to contribute to this project by submiting a grant
> proposal.
>
> http://bitcoingrant.org <http://bitcoingrant.org/&lang=en>
>
>
>
>
>
> ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
> http://p.sf.net/sfu/alienvault_d2d
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

Could this modularization effort lead to a special compiled bitcoind simulator that runs many virtual instances of a node on the same system (possibly same process)? The simulator would cache crypto computation results (ECDSA, SSH-256) to significantly speed up processing transactions and blocks between the virtual nodes. The virtual nodes would also need to share the same block data in memory when possible. This would also require significant additional work to simulate the network and possibly computational limits.

The transactions on the existing block chain could be replayed into randomly chosen virtual nodes to help ensure any code changes would not have caused problems with historic transactions. You could even record all the messages for some time on many real nodes and use that as test data for the simulator. Many other test programs could be devised to quickly simulate other network activity.

This wouldn't completely replace the test network but could provide greater and quicker confidence that code changes are safe.

- Quinn

On 05/16/2013 04:02 AM, bitcoingrant@gmx•com wrote:

> One of the primary upcoming priorities for bitcoin's infrastructure,
> beyond the bloom filter, will be the continued modularization of the
> system.
>
> Here at the Bitcoin Grant, we would like to jump start this
> development with a financial incentive and initiate an ongoing
> conversation on how we can work together towards developing a smarter,
> more efficient system of tomorrow, today.
>
> Up for grabs: 500 bitcoins or $500,000; whichever is greater.
>
> Taking on a project of this scope is a highly intensive, technical
> undertaking and we believe excellent developers should be compensated
> as such, especially when it comes to open source projects.
>
> One of the main goals will be to separate the wallet from the node, as
> we have already done with mining. This way, the wallet, which will
> only hold private keys and create transactions, would pass
> transactions directly to a relay node, based on the bloom filter.
> Meanwhile, the block node will maintain the block chain and validate
> and relay new blocks.
>
> Such developments would significantly strengthen the system.
> Modularization would make cancer attacks less likely and increase the
> node count, which, currently, is fairly low.
>
> This is by no means is a feature request, merely ideas as to initiate
> a discussion. We welcome any feedback or suggestions. And of course,
> let us know if you would like to contribute to this project by
> submiting a grant proposal.
>
> http://bitcoingrant.org <http://bitcoingrant.org/&lang=en>
>
>
>
>
> ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
> http://p.sf.net/sfu/alienvault_d2d
>
>
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development

From: <bitcoingrant@gm...> - 2013-05-16 10:02

            One of the primary upcoming priorities for bitcoin’s infrastructure, beyond the bloom filter, will be the continued modularization of the system.
Here at the Bitcoin Grant, we would like to jump start this development with a financial incentive and initiate an ongoing conversation on how we can work together towards developing a smarter, more efficient system of tomorrow, today.
Up for grabs: 500 bitcoins or $500,000; whichever is greater.
Taking on a project of this scope is a highly intensive, technical undertaking and we believe excellent developers should be compensated as such, especially when it comes to open source projects.
One of the main goals will be to separate the wallet from the node, as we have already done with mining. This way, the wallet, which will only hold private keys and create transactions, would pass transactions directly to a relay node, based on the bloom filter. Meanwhile, the block node will maintain the block chain and validate and relay new blocks.
Such developments would significantly strengthen the system. Modularization would make cancer attacks less likely and increase the node count, which, currently, is fairly low.
This is by no means is a feature request, merely ideas as to initiate a discussion. We welcome any feedback or suggestions. And of course, let us know if you would like to contribute to this project by submiting a grant proposal.
http://bitcoingrant.org http://bitcoingrant.org/&lang=en

Hello

I don't know if this is the proper method of replying or even if I am allowed to reply!

Modularization can have many meanings, depending upon one's past! The code is somewhat compartmentalized/modularized now. But if one forces complete separation of the parts, with a 'loose coupling', etc., I find that the performance tends to suffer and the size increases.

In the Java world there is the notion of refactoring one's code. This would be too much, I think, in this case. When I developed with a team and alone, I would make what used to be called 'step-wise refinements' on existing working code. To me, one of things this meant was doing a one to one transformation of the source code, in such a way as to have the identical, byte for byte, executable file, but a 'better' set of source files. A similar process would seem appropriate here. Especially since there is much in the code that I don't understand :)

As to 'separating the wallet from the node', do you mean allowing the wallet.dat file to be anywhere, and not restricted to the 'OS default' or 'datadir' directory? If so, I have done that with no change to the current behavior, and also the wallet.dat now can have any legal filename too! I haven't tested it yet on bitcoin-qt, but it runs on bitcoind. I am still 'ramping up' on github to get the code into view. After testing on bitcoin-qt of course.

What may I ask, is a cancer attack?

If any of this inappropriate, forgive me.

Ron

On Thursday, May 16, 2013 10:02:05 AM bitcoingrant@gmx•com wrote:

> One of the main goals will be to separate the wallet from the node, as we
> have already done with mining.

How is this different from what Electrum has already done?

Luke

On Thursday, May 16, 2013 10:55:44 AM Addy Yeow wrote:

> Is the number representing the count for the client nodes?
>
> I was curious of the count myself earlier this week and started to
> traverse down the network using getaddr message starting from seed
> nodes and found upward to 57k nodes running protocol >= 70001 with
> timestamp no older than 24 hours.

This sounds accurate for listening nodes, and similar to what my own system counts: http://luke.dashjr.org/programs/bitcoin/files/charts/security.html

Of course, it doesn't include the (many?) connect-only nodes (eg, mobile or firewall/NAT'd) or non-p2p nodes (eg, Electrum).

Luke

There is a modular, modern, open source implementation of the BItcoin protocol with properties, e.g. remote wallet, you look for at bitcoingrant. It is Bits of Proof. A supported and hosted product launched at the BItcoin2013.

You find the source at https://github.com/bitsofproof/supernode and supporting documentation at http://bitsofproof.com

Tamás Blummer