Horizontal scaling of blockchain

Hi,

I was thinking about how to scale the block-chain.

The fundamental problem is that if the miners add capacity it will only increase (protect) their share of block reward, but does not increase the speed of transactions. This will only raise the fee in the long run with the block reward decreasing. The throughput is limited by the block size and the complexity. Changing any of variables in the above equation was raised already many times and there was no consensus on them.

The current chain is effectively single threaded. If we look around in the sw industry how single threaded applications could be scaled, one viable option emerge: horizontal scaling. This is an option if the problem can be partitioned, and transactions in partitions could be processed alongside. Number of partitions would be start with a fairly low number, something between 2-10, but nothing is against setting it to a higher number later on according to a schedule.

Partitioning key alternatives: *Ordering on inputs:*

1) In this case transactions would need to be mined per input address partition. 2) TX have inputs in partition 1 and 2, then needs a confirmation in both partitions. 3) All partitioned chains have the same longest valid approach. 4) Only one chain needed to be considered for double spending, others are invalid in case they contain that input.

This opens up questions like: - how the fee will be shared? Fees per partition? - Ensure a good hash function which spreads evenly, because the inputs cannot be manipulated for load balancing - What to do about half mined transactions (Maybe they should be two transactions and then there is less effect about it, but payment won't be atomic in both partitions)

*Ordering on transaction ids:*

1) Transactions would be partitioned by their TX-id. Maybe a field allowing txid-s to match a given partition. 2) Creating blocks like this parallel would be safe for bonefide transactions. A block will be created each 10 mins. 3) In order to get malicious/doublespent transactions out of the system another layer must be introduced. - This layer would be used to merge the parallel blocks. It would have to refer all previous blocks considered for unspent inputs. - Most of the blocks will merge just fine as normally block 1 and block 2 would not contain double spending. (of course malicious double spending will revert speed to current levels, because the miner might have to drop a block in the partition because it contains a spent input on another stronger branch) - The standard longest chain wins strategy would be used for validity on the meta level - Meta does not require mining, a branches can be added and they are valid unless there are double spent inputs inside. Block inside this meta already "paid for".

Generally both ways would have an effect on the block reward and complexity, which is needs to be adjusted. (not to create more BTC at the end, reduced hashing power on partitions.) I think complexity is not an issue, the important thing is that we tune it to 10mins / block rate per partition.

Activation could be done by creating the infrastructure first and using only one partitions only, which is effectively the same as today. Then activate partitions on a certain block according to a schedule. From that block, partition enforcement will be active and the transactions will be sorted to the right partition / chain.

It is easy to make new partitions, just need to activate them on branch block number. Closing partitions is a bit more complex in case of TX partitioned transactions, but managed by the meta layer and activated at a certain partition block. Maybe it is not even possible in case of input partitions.

I could imagine that it is too big change. Many cons and pros on partition keys.

What is your opinion about it?

Cheers,

Tamas

On Friday, 1 September 2017 14:47:08 CEST Cserveny Tamas via bitcoin-dev wrote:

> The fundamental problem is that if the miners add capacity it will only
> increase (protect) their share of block reward, but does not increase the
> speed of transactions.

Adding more space in blocks has no effect on the block-reward. It does actually increase the throughput speed of transactions.

> This will only raise the fee in the long run with
> the block reward decreasing.

Also this is exactly the opposite of what actually happened.

> The current chain is effectively single threaded.

This is not true, since xthin/compactblocks have been introduced we completely removed this bottle-neck. The transactions will be validated continuously, in parallel and not just when a block is found.

> > The current chain is effectively single threaded.
>
> This is not true, since xthin/compactblocks have been introduced we
> completely removed this bottle-neck.
> The transactions will be validated continuously, in parallel and not just
> when a block is found.
>

If I understood correctly, OP was not talking about the process inside a node being single threaded, but instead that the whole bitcoin distributed system behaves as single threaded computation. OP seems to be describing a system closer to what IOTA uses, by distributing among the miners the task of validating the transactions. Although, without more specific details, it is hard to judge the benefits.

On 01/09/17 01:15 PM, Lucas Clemente Vella via bitcoin-dev wrote:

>
>     > The current chain is effectively single threaded.
>
>     This is not true, since xthin/compactblocks have been introduced we
>     completely removed this bottle-neck.
>     The transactions will be validated continuously, in parallel and
>     not just
>     when a block is found.
>
>
> If I understood correctly, OP was not talking about the process inside
> a node being single threaded, but instead that the whole bitcoin
> distributed system behaves as single threaded computation. OP seems to
> be describing a system closer to what IOTA uses, by distributing among
> the miners the task of validating the transactions. Although, without
> more specific details, it is hard to judge the benefits.
>

If the goal is reducing the delay for validation then I don't get what advantage there would be vs. reducing the difficulty.

Also it is my understanding that with the Lightning network transactions could be validated instantly by third parties and could be subject to smaller fees overall...

Regards,

Yes. I meant the single thread as an analogy, if a block is found, other blocks are worthless. (more or less) Longest chain wins.

My line of though was, that currently the only way to scale with the traffic (and lowering the fees) is increasing the block size (which is hard as I learned from recent events), or reducing the complexity which is less secure (most likely more controversial)

Splitting the chain is effectively increasing the block size, but without the increased hashing and validation overhead.

The usage growth seems to be more of exponential rather than linear. Sooner or later the block size will need to be 4 mb then 40 mb, then what is the real limit? Otherwise waiting times and thus the fees will just grow rapidly. I don't think that it is desirable.

With splitting the ledger, the block size can remain 1-2 mb for long time, only new partitions needs to be added on a schedule. This would also make better use of the hashing capacity.

Cheers,

Tamas

On Fri, Sep 1, 2017 at 7:15 PM Lucas Clemente Vella <lvella@gmail•com> wrote:

> > The current chain is effectively single threaded.
>>
>> This is not true, since xthin/compactblocks have been introduced we
>> completely removed this bottle-neck.
>> The transactions will be validated continuously, in parallel and not just
>> when a block is found.
>>
>
> If I understood correctly, OP was not talking about the process inside a
> node being single threaded, but instead that the whole bitcoin distributed
> system behaves as single threaded computation. OP seems to be describing a
> system closer to what IOTA uses, by distributing among the miners the task
> of validating the transactions. Although, without more specific details, it
> is hard to judge the benefits.
>
> --
> Lucas Clemente Vella
> lvella@gmail•com
>

On 01/09/17 02:15 PM, Cserveny Tamas via bitcoin-dev wrote:

> Yes. I meant the single thread as an analogy, if a block is found,
> other blocks are worthless. (more or less) Longest chain wins.
>
> My line of though was, that currently the only way to scale with the
> traffic (and lowering the fees) is increasing the block size (which is
> hard as I learned from recent events), or reducing the complexity
> which is less secure (most likely more controversial)
>

It wouldn't be less secure as long as you adjust the confirmation accordingly. If we decided to mine one block every minute, then your usual 6 confirmation should become 60 confirmation. In the end, the same amount of work will have been done to prove the transaction is legit and so it's just as secure... Actually, one could argue since the average hash rate over 60 block is more accurate than over 6, it's actually more secure if you also pay attention to hash rate variation as part of the confirmation... That help in the scenario a very large pool goes dark to mine a sidechain.

On Friday, 1 September 2017 20:15:53 CEST Cserveny Tamas wrote:

> The usage growth seems to be more of exponential rather than linear.
> Sooner or later the block size will need to be 4 mb then 40 mb, then what
> is the real limit? Otherwise waiting times and thus the fees will just
> grow rapidly. I don't think that it is desirable.

The real limit is set by the technology. Just like in 1990 we could not fathom having something like YouTube and high-res video streaming (Netflix), the limits of what is possible continually shifts.

This is basically how any successful product has ever grown, I think that it is not just desirable, it is inevitable.