An implementation of BIP102 as a softfork.
14 messages, 30 Dec 2015 to 12 Jan 2016. Started by joe2015.
joe2015 (5), Peter Todd (3), Jonathan Toomim (2), Marco Falke, Martijn Meijering, Marcel Jamin, Nick ODell
Below is a proof-of-concept implementation of BIP102 as a softfork:
https://github.com/ZoomT/bitcoin/tree/2015_2mb_blocksize https://github.com/jgarzik/bitcoin/compare/2015_2mb_blocksize...ZoomT:2015_2mb_blocksize?diff=split&name=2015_2mb_blocksize
BIP102 is normally a hardfork. The softfork version (unofficial codename BIP102s) uses the idea described here: http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-December/012073.html
The basic idea is that post-fork blocks are constructed in such a way they can be mapped to valid blocks under the pre-fork rules. BIP102s is a softfork in the sense that post-fork miners are still creating a valid chain under the old rules, albeit indirectly.
From the POV of non-upgraded clients, BIP102s circumvents the block-size limit by moving transaction validation data "outside" of the block. This is a similar trick used by Segregated Witness and Extension Blocks (both softfork proposals).
From the POV of upgraded clients, the block layout is unchanged, except: - A larger 2MB block-size limit (=BIP102); - The header Merkle root has a new (backwards compatible) interpretation; - The coinbase encodes the Merkle root of the remaining txs. Aside from this, blocks maintain their original format, i.e. a block header followed by a vector of transactions. This keeps the implementation simple, and is distinct from SW and EB.
Since BIP102s is a softfork it means that: - A miner majority (e.g. 75%, 95%) force miner consensus (100%). This is not true for a hardfork. - Fraud risk is significantly reduced (6-conf unlikely depending on activation threshold). This should address some of the concerns with deploying a block-size increase using a hardfork.
Notes:
- The same basic idea could be adapted to any of the other proposals (BIP101, 2-4-8, BIP202, etc.). - I used Jeff Garzik's BIP102 implementation which is incomplete (?). The activation logic is left unchanged. - I am not a Bitcoin dev so hopefully no embarrassing mistakes in my code :-(
--joe
This is an interesting approach but I don't see how this is a soft fork. (Just because something is not a hard fork, doesn't make it a soft fork by definition) Softforks don't require any nodes to upgrade. [1] Nonetheless, as I understand your approach, it requires nodes to upgrade. Otherwise they are missing all transactions but the coinbase transactions. Thus, they cannot update their utxoset and are easily susceptible to double spends...
Am I missing something obvious?
-- Marco
[1] https://en.bitcoin.it/wiki/Softfork#Implications
That looks very interesting. But is effectively blocking old clients from seeing transactions really safe? After all, such transactions are still confirmed on the new chain. A person might try to send a similar transaction several times, perhaps with increasing fees in an attempt to get it to confirm and end up paying someone several times.
Maybe we could require the tx version number to be increased as well so transactions sent from old clients would never confirm? Perhaps your code already includes this idea, I need to look at it more closely.
As a first impression, I think this proposal is intellectually interesting, but crufty and hackish and should never actually be deployed. Writing code for Bitcoin in a future in which we have deployed a few generalized softforks this way sounds terrifying.
Instead of this:
CTransaction GetTransaction(CBlock block, unsigned int index) {
return block->vtx[index];
}
We might have this:
CTransaction GetTransaction(CBlock block, unsigned int index) {
if (!IsBIP102sBlock(block)) {
return block->vtx[index];
} else {
if (!IsOtherGeneralizedSoftforkBlock(block)) {
// hooray! only one generalized softfork level to deal with!
return LookupBlock(GetGSHashFromCoinbase(block->vtx[0].vin[0].scriptSig))->vtx[index];
} else {
throw NotImplementedError; // I'm too lazy to write pseudocode this complicated just to argue a point
}
}
It might be possible to make that a bit simpler with recursion, or by doing subsequent generalized softforks in a way that doesn't have multi-levels-deep block-within-a-block-within-a-block stuff. Still: ugh.
On Dec 29, 2015, at 9:46 PM, joe2015--- via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (52 lines)
> Below is a proof-of-concept implementation of BIP102 as a softfork: > > https://github.com/ZoomT/bitcoin/tree/2015_2mb_blocksize > https://github.com/jgarzik/bitcoin/compare/2015_2mb_blocksize...ZoomT:2015_2mb_blocksize?diff=split&name=2015_2mb_blocksize > > BIP102 is normally a hardfork. The softfork version (unofficial > codename BIP102s) uses the idea described here: > http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-December/012073.html > > The basic idea is that post-fork blocks are constructed in such a way > they can be mapped to valid blocks under the pre-fork rules. BIP102s > is a softfork in the sense that post-fork miners are still creating a > valid chain under the old rules, albeit indirectly. > > From the POV of non-upgraded clients, BIP102s circumvents the > block-size limit by moving transaction validation data "outside" of > the block. This is a similar trick used by Segregated Witness and > Extension Blocks (both softfork proposals). > > From the POV of upgraded clients, the block layout is unchanged, > except: > - A larger 2MB block-size limit (=BIP102); > - The header Merkle root has a new (backwards compatible) > interpretation; > - The coinbase encodes the Merkle root of the remaining txs. > Aside from this, blocks maintain their original format, i.e. a block > header followed by a vector of transactions. This keeps the > implementation simple, and is distinct from SW and EB. > > Since BIP102s is a softfork it means that: > - A miner majority (e.g. 75%, 95%) force miner consensus (100%). This > is not true for a hardfork. > - Fraud risk is significantly reduced (6-conf unlikely depending on > activation threshold). > This should address some of the concerns with deploying a block-size > increase using a hardfork. > > Notes: > > - The same basic idea could be adapted to any of the other proposals > (BIP101, 2-4-8, BIP202, etc.). > - I used Jeff Garzik's BIP102 implementation which is incomplete (?). > The activation logic is left unchanged. > - I am not a Bitcoin dev so hopefully no embarrassing mistakes in my > code :-( > > --joe > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
I guess the same could be said about the softfork flavoured SW implementation. In any case, the strategy pattern helps with code structure in situations like this.
2015-12-30 14:29 GMT+01:00 Jonathan Toomim via bitcoin-dev < bitcoin-dev@lists•linuxfoundation•org>:
Quoted text (100 lines)
> As a first impression, I think this proposal is intellectually
> interesting, but crufty and hackish and should never actually be deployed.
> Writing code for Bitcoin in a future in which we have deployed a few
> generalized softforks this way sounds terrifying.
>
> Instead of this:
>
> CTransaction GetTransaction(CBlock block, unsigned int index) {
> return block->vtx[index];
> }
>
> We might have this:
>
> CTransaction GetTransaction(CBlock block, unsigned int index) {
> if (!IsBIP102sBlock(block)) {
> return block->vtx[index];
> } else {
> if (!IsOtherGeneralizedSoftforkBlock(block)) {
> // hooray! only one generalized softfork level to deal
> with!
> return
> LookupBlock(GetGSHashFromCoinbase(block->vtx[0].vin[0].scriptSig))->vtx[index];
> } else {
> throw NotImplementedError; // I'm too lazy to write
> pseudocode this complicated just to argue a point
> }
> }
>
> It might be possible to make that a bit simpler with recursion, or by
> doing subsequent generalized softforks in a way that doesn't have
> multi-levels-deep block-within-a-block-within-a-block stuff. Still: ugh.
>
>
>
>
> On Dec 29, 2015, at 9:46 PM, joe2015--- via bitcoin-dev <
> bitcoin-dev@lists•linuxfoundation•org> wrote:
>
> > Below is a proof-of-concept implementation of BIP102 as a softfork:
> >
> > https://github.com/ZoomT/bitcoin/tree/2015_2mb_blocksize
> >
> https://github.com/jgarzik/bitcoin/compare/2015_2mb_blocksize...ZoomT:2015_2mb_blocksize?diff=split&name=2015_2mb_blocksize
> >
> > BIP102 is normally a hardfork. The softfork version (unofficial
> > codename BIP102s) uses the idea described here:
> >
> http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-December/012073.html
> >
> > The basic idea is that post-fork blocks are constructed in such a way
> > they can be mapped to valid blocks under the pre-fork rules. BIP102s
> > is a softfork in the sense that post-fork miners are still creating a
> > valid chain under the old rules, albeit indirectly.
> >
> > From the POV of non-upgraded clients, BIP102s circumvents the
> > block-size limit by moving transaction validation data "outside" of
> > the block. This is a similar trick used by Segregated Witness and
> > Extension Blocks (both softfork proposals).
> >
> > From the POV of upgraded clients, the block layout is unchanged,
> > except:
> > - A larger 2MB block-size limit (=BIP102);
> > - The header Merkle root has a new (backwards compatible)
> > interpretation;
> > - The coinbase encodes the Merkle root of the remaining txs.
> > Aside from this, blocks maintain their original format, i.e. a block
> > header followed by a vector of transactions. This keeps the
> > implementation simple, and is distinct from SW and EB.
> >
> > Since BIP102s is a softfork it means that:
> > - A miner majority (e.g. 75%, 95%) force miner consensus (100%). This
> > is not true for a hardfork.
> > - Fraud risk is significantly reduced (6-conf unlikely depending on
> > activation threshold).
> > This should address some of the concerns with deploying a block-size
> > increase using a hardfork.
> >
> > Notes:
> >
> > - The same basic idea could be adapted to any of the other proposals
> > (BIP101, 2-4-8, BIP202, etc.).
> > - I used Jeff Garzik's BIP102 implementation which is incomplete (?).
> > The activation logic is left unchanged.
> > - I am not a Bitcoin dev so hopefully no embarrassing mistakes in my
> > code :-(
> >
> > --joe
> >
> > _______________________________________________
> > bitcoin-dev mailing list
> > bitcoin-dev@lists•linuxfoundation•org
> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation•org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>On Wed, Dec 30, 2015 at 05:29:05AM -0800, Jonathan Toomim via bitcoin-dev wrote:
Quoted text (1 line)
> As a first impression, I think this proposal is intellectually interesting, but crufty and hackish and should never actually be deployed. Writing code for Bitcoin in a future in which we have deployed a few generalized softforks this way sounds terrifying.
<snip>
Quoted text (1 line)
> It might be possible to make that a bit simpler with recursion, or by doing subsequent generalized softforks in a way that doesn't have multi-levels-deep block-within-a-block-within-a-block stuff. Still: ugh.
Your fear is misplaced: it's trivial to avoid recursion with a bit of planning.
For instance, if Bitcoin was redesigned to incorporate the forced fork concept, instead of block headers committing to just a merkle root, they could instead commit to H(version + digest)
For version == 0, digest would be a merkle root of all transactions. If the version was > 0, any digest would be allowed and the block would be interpreted as a NOP with no effect on the UTXO set.
In the event of a major change - e.g. what would otherwise be a hard-forking change to the way the merkle root was calculated - a soft-fork would change the block validity rules to make version == 0 invalid, and verison == 1 blocks would interpret the digest according to the new merkle root rules. Again, version > 1 blocks would be treated as NOPs.
A good exercise is to apply the above to the existing Bitcoin ecosystem as a soft-fork - it certainely can be done, and done right is technically very simple.
Regardless of how it's done - existing Bitcoin compatible or clean sheet redesign - you get the significant safety advantages soft-forks have over hard-forks in nearly all situations where you'd have to do a hard-fork. OTOH, it's kinda scary how this institutionalizes what could be seen as 51% attacks, possibly giving miners significantly more control over the system politically. I'm not sure I agree with that viewpoint - miners can do this anyway - but that has made people shy away from promoting this idea in the past. (previously it's been often referred to as an "evil" soft-fork)
On Wed, Dec 30, 2015 at 12:16:22PM +0100, Martijn Meijering via bitcoin-dev wrote:
Quoted text (5 lines)
> That looks very interesting. But is effectively blocking old clients from > seeing transactions really safe? After all, such transactions are still > confirmed on the new chain. A person might try to send a similar > transaction several times, perhaps with increasing fees in an attempt to > get it to confirm and end up paying someone several times.
It's very dangerous to simply send multiple transactions in such a way that they don't double-spend each other; you have no good way of knowing for sure that you're seeing the longest block chain with software alone.
Competently designed software with fee-bumping wouldn't allow that mistake to be made; the UX should make it clear that txs sent are still pending until confirmed or clearly double-spent.
Quoted text (3 lines)
> Maybe we could require the tx version number to be increased as well so > transactions sent from old clients would never confirm? Perhaps your code > already includes this idea, I need to look at it more closely.
That can mess up pre-signed transations, e.g. refunds.
On Wed, Dec 30, 2015 at 06:19:55AM -0800, Peter Todd via bitcoin-dev wrote:
Quoted text (11 lines)
> On Wed, Dec 30, 2015 at 05:29:05AM -0800, Jonathan Toomim via bitcoin-dev wrote: > > As a first impression, I think this proposal is intellectually interesting, but crufty and hackish and should never actually be deployed. Writing code for Bitcoin in a future in which we have deployed a few generalized softforks this way sounds terrifying. > > <snip> > > > It might be possible to make that a bit simpler with recursion, or by doing subsequent generalized softforks in a way that doesn't have multi-levels-deep block-within-a-block-within-a-block stuff. Still: ugh. > > Your fear is misplaced: it's trivial to avoid recursion with a bit of > planning. > > For instance, if Bitcoin was redesigned to incorporate the forced fork
Actually, a better name is probably "forced soft-fork", making this clear we're using the soft-fork mechanism to force everyone to upgrade.
On Dec 30, 2015, at 6:19 AM, Peter Todd <pete@petertodd•org> wrote:
Quoted text (2 lines)
> Your fear is misplaced: it's trivial to avoid recursion with a bit of > planning...
That makes some sense. I downgrade my emotions from "a future in which we have deployed a few generalized softforks this way sounds terrifying" to "the idea of a future in which we have deployed at least one generalized softfork this way gives me the heebie jeebies."
On 2015-12-30 18:33, Marco Falke wrote:
Quoted text (15 lines)
> This is an interesting approach but I don't see how this is a soft > fork. (Just because something is not a hard fork, doesn't make it a > soft fork by definition) > Softforks don't require any nodes to upgrade. [1] > Nonetheless, as I understand your approach, it requires nodes to > upgrade. Otherwise they are missing all transactions but the coinbase > transactions. Thus, they cannot update their utxoset and are easily > susceptible to double spends... > > Am I missing something obvious? > > -- Marco > > > [1] https://en.bitcoin.it/wiki/Softfork#Implications
It just depends how you define "softfork". In my original write-up I called it a "generalized" softfork, Peter suggested a "firm" fork, and there are some suggestions for other names. Ultimately what you call it is not very important.
--joe.
On 2016-01-03 02:46, Marco Falke wrote:
Quoted text (32 lines)
> 2015-12-30 17:27 GMT+01:00 <joe2015@openmailbox•org>: >> On 2015-12-30 18:33, Marco Falke wrote: >>> >>> This is an interesting approach but I don't see how this is a soft >>> fork. (Just because something is not a hard fork, doesn't make it a >>> soft fork by definition) >>> Softforks don't require any nodes to upgrade. [1] >>> Nonetheless, as I understand your approach, it requires nodes to >>> upgrade. Otherwise they are missing all transactions but the coinbase >>> transactions. Thus, they cannot update their utxoset and are easily >>> susceptible to double spends... >>> >>> Am I missing something obvious? >>> >>> -- Marco >>> >>> >>> [1] https://en.bitcoin.it/wiki/Softfork#Implications >> >> >> It just depends how you define "softfork". In my original write-up I >> called >> it a "generalized" softfork, Peter suggested a "firm" fork, and there >> are >> some suggestions for other names. Ultimately what you call it is not >> very >> important. >> >> --joe. > > joe, indeed it is not important how you call it, but please, let's not > call it "soft fork".
This kind of fork (whatever it is called) has all the traditional properties of a softfork except meaningful backwards compatibility for non-upgraded clients. So I think it is reasonable to call it a softfork with some qualification.
Quoted text (4 lines)
> Besides my initial question about the coinbase > tx, I was also wondering how non-updated nodes would verify the > collected fees without the actual txs at hand. (They only have the > coinbase tx, don't they?)
Yes this appears to be an oversight in my proof-of-concept implementation. The unintended consequence being that all transactions would have to be zero-fee...
The simplest fix would be make the new rules add the fees implicitly. There are other solutions.
Quoted text (5 lines)
> Moreover, I can't see the benefits over a hard fork. A hard fork is > much cleaner in regard to code changes. As one of the intends of > "generalized soft forks" is to force user to update, at least a hard > fork doesn't lie about the fact. Am I missing any obvious advantages > of a "generalized soft fork" over a "clean" hard fork?
A "firm soft fork" also does not lie about that fact -- you must upgrade. I don't see it dishonest if it was never claimed otherwise.
I agree that hardforks can be "cleaner".
However the obvious disadvantage of a hardfork is the risk of the network splitting between upgraded and non-upgraded clients. This is not a problem if there is 100% consensus behind the hardfork, but I am not sure if 100% is realistically achievable for contentious issues such as the blocksize limit.
If 100% consensus is never achieved, then the options are: 1. Never upgrade and keep the blocksize limit unchanged forever. 2. Use a firm softfork to resolve the deadlock. 3. Hardfork anyway and split the network.
My argument is simply that 2 is better than 3 and possibly 1.
--joe
How are you collecting fees from the transactions in the block?
On Sat, Jan 2, 2016 at 8:51 PM, joe2015--- via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (88 lines)
> On 2016-01-03 02:46, Marco Falke wrote: >> >> 2015-12-30 17:27 GMT+01:00 <joe2015@openmailbox•org>: >>> >>> On 2015-12-30 18:33, Marco Falke wrote: >>>> >>>> >>>> This is an interesting approach but I don't see how this is a soft >>>> fork. (Just because something is not a hard fork, doesn't make it a >>>> soft fork by definition) >>>> Softforks don't require any nodes to upgrade. [1] >>>> Nonetheless, as I understand your approach, it requires nodes to >>>> upgrade. Otherwise they are missing all transactions but the coinbase >>>> transactions. Thus, they cannot update their utxoset and are easily >>>> susceptible to double spends... >>>> >>>> Am I missing something obvious? >>>> >>>> -- Marco >>>> >>>> >>>> [1] https://en.bitcoin.it/wiki/Softfork#Implications >>> >>> >>> >>> It just depends how you define "softfork". In my original write-up I >>> called >>> it a "generalized" softfork, Peter suggested a "firm" fork, and there are >>> some suggestions for other names. Ultimately what you call it is not >>> very >>> important. >>> >>> --joe. >> >> >> joe, indeed it is not important how you call it, but please, let's not >> call it "soft fork". > > > This kind of fork (whatever it is called) has all the traditional properties > of a softfork except meaningful backwards compatibility for non-upgraded > clients. So I think it is reasonable to call it a softfork with some > qualification. > >> Besides my initial question about the coinbase >> tx, I was also wondering how non-updated nodes would verify the >> collected fees without the actual txs at hand. (They only have the >> coinbase tx, don't they?) > > > Yes this appears to be an oversight in my proof-of-concept implementation. > The unintended consequence being that all transactions would have to be > zero-fee... > > The simplest fix would be make the new rules add the fees implicitly. There > are other solutions. > >> Moreover, I can't see the benefits over a hard fork. A hard fork is >> much cleaner in regard to code changes. As one of the intends of >> "generalized soft forks" is to force user to update, at least a hard >> fork doesn't lie about the fact. Am I missing any obvious advantages >> of a "generalized soft fork" over a "clean" hard fork? > > > A "firm soft fork" also does not lie about that fact -- you must upgrade. I > don't see it dishonest if it was never claimed otherwise. > > I agree that hardforks can be "cleaner". > > However the obvious disadvantage of a hardfork is the risk of the network > splitting between upgraded and non-upgraded clients. This is not a problem > if there is 100% consensus behind the hardfork, but I am not sure if 100% is > realistically achievable for contentious issues such as the blocksize limit. > > If 100% consensus is never achieved, then the options are: > 1. Never upgrade and keep the blocksize limit unchanged forever. > 2. Use a firm softfork to resolve the deadlock. > 3. Hardfork anyway and split the network. > > My argument is simply that 2 is better than 3 and possibly 1. > > --joe > > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
On 2016-01-05 02:04, Nick ODell wrote:
Quoted text (1 line)
> How are you collecting fees from the transactions in the block?
Probably the simplest way to do this is to map the new-rules coinbase tx (which collects the block reward and fees) into an old-rules legacy coinbase tx (which collects the block reward only). Care must be taken to ensure the mapping is not reversible. I will update my implementation in due course.
--joe.
On 2016-01-05 09:26, joe2015--- via bitcoin-dev wrote:
Quoted text (8 lines)
> On 2016-01-05 02:04, Nick ODell wrote: >> How are you collecting fees from the transactions in the block? > > Probably the simplest way to do this is to map the new-rules coinbase > tx (which collects the block reward and fees) into an old-rules legacy > coinbase tx (which collects the block reward only). Care must be > taken to ensure the mapping is not reversible. I will update my > implementation in due course.
The redesigned implementation is here:
https://github.com/ZoomT/bitcoin/tree/2015_2mb_blocksize https://github.com/jgarzik/bitcoin/compare/2015_2mb_blocksize...ZoomT:2015_2mb_blocksize
The new version maps the Merkle root onto a 'legacy' coinbase transaction, solving the problem with fees.
--joe.