On OP_RETURN in upcoming 0.9 release
19 messages, 24 to 28 Feb 2014. Started by Jeff Garzik.
Jeff Garzik (5), Mark Friedenbach (2), Pieter Wuille, Wladimir J. van der Laan, Gavin Andresen, Pavol Rusnak, Jeremy Spilman, Andreas Petersson, +6 more
An update in forthcoming 0.9 release includes a change to make OP_RETURN standard, permitted a small amount of metadata to be attached to a transaction: https://github.com/bitcoin/bitcoin/pull/2738
There was always going to be some level of controversy attached to this. However, some issues, perceptions and questions are bubbling up, and it seemed fair to cover them on the list, not just IRC.
1) FAQ: Why 80 bytes of data? This is the leading programmer question, and it was not really documented well at all. Simple answer: 2x SHA256 or 1x SHA512, plus some tiny bit of metadata. Some schemes are of the nature "BOND<hash>" rather than just plain hash. A common IRC proposal seems to lean towards reducing that from 80. I'll leave it to the crowd to argue about size from there. I do think regular transactions should have the ability to include some metadata.
2) Endorsement of chain data storage. Listening to bitcoin conference corridor discussions, reading forum posts and the occasional article have over-simplified the situation to "core devs endorse data storage over blockchain! let me start uploading my naughty movie collection! IM over blockchain, woo hoo!"
Nothing could be further from the truth. It's a way to make data /less damaging/, not an endorsement of data storage in chain as a good idea. MasterCoin and other projects were doing -even worse- things, such as storing data in forever-unspendable TX outputs, bloating the UTXO for eternity.
It seems reasonable to have a release note to this effect in the 0.9 release announcement, IMO.
On Mon, Feb 24, 2014 at 5:03 PM, Jeff Garzik <jgarzik@bitpay•com> wrote:
Quoted text (2 lines)
> I do think > regular transactions should have the ability to include some metadata.
and
Quoted text (3 lines)
> 2) Endorsement of chain data storage. > > Nothing could be further from the truth.
These two statements are in direct contradiction with each other.
Not really -- a MasterCoin transaction or JPEG
On Mon, Feb 24, 2014 at 11:16 AM, Pieter Wuille <pieter.wuille@gmail•com> wrote:
Quoted text (15 lines)
> On Mon, Feb 24, 2014 at 5:03 PM, Jeff Garzik <jgarzik@bitpay•com> wrote: > >> I do think >> regular transactions should have the ability to include some metadata. > > and > >> 2) Endorsement of chain data storage. >> >> Nothing could be further from the truth. > > These two statements are in direct contradiction with each other. > > -- > Pieter
(fscking 'send' hotkey in GMail)
Not really - a MasterCoin or JPEG image transaction is not a "regular" transaction.
On Mon, Feb 24, 2014 at 11:16 AM, Pieter Wuille <pieter.wuille@gmail•com> wrote:
Quoted text (15 lines)
> On Mon, Feb 24, 2014 at 5:03 PM, Jeff Garzik <jgarzik@bitpay•com> wrote: > >> I do think >> regular transactions should have the ability to include some metadata. > > and > >> 2) Endorsement of chain data storage. >> >> Nothing could be further from the truth. > > These two statements are in direct contradiction with each other. > > -- > Pieter
On Mon, Feb 24, 2014 at 5:03 PM, Jeff Garzik <jgarzik@bitpay•com> wrote:
Quoted text (4 lines)
> A common IRC proposal seems to lean towards reducing that from 80. > I'll leave it to the crowd to argue about size from there. I do think > regular transactions should have the ability to include some metadata. >
I'd be in favor of bringing it down to 40 for 0.9.
That'd be enough for <8 byte header/identifier><32 byte hash>.
80, as the standard line length, is almost asking for "insert your graffiti message here". I also see no need for 64 bytes hashes such as SHA512 in the context of bitcoin, as that only offers 256-bit security (at most) in the first place.
And if this is not abused, these kind of transactions become popular, and more space is really needed, the limit can always be increased in a future version.
Wladimir
40 bytes is small enough to never require an OP_PUSHDATA1, too, which will make writing the OP_RETURN-as-standard BIP simpler.
On Mon, Feb 24, 2014 at 11:39 AM, Wladimir <laanwj@gmail•com> wrote:
Quoted text (23 lines)
> > On Mon, Feb 24, 2014 at 5:03 PM, Jeff Garzik <jgarzik@bitpay•com> wrote: > >> A common IRC proposal seems to lean towards reducing that from 80. >> I'll leave it to the crowd to argue about size from there. I do think >> regular transactions should have the ability to include some metadata. >> > > I'd be in favor of bringing it down to 40 for 0.9. > > That'd be enough for <8 byte header/identifier><32 byte hash>. > > 80, as the standard line length, is almost asking for "insert your > graffiti message here". I also see no need for 64 bytes hashes such as > SHA512 in the context of bitcoin, as that only offers 256-bit security (at > most) in the first place. > > And if this is not abused, these kind of transactions become popular, and > more space is really needed, the limit can always be increased in a future > version. > > Wladimir >
On 02/24/2014 05:45 PM, Gavin Andresen wrote:
Quoted text (1 line)
> 40 bytes is small enough to never require an OP_PUSHDATA1, too
So are 75 bytes. (I'm not trying to push anything. Just saying ...)
This PR reduces the size to 40 bytes: https://github.com/bitcoin/bitcoin/pull/3737
(Note - this is not intended to close the discussion... please do keep sending in feedback)
On Mon, Feb 24, 2014 at 11:03 AM, Jeff Garzik <jgarzik@bitpay•com> wrote:
Quoted text (36 lines)
> An update in forthcoming 0.9 release includes a change to make > OP_RETURN standard, permitted a small amount of metadata to be > attached to a transaction: > https://github.com/bitcoin/bitcoin/pull/2738 > > There was always going to be some level of controversy attached to > this. However, some issues, perceptions and questions are bubbling > up, and it seemed fair to cover them on the list, not just IRC. > > 1) FAQ: Why 80 bytes of data? This is the leading programmer > question, and it was not really documented well at all. Simple > answer: 2x SHA256 or 1x SHA512, plus some tiny bit of metadata. Some > schemes are of the nature "BOND<hash>" rather than just plain hash. > A common IRC proposal seems to lean towards reducing that from 80. > I'll leave it to the crowd to argue about size from there. I do think > regular transactions should have the ability to include some metadata. > > 2) Endorsement of chain data storage. Listening to bitcoin conference > corridor discussions, reading forum posts and the occasional article > have over-simplified the situation to "core devs endorse data storage > over blockchain! let me start uploading my naughty movie collection! > IM over blockchain, woo hoo!" > > Nothing could be further from the truth. It's a way to make data > /less damaging/, not an endorsement of data storage in chain as a good > idea. MasterCoin and other projects were doing -even worse- things, > such as storing data in forever-unspendable TX outputs, bloating the > UTXO for eternity. > > It seems reasonable to have a release note to this effect in the 0.9 > release announcement, IMO. > > -- > Jeff Garzik > Bitcoin core developer and open source evangelist > BitPay, Inc. https://bitpay.com/
Given our standardization on 128-bit security / 256-bit primitives, I can't think of any crypto related data payload which requires more than 40 bytes. Even DER encoded compressed public keys will fit in there. A signature won't fit, but why would you need one in there?
There's no need to design for 64-byte hashes, and the 80-char line length comparison is a good point. As an Engineer I'd want to have a little more room as a 32-byte hash or EC point + 8 bytes identifying prefix data is the bare minimum, but it is also very important that we send a message: This is for payment related applications like stealth addresses only. Don't burden everybody by putting your junk on the block chain.
On 02/24/2014 08:39 AM, Wladimir wrote:
Quoted text (39 lines)
> > On Mon, Feb 24, 2014 at 5:03 PM, Jeff Garzik <jgarzik@bitpay•com > <mailto:jgarzik@bitpay.com>> wrote: > > A common IRC proposal seems to lean towards reducing that from 80. > I'll leave it to the crowd to argue about size from there. I do think > regular transactions should have the ability to include some metadata. > > > I'd be in favor of bringing it down to 40 for 0.9. > > That'd be enough for <8 byte header/identifier><32 byte hash>. > > 80, as the standard line length, is almost asking for "insert your > graffiti message here". I also see no need for 64 bytes hashes such as > SHA512 in the context of bitcoin, as that only offers 256-bit security > (at most) in the first place. > > And if this is not abused, these kind of transactions become popular, > and more space is really needed, the limit can always be increased in a > future version. > > Wladimir > > > ------------------------------------------------------------------------------ > Flow-based real-time traffic analytics software. Cisco certified tool. > Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer > Customize your own dashboards, set traffic alerts and generate reports. > Network behavioral analysis & security monitoring. All-in-one tool. > http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk > > > > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development >
On Mon, 24 Feb 2014 09:10:26 -0800, Jeff Garzik <jgarzik@bitpay•com> wrote:
Quoted text (2 lines)
> This PR reduces the size to 40 bytes: > https://github.com/bitcoin/bitcoin/pull/3737
Just quickly GLANCED at it, but if I understand correctly how the template matching code works, that will change max size of the <data> to 40 bytes but does not do anything to enforce most-efficient encoding.
else if (opcode2 == OP_SMALLDATA)
{
// small pushdata, <= MAX_OP_RETURN_RELAY bytes
if (vch1.size() > MAX_OP_RETURN_RELAY)
break;
}
This code was a bit hard for me to parse since it's not actually requiring any data, just disallowing more than a certain number of bytes of data. So a bare OP_RETURN would be allowed as well, for whatever good that will do.
If you want to strictly require no PUSHDATA, perhaps you could do:
else if (opcode2 == OP_SMALLDATA)
{
// small pushdata, <= MAX_OP_RETURN_RELAY bytes
if (opcode1 >= OP_PUSHDATA1 || vch1.size() > MAX_OP_RETURN_RELAY)
break;
}
Thanks, Jeremy
Sure, no objection to that.
On Mon, Feb 24, 2014 at 5:12 PM, Jeremy Spilman <jeremy@taplink•co> wrote:
Quoted text (33 lines)
> On Mon, 24 Feb 2014 09:10:26 -0800, Jeff Garzik <jgarzik@bitpay•com> wrote: >> >> This PR reduces the size to 40 bytes: >> https://github.com/bitcoin/bitcoin/pull/3737 > > > Just quickly GLANCED at it, but if I understand correctly how the template > matching code works, that will change max size of the <data> to 40 bytes but > does not do anything to enforce most-efficient encoding. > > else if (opcode2 == OP_SMALLDATA) > { > // small pushdata, <= MAX_OP_RETURN_RELAY bytes > if (vch1.size() > MAX_OP_RETURN_RELAY) > break; > } > > This code was a bit hard for me to parse since it's not actually requiring > any data, just disallowing more than a certain number of bytes of data. So a > bare OP_RETURN would be allowed as well, for whatever good that will do. > > If you want to strictly require no PUSHDATA, perhaps you could do: > > else if (opcode2 == OP_SMALLDATA) > { > // small pushdata, <= MAX_OP_RETURN_RELAY bytes > if (opcode1 >= OP_PUSHDATA1 || vch1.size() > MAX_OP_RETURN_RELAY) > break; > } > > Thanks, > Jeremy >
Regarding 80 bytes vs smaller: The objectives should be that if you are determined to put some extra data in the blockchain, OP_RETURN should be the superior alternative. if a user can include more data with less fees using a multisig TX, then this will happen.
eventually dust-limit rules will not be the deciding factor here, since i suspect block propagation times will have a stronger effect on effective fees. therefore a slightly larger payload than the biggest multisig TX is the right answer. - that would be >= 64x3 bytes = 192 bytes. (this is my understanding of how large a 3-of-3 multisig tx can be, plus 1.5 bits encoded in the "n" parameter)
On Mon, Feb 24, 2014 at 3:06 PM, Andreas Petersson <andreas@petersson•at> wrote:
Quoted text (11 lines)
> Regarding 80 bytes vs smaller: The objectives should be that if you are > determined to put some extra data in the blockchain, OP_RETURN should be > the superior alternative. if a user can include more data with less fees > using a multisig TX, then this will happen. > > eventually dust-limit rules will not be the deciding factor here, since > i suspect block propagation times will have a stronger effect on > effective fees. therefore a slightly larger payload than the biggest > multisig TX is the right answer. - that would be >= 64x3 bytes = 192 bytes. > (this is my understanding of how large a 3-of-3 multisig tx can be, plus > 1.5 bits encoded in the "n" parameter)
At least there is no ambiguity that such usage is abusive. Adoption of the practices matters too. Right now I've seen a lot of people promoting data storage as a virtuous use, and gearing up to directly store data when a commitment would work.
If it turns out that encouraging people to use hashes is a lost cause it can always be further relaxed in the future, going the other way is much harder.
On Monday, February 24, 2014 11:06:30 PM Andreas Petersson wrote:
Quoted text (11 lines)
> Regarding 80 bytes vs smaller: The objectives should be that if you are > determined to put some extra data in the blockchain, OP_RETURN should be > the superior alternative. if a user can include more data with less fees > using a multisig TX, then this will happen. > > eventually dust-limit rules will not be the deciding factor here, since > i suspect block propagation times will have a stronger effect on > effective fees. therefore a slightly larger payload than the biggest > multisig TX is the right answer. - that would be >= 64x3 bytes = 192 bytes. > (this is my understanding of how large a 3-of-3 multisig tx can be, plus > 1.5 bits encoded in the "n" parameter)
Perhaps I ought to redo my data carrier configuration option as a max size?
Luke
To each his own, but if I say "Please don't charge me for YOUR privacy by putting junk like stealth addresses in the blockchain", I think I'd get laughed out of most rooms.
Either the transaction fees are sufficient to pay the cost for whatever random junk anyone wants to put there, or they are not, and if they are not, then I suggest you re-think the fee structure rather than trying to pre-regulate me putting 80 character pithy quotes in the blockhain.
On Mon, Feb 24, 2014 at 09:23:12AM -0800, Mark Friedenbach wrote:
Quoted text (64 lines)
> Given our standardization on 128-bit security / 256-bit primitives, I > can't think of any crypto related data payload which requires more than > 40 bytes. Even DER encoded compressed public keys will fit in there. A > signature won't fit, but why would you need one in there? > > There's no need to design for 64-byte hashes, and the 80-char line > length comparison is a good point. As an Engineer I'd want to have a > little more room as a 32-byte hash or EC point + 8 bytes identifying > prefix data is the bare minimum, but it is also very important that we > send a message: This is for payment related applications like stealth > addresses only. Don't burden everybody by putting your junk on the block > chain. > > On 02/24/2014 08:39 AM, Wladimir wrote: > > > > On Mon, Feb 24, 2014 at 5:03 PM, Jeff Garzik <jgarzik@bitpay•com > > <mailto:jgarzik@bitpay.com>> wrote: > > > > A common IRC proposal seems to lean towards reducing that from 80. > > I'll leave it to the crowd to argue about size from there. I do think > > regular transactions should have the ability to include some metadata. > > > > > > I'd be in favor of bringing it down to 40 for 0.9. > > > > That'd be enough for <8 byte header/identifier><32 byte hash>. > > > > 80, as the standard line length, is almost asking for "insert your > > graffiti message here". I also see no need for 64 bytes hashes such as > > SHA512 in the context of bitcoin, as that only offers 256-bit security > > (at most) in the first place. > > > > And if this is not abused, these kind of transactions become popular, > > and more space is really needed, the limit can always be increased in a > > future version. > > > > Wladimir > > > > > > ------------------------------------------------------------------------------ > > Flow-based real-time traffic analytics software. Cisco certified tool. > > Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer > > Customize your own dashboards, set traffic alerts and generate reports. > > Network behavioral analysis & security monitoring. All-in-one tool. > > http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk > > > > > > > > _______________________________________________ > > Bitcoin-development mailing list > > Bitcoin-development@lists•sourceforge•net > > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > > > ------------------------------------------------------------------------------ > Flow-based real-time traffic analytics software. Cisco certified tool. > Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer > Customize your own dashboards, set traffic alerts and generate reports. > Network behavioral analysis & security monitoring. All-in-one tool. > http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development
On Thu, Feb 27, 2014 at 7:25 PM, Troy Benjegerdes <hozer@hozed•org> wrote:
Quoted text (7 lines)
> > Either the transaction fees are sufficient to pay the cost for whatever > random junk anyone wants to put there, or they are not, and if they are > not, then I suggest you re-think the fee structure rather than trying > to pre-regulate me putting 80 character pithy quotes in the blockhain. > >
https://github.com/litecoin-project/litecoin/commit/db4d8e21d99551bef4c807aa1534a074e4b7964d
In one way in particular, the transaction fees per kilobyte completely failed to account for the actual cost to the network. If Bitcoin had adopted a common-sense rule like this, I would have had no reason to join Litecoin development last year. This is one of the few economic design flaws that Satoshi overlooked in the original design.
As much as I personally hate the idea of data storage in the blockchain, this at least discourages the creation of permanent UTXO.
Warren Togami
Transaction fees are a DoS mitigating cost to the person making the transaction, but they are generally not paid to the people who actually incur costs in validating the blockchain. Actual transaction processing costs are an externality that is completely unpaid for.
When I add a 1Kb transaction to the blockchain, there is an attached fee which probabilistically goes to one of the miners. But every other full node on the network also receives this transaction, processes it, and adds it to local storage. From now until the heat death of the universe that 1Kb of data will be redundantly stored and transmitted to every single person who validates the block chain. None of these countless people are reimbursed for their storage, bandwidth, and processing costs. Not even a single satoshi.
Yes, transaction fees are broken. But it is their very nature which is broken (sending coins to the miners, not the greater validator set), and no little tweak like the one Warren links to will fix this.
But, in the absence of a reformed fee regime - which it is not clear is even possible - one could at least make the hand-wavey argument that people who validate the block chain receive benefit from it as a payment network. Therefore processing of the block chain is "paid for" by the utility it provides once fully synced.
However even this weak argument does not extend to general data storage. If you want to put all of wikileaks or whatever in the block chain, then you are extracting a rent from every full node which is forced to process and store this data for eternity without compensation or derived utility. You are extorting users of the payment network into providing a storage service at no cost, because the alternative (losing bitcoin as a payment network) would cost them more.
That is not ethical behavior. That is not behavior which responsible developers should allow in the reference client.
Mark
On 02/28/2014 06:42 AM, Warren Togami Jr. wrote:
Quoted text (30 lines)
> On Thu, Feb 27, 2014 at 7:25 PM, Troy Benjegerdes <hozer@hozed•org > <mailto:hozer@hozed.org>> wrote: > > > Either the transaction fees are sufficient to pay the cost for > whatever random junk anyone wants to put there, or they are not, > and if they are not, then I suggest you re-think the fee structure > rather than trying to pre-regulate me putting 80 character pithy > quotes in the blockhain. > > > https://github.com/litecoin-project/litecoin/commit/db4d8e21d99551bef4c807aa1534a074e4b7964d > > In one way in particular, the transaction fees per kilobyte > completely failed to account for the actual cost to the network. > If Bitcoin had adopted a common-sense rule like this, I would have > had no reason to join Litecoin development last year. This is one > of the few economic design flaws that Satoshi overlooked in the > original design. > > As much as I personally hate the idea of data storage in the > blockchain, this at least discourages the creation of permanent > UTXO. > > Warren Togami > > > ------------------------------------------------------------------------------ > >
Flow-based real-time traffic analytics software. Cisco certified tool.
Quoted text (13 lines)
> Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow > Analyzer Customize your own dashboards, set traffic alerts and > generate reports. Network behavioral analysis & security > monitoring. All-in-one tool. > http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk > > > > > _______________________________________________ Bitcoin-development > mailing list Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development >
On 02/28/2014 07:25 PM, Mark Friedenbach wrote:
Quoted text (4 lines)
> Transaction fees are a DoS mitigating cost to the person making the > transaction, but they are generally not paid to the people who > actually incur costs in validating the blockchain. Actual transaction > processing costs are an externality that is completely unpaid for.
What that means is the network layer is broken and needs to be fixed.
Bitcoin is the blockchain, not the P2P network. If the existing network is not incentive compatible, then that's the root cause which should be addressed.
There's no reason to enshrine the broken behavior and use it as a roadblock to stop progress.
On 28 February 2014 14:42, Warren Togami Jr. <wtogami@gmail•com> wrote:
Quoted text (9 lines)
> > https://github.com/litecoin-project/litecoin/commit/db4d8e21d99551bef4c807aa1534a074e4b7964d > > In one way in particular, the transaction fees per kilobyte completely > failed to account for the actual cost to the network. If Bitcoin had > adopted a common-sense rule like this, I would have had no reason to join > Litecoin development last year. This is one of the few economic design > flaws that Satoshi overlooked in the original design. >
Is there any particular reason that patch would not make it into bitcoin if submitted?
Drak