deterministic transaction expiration
34 messages, 1 to 8 Aug 2014. Started by Kaz Wesley.
Jeff Garzik (8), Peter Todd (5), Tom Harding (5), Kaz Wesley (4), Mark Friedenbach (3), Matt Whitlock (2), Greg Maxwell (2), Mike Hearn (2), +3 more
There is currently little in place for managing transaction lifetime in the network's mempools (see discussion in github in #3722 "mempool transaction expiration", and it seems to be a major factor blocking some mempool exchange, see #1833/1918, #3721). Expiry per-node a certain amount of wall time after receipt has been proposed, but that's a fragile mechanism -- a single node could keep all relayable transactions alive forever by remembering transactions until most nodes have dropped them and then releasing them back into the wild.
I have a proposal for a way to add finite and predictable lifespans to transactions in mempools: we d̶e̶s̶t̶r̶o̶y̶ ̶t̶h̶e̶ ̶r̶e̶s̶u̶r̶r̶e̶c̶t̶i̶o̶n̶ ̶h̶u̶b̶ use nLockTime and a new standardness rule. It could be done in stages, would not necessarily require even a soft fork, and does not cause problems with reorgs like the proposal in #3509: 1. start setting nLockTime to the current height by default in newly created transactions (or slightly below the current height, for reorg-friendliness) 2. once users have had some time to upgrade to clients that set nLockTime, start discouraging transactions without nLockTime -- possibly with a slightly higher fee required for relay 3. start rate-limiting relay of transactions without an nLockTime (maybe this alone could be used to achieve [2]) 4. add a new IsStandard rule rejecting transactions with an nLockTime more than N blocks behind the current tip (for some fixed value N, to be determined)
Transactions would stop being relayed and drop out of mempools a fixed number of blocks from their creation; once that window had passed, the sender's wallet could begin to expect the transaction would not be confirmed. In case a reorg displaces a transaction until after its expiry height, a miner can still put it back in the blockchain; the expiry height is just a relay rule. Also, a user who needed to get their original "expired" transaction confirmed could still do so by submitting it directly to a miner with suitable policies.
On Thu, Jul 31, 2014 at 05:58:23PM -0700, Kaz Wesley wrote:
Quoted text (6 lines)
> I have a proposal for a way to add finite and predictable lifespans to > transactions in mempools: we d̶e̶s̶t̶r̶o̶y̶ ̶t̶h̶e̶ > ̶r̶e̶s̶u̶r̶r̶e̶c̶t̶i̶o̶n̶ ̶h̶u̶b̶ use nLockTime and a new standardness > rule. It could be done in stages, would not necessarily require even a > soft fork, and does not cause problems with reorgs like the proposal > in #3509:
Anything that changes the semantics of nLockTime will do harm to existing and future applications that make use of nLockTime for things like refund transactions.
In any case, why do transactions need finite lifespans in mempools? If you want to double-spend them with higher fees, then implement replace-by-fee. In any case, lifetimes will never be deterministic as not everyone runs the same software.
Quoted text (8 lines)
> Transactions would stop being relayed and drop out of mempools a fixed > number of blocks from their creation; once that window had passed, the > sender's wallet could begin to expect the transaction would not be > confirmed. In case a reorg displaces a transaction until after its > expiry height, a miner can still put it back in the blockchain; the > expiry height is just a relay rule. Also, a user who needed to get > their original "expired" transaction confirmed could still do so by > submitting it directly to a miner with suitable policies.
...in which case someone will circumvent this IsStandard() rule by submitting "expired" transactions directly to miners with suitable policies.
On Thu, Jul 31, 2014 at 6:06 PM, Peter Todd <pete@petertodd•org> wrote:
Quoted text (3 lines)
> Anything that changes the semantics of nLockTime will do harm to > existing and future applications that make use of nLockTime for things > like refund transactions.
I think this would be compatible with most uses of nLockTime -- e.g., at the time a refund transaction becomes broadcastable, its beneficiary would usually have no reason to wait for a long time before broadcasting it; if they did so (probably because they weren't online to redeem the refund), they'd need to use the submit-directly-to-miner option, but they wouldn't lose their refund.
Quoted text (3 lines)
> In any case, why do transactions need finite lifespans in mempools? If > you want to double-spend them with higher fees, then implement > replace-by-fee.
Perpetuating transactions that have been in mempools for a long time and are not being confirmed has been cited as a reason for nodes not to exchange mempools (#3721, #1833, #3722); it's been implied that it would be desirable for wallets to wait until a transaction had had a chance to be accepted before double-spending with a higher fee (#3722); and an unconfirmed transaction-age-based policy for preventing mempool accumulation has been advocated (#3753, #3722) [I hope my summarization is not misrepresenting anyone's opinions here; please see the arguments made in the actual comments on the bugs]. These discussions are mostly fairly old, but I don't know of any changes that have been made that provide alternative answers to these concerns mentioned by at least three different developers.
Quoted text (2 lines)
> In any case, lifetimes will never be deterministic as not everyone runs > the same software.
That's true, but none of the benefits of these changes require the policy to be unanimous; most of the effect could be provided by most of the network following these rules.
Quoted text (12 lines)
>> Transactions would stop being relayed and drop out of mempools a fixed >> number of blocks from their creation; once that window had passed, the >> sender's wallet could begin to expect the transaction would not be >> confirmed. In case a reorg displaces a transaction until after its >> expiry height, a miner can still put it back in the blockchain; the >> expiry height is just a relay rule. Also, a user who needed to get >> their original "expired" transaction confirmed could still do so by >> submitting it directly to a miner with suitable policies. > > ...in which case someone will circumvent this IsStandard() rule by > submitting "expired" transactions directly to miners with suitable > policies.
Yes, that is a feature. None of the benefits of transaction expiration rely on expiration being final, and any possible downsides of expiration are largely mitigated by the option still being available to get expired transactions mined.
It would make more sense to introduce a new script opcode that pushes the current block height onto the operand stack. Then you could implement arbitrary logic about which blocks the transaction can be valid in. This would require that the client revalidate all transactions in its mempool (really, only those making use of this opcode) whenever the chain tip changes.
On Thursday, 31 July 2014, at 5:58 pm, Kaz Wesley wrote:
Quoted text (35 lines)
> There is currently little in place for managing transaction lifetime > in the network's mempools (see discussion in github in #3722 "mempool > transaction expiration", and it seems to be a major factor blocking > some mempool exchange, see #1833/1918, #3721). Expiry per-node a > certain amount of wall time after receipt has been proposed, but > that's a fragile mechanism -- a single node could keep all relayable > transactions alive forever by remembering transactions until most > nodes have dropped them and then releasing them back into the wild. > > I have a proposal for a way to add finite and predictable lifespans to > transactions in mempools: we d̶e̶s̶t̶r̶o̶y̶ ̶t̶h̶e̶ > ̶r̶e̶s̶u̶r̶r̶e̶c̶t̶i̶o̶n̶ ̶h̶u̶b̶ use nLockTime and a new standardness > rule. It could be done in stages, would not necessarily require even a > soft fork, and does not cause problems with reorgs like the proposal > in #3509: > 1. start setting nLockTime to the current height by default in newly > created transactions (or slightly below the current height, for > reorg-friendliness) > 2. once users have had some time to upgrade to clients that set > nLockTime, start discouraging transactions without nLockTime -- > possibly with a slightly higher fee required for relay > 3. start rate-limiting relay of transactions without an nLockTime > (maybe this alone could be used to achieve [2]) > 4. add a new IsStandard rule rejecting transactions with an nLockTime > more than N blocks behind the current tip (for some fixed value N, to > be determined) > > Transactions would stop being relayed and drop out of mempools a fixed > number of blocks from their creation; once that window had passed, the > sender's wallet could begin to expect the transaction would not be > confirmed. In case a reorg displaces a transaction until after its > expiry height, a miner can still put it back in the blockchain; the > expiry height is just a relay rule. Also, a user who needed to get > their original "expired" transaction confirmed could still do so by > submitting it directly to a miner with suitable policies.
On Thu, Jul 31, 2014 at 6:38 PM, Matt Whitlock <bip@mattwhitlock•name> wrote:
Quoted text (1 line)
> It would make more sense to introduce a new script opcode that pushes the current block height onto the operand stack. Then you could implement arbitrary logic about which blocks the transaction can be valid in. This would require that the client revalidate all transactions in its mempool (really, only those making use of this opcode) whenever the chain tip changes.
Transactions that become invalid later are have pretty severe consequences because they might mean that completely in an absence of fraud transactions are forever precluded due to a otherwise harmless reorg.
While there may be uses for that, the resulting outputs should be considered differently fungible— like coinbases which are immature— and should probably be only used with great caution... not as a mechanism for ordinary transactions.
On Thursday, 31 July 2014, at 7:28 pm, Gregory Maxwell wrote:
Quoted text (7 lines)
> On Thu, Jul 31, 2014 at 6:38 PM, Matt Whitlock <bip@mattwhitlock•name> wrote: > > It would make more sense to introduce a new script opcode that pushes the current block height onto the operand stack. Then you could implement arbitrary logic about which blocks the transaction can be valid in. This would require that the client revalidate all transactions in its mempool (really, only those making use of this opcode) whenever the chain tip changes. > > Transactions that become invalid later are have pretty severe > consequences because they might mean that completely in an absence of > fraud transactions are forever precluded due to a otherwise harmless > reorg.
I understand what you're saying, but I don't understand why it's a problem. Transactions shouldn't be considered "final" until a reasonable number of confirmations anyway, so the possibility that an "accepted" transaction could become invalid due to a chain reorganization is not a new danger. Ordinary transactions can similarly become invalid due to chain reorganizations, due to inputs already having been spent in the new branch.
On Thu, Jul 31, 2014 at 8:26 PM, Matt Whitlock <bip@mattwhitlock•name> wrote:
Quoted text (1 line)
> I understand what you're saying, but I don't understand why it's a problem. Transactions shouldn't be considered "final" until a reasonable number of confirmations anyway, so the possibility that an "accepted" transaction could become invalid due to a chain reorganization is not a new danger. Ordinary transactions can similarly become invalid due to chain reorganizations, due to inputs already having been spent in the new branch.
A distinction there is that they can only become invalid via a conflict— replaced by another transaction authored by the prior signers. If no other transaction could be created (e.g. you're a multisigner and won't sign it again) then there is no such risk. It now introduces chance events ("act of god") into the mix where they they didn't exist before. Basically it takes was what is a very loose one way coupling and makes it much tighter. I'm sure if you spend a bit thinking you can come up with some more corner cases that it would expose— e.g. flooding the network with unrelated high fee transactions in order to push a transaction out to where it can never be mined at all.
On 7/31/2014 5:58 PM, Kaz Wesley wrote:
Quoted text (3 lines)
> 1. start setting nLockTime to the current height by default in newly > created transactions (or slightly below the current height, for > reorg-friendliness)
Reorg-frendliness is the opposite of the rationale behind #2340, which proposes setting nLockTime at current-height + 1 to prevent "fee-sniping" reorgs...
Quoted text (9 lines)
> 2. once users have had some time to upgrade to clients that set > nLockTime, start discouraging transactions without nLockTime -- > possibly with a slightly higher fee required for relay > 3. start rate-limiting relay of transactions without an nLockTime > (maybe this alone could be used to achieve [2]) > 4. add a new IsStandard rule rejecting transactions with an nLockTime > more than N blocks behind the current tip (for some fixed value N, to > be determined) >
One way to proceed is implement #3753 (mempool janitor) in such a way that transactions with nLockTime are allowed to live a bit longer in the mempool (say 500 blocks) than those without (72 hours). In other words, as a first step, just actually start expiring things from the mempool in bitcoin core, and leave any relay fee adjustments or rate limiting for later. The isStandard change would be a good complement to #3753, to avoid relaying a tx that will soon expire by the nLockTime rule anyway.
Quoted text (1 line)
> It would make more sense to introduce a new script opcode that pushes the
current block height onto the operand stack. Then you could implement arbitrary logic about which blocks the transaction can be valid in. This would require that the client revalidate all transactions in its mempool (really, only those making use of this opcode) whenever the chain tip changes.
I have to say I like this idea, this would allow someone to prove they can't spend funds before a given date, and vice versa, prove that the funds can't ever be spent after a given date (and this is provably prunable, isn't it?). Of course, there are some risks associated with that, but nobody is forced to use it.
Quoted text (3 lines)
> flooding the network with unrelated high fee transactions > in order to push a transaction out to where it can never be mined at > all.
This becomes increasingly expensive as the deadline is further away, so not very hard to mitigate.
On Sat, Aug 2, 2014 at 1:36 AM, Tom Harding <tomh@thinlink•com> wrote:
Quoted text (42 lines)
> On 7/31/2014 5:58 PM, Kaz Wesley wrote: > > 1. start setting nLockTime to the current height by default in newly > > created transactions (or slightly below the current height, for > > reorg-friendliness) > > Reorg-frendliness is the opposite of the rationale behind #2340, which > proposes setting nLockTime at current-height + 1 to prevent > "fee-sniping" reorgs... > > > > 2. once users have had some time to upgrade to clients that set > > nLockTime, start discouraging transactions without nLockTime -- > > possibly with a slightly higher fee required for relay > > 3. start rate-limiting relay of transactions without an nLockTime > > (maybe this alone could be used to achieve [2]) > > 4. add a new IsStandard rule rejecting transactions with an nLockTime > > more than N blocks behind the current tip (for some fixed value N, to > > be determined) > > > > One way to proceed is implement #3753 (mempool janitor) in such a way > that transactions with nLockTime are allowed to live a bit longer in the > mempool (say 500 blocks) than those without (72 hours). In other words, > as a first step, just actually start expiring things from the mempool in > bitcoin core, and leave any relay fee adjustments or rate limiting for > later. The isStandard change would be a good complement to #3753, to > avoid relaying a tx that will soon expire by the nLockTime rule anyway. > > > > > ------------------------------------------------------------------------------ > Want fast and easy access to all the code in your enterprise? Index and > search up to 200,000 lines of code with a free copy of Black Duck > Code Sight - the same software that powers the world's largest code > search on Ohloh, the Black Duck Open Hub! Try it now. > http://p.sf.net/sfu/bds > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development >
Glad this was brought up.
Transaction expiration is something that I have wanted to see happen in bitcoin for a long, long time. The user experience of unconfirming transactions setting around in limbo is just horrible. Bitcoin software by necessity has gotten better about attaching fees so this sort of behavior is uncommon, but that does not eliminate the problem.
Of course, we cannot presume that a transaction will truly disappear -- The Internet Never Forgets -- but given a bit of mempool adjusting, we can achieve the next best thing: the majority of the network "forgets" the transaction and becomes willing to relay a respend of some or all of the inputs. This uses existing client logic where the client must rebroadcast a transaction until it is confirmed.
In general, if a transaction has not made it into a block within 144*X blocks, there is _some_ reason it is getting rejected by the miners.
The mempool janitor is a garbage collector design. This is inferior to the "superblock" model described at https://github.com/bitcoin/bitcoin/issues/3723 Other models can also achieve similar results.
There are a lot of issues tied together here: transaction expiration, the desire to cap the mempool ram usage, scalability, DoS prevention, ... mempool ties a lot together.
Quoted text (5 lines)
> > A distinction there is that they can only become invalid via a > conflict— replaced by another transaction authored by the prior > signers. If no other transaction could be created (e.g. you're a > multisigner and won't sign it again) then there is no such risk.
You need to check transaction's dependencies up to a certain depth to know whether it is safe: If one of inputs depends on transaction which is signed by parties with unknown trustworthiness, then it isn't safe.
Quoted text (2 lines)
> It now introduces chance events ("act of god") into the mix where they
> they didn't exist before.You need to check transaction's dependencies up to a certain depth to know whether it is safe: If one of inputs depends on transaction time-locked script (or other unrecognized script), then it isn't safe.
Situation is identical, you might need several extra lines of code.
I think it would matter only if we had deterministic, reliable mempool and reorganization behavior. But it's not something we can depend on.
Quoted text (6 lines)
> > The user experience of unconfirming transactions setting around in limbo > is just horrible. Bitcoin software by necessity has gotten better about > attaching fees so this sort of behavior is uncommon, but that does not > eliminate the problem. >
Yes, indeed. I suspect there's a quick hack that could make this problem a lot better though.
I think I brought up this idea before, but can't quite remember. Anyway I'm willing to bet that if we analysed the data some more, we'd discover that most "legitimate" i.e. non-DoS unconfirmed transactions that sit around for ages are linked back to the block chain within two hops and not more. That is people send a transaction that uses up their coin age, and then immediately those coins are immediately respent again, but then those final new coins are not spent.
On the other hand DoS attacks look like bouncing your coins around over and over forever, i.e. more than two or three hops back to the chain.
So I wonder if making priority look back two or three transactions but not more would help real users a lot, whilst not opening up any significant new potential for DoS.
I feel like a lot of this will be driven by implementation, and costs of changing the implementation. Additional look-backs are of course doable, but they incur some disk I/O costs. The fields available in memory for each mempool TX are
uint256 tx_hash; // hash of next field
CTransaction tx;
int64_t nFee; // Cached to avoid expensive parent-transaction lookups
size_t nTxSize; // ... and avoid recomputing tx size
int64_t nTime; // Local time when entering the mempool
double dPriority; // Priority when entering the mempool
unsigned int nHeight; // Chain height when entering the mempool
As a first pass, we may prune the mempool without additional db lookups quite easily based on time criteria. Or, additional in-memory indexes may be constructed to maintain hashes ordered by priority/fees.
Those techniques seem likely to be attempted before resorting to looking back two or three TXs deep at coin age -- which I admit is an interesting metric.
Quoted text (1 line)
> In general, if a transaction has not made it into a block within 144*X blocks, there is _some_ reason it is getting rejected by the miners.
This is the crux of my concern: relay policy and miner priorities will not necessarily always be in sync, and node resource management shouldn't rely on them being compatible. There are other solutions than transaction expiration; I think Gavin's idea from the block-squashing thread, in which miners explicitly provide information about their policies, would go a long way to address this. But even when mechanisms for reconciling nodes' expectations about miners' behavior with miners' actual behavior are available, it may be desirable to keep an expiry mechanism in place in case of glitches between node understanding of policy and actual miner policy.
Any approach based on beginning a transaction expiry countdown when a transaction is received (as in mempool janitor) seems unviable to me: once a node has forgotten a transaction, it must be susceptible to reaccepting it; all the functionality of such an expiry mechanism depends on the network not containing any nodes with slightly different relay behavior from bitcoind. I could accidentally debilitate mempool janitors across the entire network if I set up two nodes to exchange mempools whenever they reconnected to each other, and restarted each frequently.
That's why I think including information in the transaction itself, as with my nLockTime/IsStandard proposal, is necessary for transactions to reliably eventually die off from mempools. There's a modification I've been thinking about: allow a transaction's lifetime to be refreshed (even after expiry) by a child transaction, along the lines of child-pays-for-parent fee policy. This would eliminate the need to reuse a key to make a replacement for an expired transaction (when submitting the tx directly to a miner is not an option), as well as alleviating the potential inconvenience in cases like Peter brought up, where nLockTime is used for exchanged locked transactions as part of a multi-transaction contract. With child-refreshes-parent, a transaction's receivers and senders would have the ability to keep trying to get their payment confirmed, but anyone on the network can't just keep all transactions alive.
On Tue, Aug 5, 2014 at 10:48 AM, Jeff Garzik <jgarzik@bitpay•com> wrote:
Quoted text (31 lines)
> Glad this was brought up. > > Transaction expiration is something that I have wanted to see happen in > bitcoin for a long, long time. The user experience of unconfirming > transactions setting around in limbo is just horrible. Bitcoin software by > necessity has gotten better about attaching fees so this sort of behavior is > uncommon, but that does not eliminate the problem. > > Of course, we cannot presume that a transaction will truly disappear -- The > Internet Never Forgets -- but given a bit of mempool adjusting, we can > achieve the next best thing: the majority of the network "forgets" the > transaction and becomes willing to relay a respend of some or all of the > inputs. This uses existing client logic where the client must rebroadcast a > transaction until it is confirmed. > > In general, if a transaction has not made it into a block within 144*X > blocks, there is _some_ reason it is getting rejected by the miners. > > The mempool janitor is a garbage collector design. This is inferior to the > "superblock" model described at > https://github.com/bitcoin/bitcoin/issues/3723 Other models can also > achieve similar results. > > There are a lot of issues tied together here: transaction expiration, the > desire to cap the mempool ram usage, scalability, DoS prevention, ... > mempool ties a lot together. > > -- > Jeff Garzik > Bitcoin core developer and open source evangelist > BitPay, Inc. https://bitpay.com/
On Tue, Aug 5, 2014 at 3:10 PM, Kaz Wesley <keziahw@gmail•com> wrote:
Quoted text (3 lines)
> Any approach based on beginning a transaction expiry countdown when a > transaction is received (as in mempool janitor) seems unviable to me: >
...
Quoted text (4 lines)
> That's why I think including information in the transaction itself, as > with my nLockTime/IsStandard proposal, is necessary for transactions > to reliably eventually die off from mempools. >
"reliably die off from mempools" leads into the land of "tightly synchronizing memory pools across the network" which is a problem of... large scope and much debate. :)
For the moment, simply capping the mempool's size at each local node is a much more reachable goal. Capping, then, implies some culling policy. In general, bitcoind Tx mempool size is rather open ended, and that needs sorting out.
On 8/5/2014 12:10 PM, Kaz Wesley wrote:
Quoted text (4 lines)
> Any approach based on beginning a transaction expiry countdown when a > transaction is received (as in mempool janitor) seems unviable to me: > once a node has forgotten a transaction, it must be susceptible to > reaccepting it;
It's hard to argue with that logic.
If nLockTime is used for expiration, transaction creator can't lie to help tx live longer without pushing initial confirmation eligibility into the future. Very pretty. It would also enable "fill or kill" transactions with a backdated nLockTime, which must be confirmed in a few blocks, or start vanishing from mempools.
...and existing users and uses of nLockTime suddenly become worthless, breaking payment channel refunds and other active uses of nLockTime.
You cannot assume the user is around to rewrite their nLockTime, if it fails to be confirmed before some arbitrary deadline being set.
On Wed, Aug 6, 2014 at 12:01 AM, Tom Harding <tomh@thinlink•com> wrote:
Quoted text (28 lines)
> On 8/5/2014 12:10 PM, Kaz Wesley wrote: > > Any approach based on beginning a transaction expiry countdown when a > > transaction is received (as in mempool janitor) seems unviable to me: > > once a node has forgotten a transaction, it must be susceptible to > > reaccepting it; > > It's hard to argue with that logic. > > If nLockTime is used for expiration, transaction creator can't lie to > help tx live longer without pushing initial confirmation eligibility > into the future. Very pretty. It would also enable "fill or kill" > transactions with a backdated nLockTime, which must be confirmed in a > few blocks, or start vanishing from mempools. > > > > ------------------------------------------------------------------------------ > Infragistics Professional > Build stunning WinForms apps today! > Reboot your WinForms applications with our WinForms controls. > Build a bridge from your legacy apps to the future. > > http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development >
We could however introduce a new field in a new tx version. We know we need to rev the format at some point anyway.
On Wed, Aug 6, 2014 at 2:55 PM, Jeff Garzik <jgarzik@bitpay•com> wrote:
Quoted text (60 lines)
> ...and existing users and uses of nLockTime suddenly become worthless, > breaking payment channel refunds and other active uses of nLockTime. > > You cannot assume the user is around to rewrite their nLockTime, if it > fails to be confirmed before some arbitrary deadline being set. > > > > On Wed, Aug 6, 2014 at 12:01 AM, Tom Harding <tomh@thinlink•com> wrote: > >> On 8/5/2014 12:10 PM, Kaz Wesley wrote: >> > Any approach based on beginning a transaction expiry countdown when a >> > transaction is received (as in mempool janitor) seems unviable to me: >> > once a node has forgotten a transaction, it must be susceptible to >> > reaccepting it; >> >> It's hard to argue with that logic. >> >> If nLockTime is used for expiration, transaction creator can't lie to >> help tx live longer without pushing initial confirmation eligibility >> into the future. Very pretty. It would also enable "fill or kill" >> transactions with a backdated nLockTime, which must be confirmed in a >> few blocks, or start vanishing from mempools. >> >> >> >> ------------------------------------------------------------------------------ >> Infragistics Professional >> Build stunning WinForms apps today! >> Reboot your WinForms applications with our WinForms controls. >> Build a bridge from your legacy apps to the future. >> >> http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk >> _______________________________________________ >> Bitcoin-development mailing list >> Bitcoin-development@lists•sourceforge•net >> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >> > > > > -- > Jeff Garzik > Bitcoin core developer and open source evangelist > BitPay, Inc. https://bitpay.com/ > > > ------------------------------------------------------------------------------ > Infragistics Professional > Build stunning WinForms apps today! > Reboot your WinForms applications with our WinForms controls. > Build a bridge from your legacy apps to the future. > > http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > >
How is eventual expiration of a tx that started life with an nLockTime in the future "breaking", any more than any other tx expiring?
On 8/6/2014 6:54 AM, Mike Hearn wrote:
Quoted text (31 lines)
> We could however introduce a new field in a new tx version. We know we > need to rev the format at some point anyway. > > > On Wed, Aug 6, 2014 at 2:55 PM, Jeff Garzik <jgarzik@bitpay•com > <mailto:jgarzik@bitpay.com>> wrote: > > ...and existing users and uses of nLockTime suddenly become > worthless, breaking payment channel refunds and other active uses > of nLockTime. > > You cannot assume the user is around to rewrite their nLockTime, > if it fails to be confirmed before some arbitrary deadline being set. > > > > On Wed, Aug 6, 2014 at 12:01 AM, Tom Harding <tomh@thinlink•com > <mailto:tomh@thinlink.com>> wrote: > > ... > > If nLockTime is used for expiration, transaction creator can't > lie to > help tx live longer without pushing initial confirmation > eligibility > into the future. Very pretty. It would also enable "fill or > kill" > transactions with a backdated nLockTime, which must be > confirmed in a > few blocks, or start vanishing from mempools. >
...because nLockTime is the exact opposite of expiration. A locked TX begins life invalid, and becomes valid (not-expired) after nLockTime passes.
A new field containing expiration time would work.
On Wed, Aug 6, 2014 at 10:44 AM, Tom Harding <tomh@thinlink•com> wrote:
Quoted text (35 lines)
> > How is eventual expiration of a tx that started life with an nLockTime in > the future "breaking", any more than any other tx expiring? > > > > On 8/6/2014 6:54 AM, Mike Hearn wrote: > > We could however introduce a new field in a new tx version. We know we > need to rev the format at some point anyway. > > > On Wed, Aug 6, 2014 at 2:55 PM, Jeff Garzik <jgarzik@bitpay•com> wrote: > >> ...and existing users and uses of nLockTime suddenly become worthless, >> breaking payment channel refunds and other active uses of nLockTime. >> >> You cannot assume the user is around to rewrite their nLockTime, if it >> fails to be confirmed before some arbitrary deadline being set. >> >> >> >> On Wed, Aug 6, 2014 at 12:01 AM, Tom Harding <tomh@thinlink•com> wrote: >> >>> ... >>> >> > If nLockTime is used for expiration, transaction creator can't lie to >>> help tx live longer without pushing initial confirmation eligibility >>> into the future. Very pretty. It would also enable "fill or kill" >>> transactions with a backdated nLockTime, which must be confirmed in a >>> few blocks, or start vanishing from mempools. >>> >> >
+1 for the new field, overloading fields with new meaning is definitely not a good idea.
Something like nExpireAt with a block height sounds reasonable to me, but we need to document that the usual caveats with blockchain reorgs apply.
On Wed, Aug 6, 2014 at 4:08 PM, Jeff Garzik <jgarzik@bitpay•com> wrote:
Quoted text (66 lines)
> ...because nLockTime is the exact opposite of expiration. A locked TX > begins life invalid, and becomes valid (not-expired) after nLockTime passes. > > A new field containing expiration time would work. > > > > On Wed, Aug 6, 2014 at 10:44 AM, Tom Harding <tomh@thinlink•com> wrote: > >> >> How is eventual expiration of a tx that started life with an nLockTime in >> the future "breaking", any more than any other tx expiring? >> >> >> >> On 8/6/2014 6:54 AM, Mike Hearn wrote: >> >> We could however introduce a new field in a new tx version. We know we >> need to rev the format at some point anyway. >> >> >> On Wed, Aug 6, 2014 at 2:55 PM, Jeff Garzik <jgarzik@bitpay•com> wrote: >> >>> ...and existing users and uses of nLockTime suddenly become worthless, >>> breaking payment channel refunds and other active uses of nLockTime. >>> >>> You cannot assume the user is around to rewrite their nLockTime, if it >>> fails to be confirmed before some arbitrary deadline being set. >>> >>> >>> >>> On Wed, Aug 6, 2014 at 12:01 AM, Tom Harding <tomh@thinlink•com> wrote: >>> >>>> ... >>>> >>> >> If nLockTime is used for expiration, transaction creator can't lie >>>> to >>>> help tx live longer without pushing initial confirmation eligibility >>>> into the future. Very pretty. It would also enable "fill or kill" >>>> transactions with a backdated nLockTime, which must be confirmed in a >>>> few blocks, or start vanishing from mempools. >>>> >>> >> > > > -- > Jeff Garzik > Bitcoin core developer and open source evangelist > BitPay, Inc. https://bitpay.com/ > > > ------------------------------------------------------------------------------ > Infragistics Professional > Build stunning WinForms apps today! > Reboot your WinForms applications with our WinForms controls. > Build a bridge from your legacy apps to the future. > > http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > >
On 6 August 2014 08:17:02 GMT-07:00, Christian Decker <decker.christian@gmail•com> wrote:
Quoted text (3 lines)
>+1 for the new field, overloading fields with new meaning is definitely >not >a good idea.
To add a new field the best way to do it is create a new, parallel, tx format where fields are committed by merkle radix tree in an extensible and provable way. You'd then commit to that tree with a mandatory OP_RETURN output in the last txout, or with a new merkle root.
Changing the tx format itself in a hard-fork is needlessly disruptive, and in this case, wastes opportunities for improvement.
A fork is not necessarily required, if you are talking about information that deals primarily with pre-consensus mempool behavior. You can make a "network TX" with some information that is digitally signed, yet discarded before it reaches miners.
On Wed, Aug 6, 2014 at 11:42 AM, Peter Todd <pete@petertodd•org> wrote:
Quoted text (33 lines)
> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > > > On 6 August 2014 08:17:02 GMT-07:00, Christian Decker < > decker.christian@gmail•com> wrote: > >+1 for the new field, overloading fields with new meaning is definitely > >not > >a good idea. > > To add a new field the best way to do it is create a new, parallel, tx > format where fields are committed by merkle radix tree in an extensible and > provable way. You'd then commit to that tree with a mandatory OP_RETURN > output in the last txout, or with a new merkle root. > > Changing the tx format itself in a hard-fork is needlessly disruptive, and > in this case, wastes opportunities for improvement. > -----BEGIN PGP SIGNATURE----- > Version: APG v1.1.1 > > iQFQBAEBCAA6BQJT4kzQMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8 > cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhamzCAC+zRaXRodP63+ke3K+ > Viapiepvk4uIOlqxqtMB2O0zWcyu2+xCJDiRPykK/6HLDBeFDEC9/dGK8++Lovl6 > //qZ340LOPFlgT2kYy9E5h/yX469fhtsWhBCv2K47fWwkMS0S/0r4SQnCkbt2R2c > 4dQjkoldhw6rNMBTUmwvhSlL30KsT/msWTZiX7DW/YjfOzezEJzy+mYyKp9Sk7ba > 1fOiBXORk7mNOs7sTYTvje3sqEGpGTOLP08cY/RCEvl6bG8mHkPqwiojq+3biHFP > RsoBVu1f5cbnU7Wq0gPNdVnQssnEQDadyTX8gT0Wze7PuVyaZT2mXFZBKzSHuLy2 > sJKN > =oPSo > -----END PGP SIGNATURE----- > >
On 08/06/2014 11:42 AM, Peter Todd wrote:
Quoted text (13 lines)
> On 6 August 2014 08:17:02 GMT-07:00, Christian Decker > <decker.christian@gmail•com> wrote: >> +1 for the new field, overloading fields with new meaning is >> definitely not a good idea. > > To add a new field the best way to do it is create a new, parallel, > tx format where fields are committed by merkle radix tree in an > extensible and provable way. You'd then commit to that tree with a > mandatory OP_RETURN output in the last txout, or with a new merkle > root. > > Changing the tx format itself in a hard-fork is needlessly > disruptive, and in this case, wastes opportunities for improvement.
I highly doubt that is the best approach.
If this nExpiry field is a consensus rule, then the Merkle tree or the appropriate paths through needs to be included with the transaction as part of the network and on-disk data structures, so that proper validation can be done. This would be both more disruptive and less efficient than simply adding an nExpiry field to the transaction format, as we do in Freimarkets.
If the field is pre-consensus (a mempool gentleman's agreement), then it has no business in the transaction structure at all and should be packaged in some sort of envelope container.
Today we have first-eligible-height (nLockTime), and mempool expiration measured from this height would work for the goals being discussed, no fork or protocol rev.
With first-eligible-height and last-eligible-height, creator could choose a lifetime shorter than the max, and in addition, lock the whole thing until some point in the future.
On 8/6/2014 9:15 AM, Jeff Garzik wrote:
Quoted text (65 lines)
> A fork is not necessarily required, if you are talking about > information that deals primarily with pre-consensus mempool behavior. > You can make a "network TX" with some information that is digitally > signed, yet discarded before it reaches miners. > > > On Wed, Aug 6, 2014 at 11:42 AM, Peter Todd <pete@petertodd•org > <mailto:pete@petertodd.org>> wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > > > On 6 August 2014 08:17:02 GMT-07:00, Christian Decker > <decker.christian@gmail•com <mailto:decker.christian@gmail.com>> > wrote: > >+1 for the new field, overloading fields with new meaning is > definitely > >not > >a good idea. > > To add a new field the best way to do it is create a new, > parallel, tx format where fields are committed by merkle radix > tree in an extensible and provable way. You'd then commit to that > tree with a mandatory OP_RETURN output in the last txout, or with > a new merkle root. > > Changing the tx format itself in a hard-fork is needlessly > disruptive, and in this case, wastes opportunities for improvement. > -----BEGIN PGP SIGNATURE----- > Version: APG v1.1.1 > > iQFQBAEBCAA6BQJT4kzQMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8 > cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhamzCAC+zRaXRodP63+ke3K+ > Viapiepvk4uIOlqxqtMB2O0zWcyu2+xCJDiRPykK/6HLDBeFDEC9/dGK8++Lovl6 > //qZ340LOPFlgT2kYy9E5h/yX469fhtsWhBCv2K47fWwkMS0S/0r4SQnCkbt2R2c > 4dQjkoldhw6rNMBTUmwvhSlL30KsT/msWTZiX7DW/YjfOzezEJzy+mYyKp9Sk7ba > 1fOiBXORk7mNOs7sTYTvje3sqEGpGTOLP08cY/RCEvl6bG8mHkPqwiojq+3biHFP > RsoBVu1f5cbnU7Wq0gPNdVnQssnEQDadyTX8gT0Wze7PuVyaZT2mXFZBKzSHuLy2 > sJKN > =oPSo > -----END PGP SIGNATURE----- > > > > > -- > Jeff Garzik > Bitcoin core developer and open source evangelist > BitPay, Inc. https://bitpay.com/ > > > ------------------------------------------------------------------------------ > Infragistics Professional > Build stunning WinForms apps today! > Reboot your WinForms applications with our WinForms controls. > Build a bridge from your legacy apps to the future. > http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk > > > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development
On 6 August 2014 09:31:24 GMT-07:00, Mark Friedenbach <mark@monetize•io> wrote:
Quoted text (9 lines)
>I highly doubt that is the best approach. > >If this nExpiry field is a consensus rule, then the Merkle tree or the >appropriate paths through needs to be included with the transaction as >part of the network and on-disk data structures, so that proper >validation can be done. This would be both more disruptive and less >efficient than simply adding an nExpiry field to the transaction >format, >as we do in Freimarkets.
The general case doesn't require transmission of any merkle data; it is derived from the tx data. Equally changing a data format is certainly: note how Freimarkets has no third-party library support because you've made it incompatible with the standard Bitcoin data structures. Merkle radix tree formatting OTOH is just a cryptographically committed extension of the tag-value concept seen in protobuf, among others.
re: efficiency, we need fundamental improvements in efficiency, not little micro-optimisations everywhere done at high cost to maintainability.
re: validation, note how the merkle radix tree meets that need by allowing the absence of data to be proven.
Quoted text (4 lines)
>If the field is pre-consensus (a mempool gentleman's agreement), then >it >has no business in the transaction structure at all and should be >packaged in some sort of envelope container.
It's also rather useless without consensus. Expiry is only useful if it is a guarantee, if not you might as well just implement tx replacement directly.
On 08/06/2014 01:02 PM, Tom Harding wrote:
Quoted text (3 lines)
> With first-eligible-height and last-eligible-height, creator could > choose a lifetime shorter than the max, and in addition, lock the whole > thing until some point in the future.
Note that this would be a massive, *massive* change that would completely break bitcoin output frangibility. Merchants would have to start demanding input history back to a certain depth in order to ensure they are not exposing themselves to undue reorg-expiry risk.
There are useful applications of a consensus-enforced expiry, particularly within a private (signed block) side chain, and for that reason it is useful to have a discussion about the merits of an nExpiry field or BLOCK_HEIGHT / BLOCK_TIME opcode, and methods for achieving either. However I don't see this ever becoming part of the public bitcoin network.
That won't necessarily work through large reorgs. You don't want to create a situation where a miner cannot mine a previously mined transactions.
On Wed, Aug 6, 2014 at 1:02 PM, Tom Harding <tomh@thinlink•com> wrote:
Quoted text (89 lines)
> > Today we have first-eligible-height (nLockTime), and mempool expiration > measured from this height would work for the goals being discussed, no fork > or protocol rev. > > With first-eligible-height and last-eligible-height, creator could choose > a lifetime shorter than the max, and in addition, lock the whole thing > until some point in the future. > > > > On 8/6/2014 9:15 AM, Jeff Garzik wrote: > > A fork is not necessarily required, if you are talking about information > that deals primarily with pre-consensus mempool behavior. You can make a > "network TX" with some information that is digitally signed, yet discarded > before it reaches miners. > > > On Wed, Aug 6, 2014 at 11:42 AM, Peter Todd <pete@petertodd•org> wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA256 >> >> >> >> On 6 August 2014 08:17:02 GMT-07:00, Christian Decker < >> decker.christian@gmail•com> wrote: >> >+1 for the new field, overloading fields with new meaning is definitely >> >not >> >a good idea. >> >> To add a new field the best way to do it is create a new, parallel, tx >> format where fields are committed by merkle radix tree in an extensible and >> provable way. You'd then commit to that tree with a mandatory OP_RETURN >> output in the last txout, or with a new merkle root. >> >> Changing the tx format itself in a hard-fork is needlessly disruptive, >> and in this case, wastes opportunities for improvement. >> -----BEGIN PGP SIGNATURE----- >> Version: APG v1.1.1 >> >> iQFQBAEBCAA6BQJT4kzQMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8 >> cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhamzCAC+zRaXRodP63+ke3K+ >> Viapiepvk4uIOlqxqtMB2O0zWcyu2+xCJDiRPykK/6HLDBeFDEC9/dGK8++Lovl6 >> //qZ340LOPFlgT2kYy9E5h/yX469fhtsWhBCv2K47fWwkMS0S/0r4SQnCkbt2R2c >> 4dQjkoldhw6rNMBTUmwvhSlL30KsT/msWTZiX7DW/YjfOzezEJzy+mYyKp9Sk7ba >> 1fOiBXORk7mNOs7sTYTvje3sqEGpGTOLP08cY/RCEvl6bG8mHkPqwiojq+3biHFP >> RsoBVu1f5cbnU7Wq0gPNdVnQssnEQDadyTX8gT0Wze7PuVyaZT2mXFZBKzSHuLy2 >> sJKN >> =oPSo >> -----END PGP SIGNATURE----- >> >> > > > -- > Jeff Garzik > Bitcoin core developer and open source evangelist > BitPay, Inc. https://bitpay.com/ > > > ------------------------------------------------------------------------------ > Infragistics Professional > Build stunning WinForms apps today! > Reboot your WinForms applications with our WinForms controls. > Build a bridge from your legacy apps to the future.http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk > > > > _______________________________________________ > Bitcoin-development mailing listBitcoin-development@lists•sourceforge•nethttps://lists.sourceforge.net/lists/listinfo/bitcoin-development > > > > > ------------------------------------------------------------------------------ > Infragistics Professional > Build stunning WinForms apps today! > Reboot your WinForms applications with our WinForms controls. > Build a bridge from your legacy apps to the future. > > http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > >
On 08/06/2014 01:20 PM, Peter Todd wrote:
Quoted text (2 lines)
> The general case doesn't require transmission of any merkle data; it > is derived from the tx data.
How can that possibly be the case? The information is hidden behind the Merkle root in the transaction. The validator needs to know whether there is an expiry and what it is. What's it supposed to do, guess?
On 6 August 2014 10:21:56 GMT-07:00, Mark Friedenbach <mark@monetize•io> wrote:
Quoted text (11 lines)
>On 08/06/2014 01:02 PM, Tom Harding wrote: >> With first-eligible-height and last-eligible-height, creator could >> choose a lifetime shorter than the max, and in addition, lock the >whole >> thing until some point in the future. > >Note that this would be a massive, *massive* change that would >completely break bitcoin output frangibility. Merchants would have to >start demanding input history back to a certain depth in order to >ensure >they are not exposing themselves to undue reorg-expiry risk.
Bitcoin is already "broken" in that regard due to malleability, and more fundamentally, the existence of anyone-can-spend outputs, known private keys, SIGHASH_ANYONECANPAY, etc.
In any case, reorg-doublespend risk is no different than reorg-expiry risk.
On 6 August 2014 10:30:11 GMT-07:00, Mark Friedenbach <mark@monetize•io> wrote:
Quoted text (7 lines)
>On 08/06/2014 01:20 PM, Peter Todd wrote: >> The general case doesn't require transmission of any merkle data; it >> is derived from the tx data. > >How can that possibly be the case? The information is hidden behind the >Merkle root in the transaction. The validator needs to know whether >there is an expiry and what it is. What's it supposed to do, guess?
The general case is all committed information is included in the transaction; the merkle tree is a compatibility path, as well as an optimisation for lite clients and applications.
You should read more about soft-forks; see the BIP. Remember that Bitcoin protocol development and deployment is not a centrally controlled activity.
Having explored more drastic approaches, it looks like Kaz' basic idea stands well. His #1...
Quoted text (3 lines)
> 1. start setting nLockTime to the current height by default in newly > created transactions (or slightly below the current height, for > reorg-friendliness)
is already implemented in bitcoin-qt #2340, and a "final call" on merging it was already sent to this list. After some thought I agree with its policy of eventually setting nLockTime at current-height + 1 by default. This is the "best reasonably expected height" of any tx created right now. It discourages fee-sniping, and if a reorg happens anyway, it won't actually delay inclusion of tx beyond the reasonable expectation sans reorg.
However right now, #2340 takes a very cautious approach and sets to current-height - 10 by default, with randomness to mitigate worries about loss of privacy.
Kaz' #2, #3 and #4 are future actions. #4 only goes most of the way ...
Quoted text (3 lines)
> 4. add a new IsStandard rule rejecting transactions with an nLockTime > more than N blocks behind the current tip (for some fixed value N, to > be determined)
... a janitor mechanism is desirable to purge mempool of txes more than N behind current-height.
Nodes dropping a tx N blocks after they became eligible to be mined (the meaning of nLockTime) makes sense. It is not an overloading or new use for nLockTime, but a logical extension of it. As Kaz pointed out, this solves a big problem with expiring by locally measured age: unintentional resurrection.
On Fri, Aug 8, 2014 at 1:38 PM, Tom Harding <tomh@thinlink•com> wrote:
Quoted text (3 lines)
>> 4. add a new IsStandard rule rejecting transactions with an nLockTime >> more than N blocks behind the current tip (for some fixed value N, to >> be determined)
It cannot be assumed that transaction creation time and transaction publish-to-outside-world time are the same, even though they often are.
A new network tx field would have the same problem, right?
With a child-refreshes-parent policy, someone wishing to redeem a transaction that has passed its relay window without being confirmed could still do so. On Aug 8, 2014 11:16 AM, "Jeff Garzik" <jgarzik@bitpay•com> wrote:
Quoted text (26 lines)
> On Fri, Aug 8, 2014 at 1:38 PM, Tom Harding <tomh@thinlink•com> wrote: > >> 4. add a new IsStandard rule rejecting transactions with an nLockTime > >> more than N blocks behind the current tip (for some fixed value N, to > >> be determined) > > It cannot be assumed that transaction creation time and transaction > publish-to-outside-world time are the same, even though they often > are. > > -- > Jeff Garzik > Bitcoin core developer and open source evangelist > BitPay, Inc. https://bitpay.com/ > > > ------------------------------------------------------------------------------ > Want fast and easy access to all the code in your enterprise? Index and > search up to 200,000 lines of code with a free copy of Black Duck > Code Sight - the same software that powers the world's largest code > search on Ohloh, the Black Duck Open Hub! Try it now. > http://p.sf.net/sfu/bds > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development >