Improving RBF policy

Gloria Zhao wrote:

>
> This post discusses limitations of current Bitcoin Core RBF policy and
> attempts to start a conversation about how we can improve it,
> summarizing some ideas that have been discussed. Please reply if you
> have any new input on issues to be solved and ideas for improvement!
>

Is it still verboten to acknowledge that RBF is normal behavior and disallowing it is the feature, and that feature is mostly there to appease some people's delusions that zeroconf is a thing? It seems a bit overdue to disrespect the RBF flag in the direction of always assuming it's on.

> - **Incentive Compatibility**: Ensure that our RBF policy would not
>   accept replacement transactions which would decrease fee profits
>   of a miner. In general, if our mempool policy deviates from what is
> economically rational, it's likely that the transactions in our
> mempool will not match the ones in miners' mempools, making our
> fee estimation, compact block relay, and other mempool-dependent
> functions unreliable. Incentive-incompatible policy may also
> encourage transaction submission through routes other than the p2p
> network, harming censorship-resistance and privacy of Bitcoin payments.
>

There are two different common regimes which result in different incentivized behavior. One of them is that there's more than a block's backlog in the mempool in which case between two conflicting transactions the one with the higher fee rate should win. In the other case where there isn't a whole block's worth of transactions the one with higher total value should win. It would be nice to have consolidated logic which handles both, it seems the issue has to do with the slope of the supply/demand curve which in the first case is gentle enough to keep the one transaction from hitting the rate but in the second one is basically infinite.

> On Jan 31, 2022, at 15:15, Bram Cohen via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> wrote:

…

> Is it still verboten to acknowledge that RBF is normal behavior and disallowing it is the feature, and that feature is mostly there to appease some people's delusions that zeroconf is a thing? It seems a bit overdue to disrespect the RBF flag in the direction of always assuming it's on.

What flag?

>> - **Incentive Compatibility**: Ensure that our RBF policy would not
>>   accept replacement transactions which would decrease fee profits
>>   of a miner. In general, if our mempool policy deviates from what is
>> economically rational, it's likely that the transactions in our
>> mempool will not match the ones in miners' mempools, making our
>> fee estimation, compact block relay, and other mempool-dependent
>> functions unreliable. Incentive-incompatible policy may also
>> encourage transaction submission through routes other than the p2p
>> network, harming censorship-resistance and privacy of Bitcoin payments.
>
> There are two different common regimes which result in different incentivized behavior. One of them is that there's more than a block's backlog in the mempool in which case between two conflicting transactions the one with the higher fee rate should win. In the other case where there isn't a whole block's worth of transactions the one with higher total value should win.

These are not distinct scenarios. The rational choice is the highest fee block-valid subgraph of the set of unconfirmed transactions, in both cases (within the limits of what is computationally feasible of course).

When collecting pooled txs the only issue is DoS protection, which is simply a question of what any given miner is willing to pay, in terms of disk space, to archive conflicts for the opportunity to optimize block reward.

> It would be nice to have consolidated logic which handles both, it seems the issue has to do with the slope of the supply/demand curve which in the first case is gentle enough to keep the one transaction from hitting the rate but in the second one is basically infinite.

> Is it still verboten to acknowledge that RBF is normal behavior and

disallowing it is the feature, and that feature is mostly there to appease some people's delusions that zeroconf is a thing? It seems a bit overdue to disrespect the RBF flag in the direction of always assuming it's on.

If you're thinking about the opt-in flag, not the RBF rules, please see https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2021-June/019074.html The latest state of the discussion is here : https://gnusha.org/bitcoin-core-dev/2021-10-21.log A gradual, multi-year deprecation sounds to be preferred to ease adaptation of the affected Bitcoin applications.

Ultimately, I think it might not be the last time we have to change high-impact tx-relay/mempool rules and a more formalized Core policy deprecation process would be good.

Le lun. 31 janv. 2022 à 18:15, Bram Cohen via bitcoin-dev < bitcoin-dev@lists•linuxfoundation•org> a écrit :

> Gloria Zhao wrote:
>
>>
>> This post discusses limitations of current Bitcoin Core RBF policy and
>> attempts to start a conversation about how we can improve it,
>> summarizing some ideas that have been discussed. Please reply if you
>> have any new input on issues to be solved and ideas for improvement!
>>
>
> Is it still verboten to acknowledge that RBF is normal behavior and
> disallowing it is the feature, and that feature is mostly there to appease
> some people's delusions that zeroconf is a thing? It seems a bit overdue to
> disrespect the RBF flag in the direction of always assuming it's on.
>
>
>> - **Incentive Compatibility**: Ensure that our RBF policy would not
>>   accept replacement transactions which would decrease fee profits
>>   of a miner. In general, if our mempool policy deviates from what is
>> economically rational, it's likely that the transactions in our
>> mempool will not match the ones in miners' mempools, making our
>> fee estimation, compact block relay, and other mempool-dependent
>> functions unreliable. Incentive-incompatible policy may also
>> encourage transaction submission through routes other than the p2p
>> network, harming censorship-resistance and privacy of Bitcoin payments.
>>
>
> There are two different common regimes which result in different
> incentivized behavior. One of them is that there's more than a block's
> backlog in the mempool in which case between two conflicting transactions
> the one with the higher fee rate should win. In the other case where there
> isn't a whole block's worth of transactions the one with higher total value
> should win. It would be nice to have consolidated logic which handles both,
> it seems the issue has to do with the slope of the supply/demand curve
> which in the first case is gentle enough to keep the one transaction from
> hitting the rate but in the second one is basically infinite.
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation•org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

Hi Bastein,

> This work will highly improve the security of any multi-party contract trying to build on top of bitcoin

Do you think such multi party contracts are vulnerable by design considering they rely on policy that cannot be enforced?

> For starters, let me quickly explain why the current rules are hard to work with in the context of lightning

Using the term 'rules' can be confusing sometimes because it's just a policy and different from consensus rules. I wish we could change this in the BIP with something else.

> I'm actually paying a high fee twice instead of once (and needlessly using on-chain space, our scarcest asset, because we could have avoided that additional transaction

Not sure I understand this part because if a transaction is on-chain it can't be replaced.

> The second biggest pain point is rule 3. It prevents me from efficiently using my capital while it's unconfirmed
> I'm curious to hear other people's thoughts on that. If it makes sense, I would propose the following very simple rules

Looks interesting however not sure about X and Y.

On Mon, Jan 31, 2022 at 4:08 PM Eric Voskuil <eric@voskuil•org> wrote:

>
>
> On Jan 31, 2022, at 15:15, Bram Cohen via bitcoin-dev <
> bitcoin-dev@lists•linuxfoundation•org> wrote:
>
> Is it still verboten to acknowledge that RBF is normal behavior and
> disallowing it is the feature, and that feature is mostly there to appease
> some people's delusions that zeroconf is a thing? It seems a bit overdue to
> disrespect the RBF flag in the direction of always assuming it's on.
>
> What flag?
>

The opt-in RBF flag in transactions.

> There are two different common regimes which result in different
> incentivized behavior. One of them is that there's more than a block's
> backlog in the mempool in which case between two conflicting transactions
> the one with the higher fee rate should win. In the other case where there
> isn't a whole block's worth of transactions the one with higher total value
> should win.
>
> These are not distinct scenarios. The rational choice is the highest fee
> block-valid subgraph of the set of unconfirmed transactions, in both cases
> (within the limits of what is computationally feasible of course).
>

It's weird because which of two or more conflicting transactions should win can oscillate back and forth depending on other stuff going on in the mempool. There's already a bit of that with child pays but this is stranger and has more oddball edge cases about which transactions to route.

Hi AJ, Prayank,

> I think that's backwards: we're trying to discourage people from wasting
> the network's bandwidth, which they would do by publishing transactions
> that will never get confirmed -- if they were to eventually get confirmed
> it wouldn't be a waste of bandwith, after all. But if the original
> descendent txs were that sort of spam, then they may well not be
> submitted again if the ancestor tx reaches a fee rate that's actually
> likely to confirm.

But do you agree that descendants only matter for DoS resistance then, not for miner incentives?

I'm asking this because I think a new set of policies should separate policies that address the miner incentives from policies that address the DoS issues.

The two policies I proposed address miner incentives. I think they're insufficient to address DoS issues. But adding a 3rd policy to address DoS issues may be a good solution?

I think that rate-limiting p2p as you suggest (and Gloria also mentioned it) is likely a better way of fixing the DoS concerns than a descendant rule like BIP 125 rule 5 (which as I mentioned earlier, is problematic because the descendent set varies from one mempool to another).

I would like to add a small update to my policy suggestions. The X and Y percentage increase should be met for both the ancestor scores AND the transaction in isolation. Otherwise I could replace txA with txA' that uses a new ancestor txB that has a high fee and high feerate, while txA' has a low fee and low feerate. It's then possible for txB to confirm without txA', and what would remain then in the mempool would be worse than before the replacement.

> All you need is for there to be *a* path that follows the new relay rules
> and gets from your node/wallet to perhaps 10% of hashpower, which seems
> like something wallet providers could construct relatively quickly?

That's true, maybe we can be more optimistic about the timeline for using an updated set of policies ;)

> Do you think such multi party contracts are vulnerable by design
> considering they rely on policy that cannot be enforced?

It's a good question. Even though these policies cannot be enforced, if they are rational to apply by nodes, I think it's ok to rely on them. Others may disagree with that, but I guess it's worth a separate thread.

> Not sure I understand this part because if a transaction is on-chain
> it can't be replaced.

Sorry, that was a bit unclear.

Suppose I have txA that I want to RBF, but I only have unconfirmed utxos and I can't simply lower its existing outputs to reach my desired feerate.

I must make one of my unconfirmed utxos confirm asap just to be able to use it to RBF txA. That means I'll need to pay fees a first time just to convert one of my unconfirmed utxos to a confirmed one. Then I'll pay the fees to bump txA. I had to overpay fees compared to just using my unconfirmed utxo in the first place (and manage more complexity to track the confirmation of my unconfirmed utxo).

Thanks for your feedback! Bastien

Le mar. 1 févr. 2022 à 03:47, Prayank <prayank@tutanota•de> a écrit :

> Hi Bastein,
>
> > This work will highly improve the security of any multi-party contract
> trying to build on top of bitcoin
>
> Do you think such multi party contracts are vulnerable by design
> considering they rely on policy that cannot be enforced?
>
> > For starters, let me quickly explain why the current rules are hard to
> work with in the context of lightning
>
> Using the term 'rules' can be confusing sometimes because it's just a
> policy and different from consensus rules. I wish we could change this in
> the BIP with something else.
>
> > I'm actually paying a high fee twice instead of once (and needlessly
> using on-chain space, our scarcest asset, because we could have avoided
> that additional transaction
>
> Not sure I understand this part because if a transaction is on-chain it
> can't be replaced.
>
> > The second biggest pain point is rule 3. It prevents me from efficiently
> using my capital while it's unconfirmed
>
> > I'm curious to hear other people's thoughts on that. If it makes sense,
> I would propose the following very simple rules
>
> Looks interesting however not sure about X and Y.
>
>
> --
> Prayank
>
> A3B1 E430 2298 178F
>

> On Feb 1, 2022, at 00:32, Bram Cohen <bram@chia•net> wrote:
>
>> On Mon, Jan 31, 2022 at 4:08 PM Eric Voskuil <eric@voskuil•org> wrote:
>>
>>
>>>> On Jan 31, 2022, at 15:15, Bram Cohen via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> wrote:
>>> Is it still verboten to acknowledge that RBF is normal behavior and disallowing it is the feature, and that feature is mostly there to appease some people's delusions that zeroconf is a thing? It seems a bit overdue to disrespect the RBF flag in the direction of always assuming it's on.
>> What flag?
>
> The opt-in RBF flag in transactions.

Was being facetious. The “disrespect” referred to above assumes respect that some implementations have never given.

>>> There are two different common regimes which result in different incentivized behavior. One of them is that there's more than a block's backlog in the mempool in which case between two conflicting transactions the one with the higher fee rate should win. In the other case where there isn't a whole block's worth of transactions the one with higher total value should win.
>> These are not distinct scenarios. The rational choice is the highest fee block-valid subgraph of the set of unconfirmed transactions, in both cases (within the limits of what is computationally feasible of course).
>
> It's weird because which of two or more conflicting transactions should win can oscillate back and forth depending on other stuff going on in the mempool.

The assumption of RAM storage is an error and unrelated to network protocol. There is nothing “going on” in a set of unconfirmed valid transactions. They are logically unchanging.

> There's already a bit of that with child pays but this is stranger and has more oddball edge cases about which transactions to route.

There’s really no such thing. The p2p network is necessarily permissionless. A person can route whatever he wants. Presumably people will not generally waste their own bandwidth by routing what they believe to be unconfirmable. And whatever they would retain themselves is their presumption of confirmable.

This decision of what to retain one’s self is just a graph traversal to determine the most valuable subset - an optimizing CSP (though generally suboptimal due to the time constraint).

Short of DoS, the most profitable model is to retain *all* valid transactions. [Note that a spend conflict is not an invalidity. Two valid transactions can be confirmed in sibling branch blocks - both valid in some context.]

So the only consideration is low cost storage fill. The fee is a proof of spend, which like proof of work (for headers/blocks), is the basis of DoS protection (for unconfirmed transactions). The issue with two conflicting subgraphs is that one or the other is ultimately unspendable. As such the fee on each is non-cumulative and therefore only one (the highest) is providing DoS protection. Any subsequent conflicting subgraph must pay not only for itself, but for all preceding conflicting subgraphs.

This pays for the storage, which is a trade accepted by the owner of the node in order to have a preview of confirmable transactions. This supports both mining generation of candidate blocks and rapid validation/confirmation of blocks.

It’s a rather straightforward system when considered in terms of how it actually works (ie from a consensus standpoint). The only p2p issue is the need to package transactions for consideration as a set, as otherwise parents may be discarded before children can pay for them. Any set up to a full block is entirely reasonable for consideration.

e

On Tue, Feb 01, 2022 at 10:30:12AM +0100, Bastien TEINTURIER via bitcoin-dev wrote:

> But do you agree that descendants only matter for DoS resistance then,
> not for miner incentives?

There's an edge case where you're replacing tx A with tx X, and X's fee rate is higher than A's, but you'd be obsoleting descendent txs (B, C, D...) and thus replacing them with unrelated txs (L, M, N...), and the total feerate/fees of A+B+C+D... is nevertheless higher than X+L+M+N...

But I think that's probably unusual (transactions D and L are adjacent in the mempool, that's why L is chosen for the block; but somehow there's a big drop off in value somewhere between B/C/D and L/M/N), and at least today, I don't think miners consider it essential to eke out every possible sat in fee income.

(If, as per your example, you're actually replacing {A,B,C,D} with {X,Y,Z,W} where X pays higher fees than A and the package in total pays either the same or higher fees, that's certainly incentive compatible. The tricky question is what happens when X arrives on its own and it might be that no one ever sends a replacement for B,C,D)

> The two policies I proposed address miner incentives. I think they're
> insufficient to address DoS issues. But adding a 3rd policy to address
> DoS issues may be a good solution?

>>> 1. The transaction's ancestor absolute fees must be X% higher than the
>>> previous transaction's ancestor fees
>>> 2. The transaction's ancestor feerate must be Y% higher than the
>>> previous transaction's ancestor feerate

Absolute fees only matter if your backlog's feerate drops off. If you've got 100MB of txs offering 5sat/vb, then exchanging 50kB at 5sat/vb for 1kB at 6sat/vb is still a win: your block gains 1000 sats in fees even though your mempool loses 245,000 sats in fees.

But if your backlog's feerate does drop off, *and* that matters, then I don't think you can ignore the impact of the descendent transactions that you might not get a replacement for.

I think "Y% higher" rather than just "higher" is only useful for rate-limiting, not incentive compatibility. (Though maybe it helps stabilise a greedy algorithm in some cases?)

Cheers, aj

Changing the way that RBF works is an excellent idea. Bitcoin is overdue for revisiting these rules. Thanks to @glowzo et al for kicking off the discussion.

I've been thinking about RBF for a very long time[1], and it's been fun to see other people's thoughts on the topics. Here's my current thinking about this, and proposal for how we should update the rules.

### Changing the Rules? How to Change Them Regarding how to change them, Bram and aj are right -- we should move to a model where transaction relay candidates are evaluated on their net increase in fees per byte paid, and remove the requirement that the gross fee of the preceding transaction is met or exceeded.

Our current ruleset is over complicated because it attempts to solve two problems at once: the miner's best interests (highest possible fee take) and relay policy.

I believe this is a mistake and the mempool should change its goals. Instead, the mempool relay design for RBFs should be built around 1) increasing the the per-byte fees paid of a transaction and 2) providing a simple policy for applications building on top of bitcoin, such that knowledge of the mempool is not required for successfully issuing relay-able RBF transactions.

(A simple "must increase the per-byte feerate" heuristic for RBF relay candidates has the nice benefit of being easy to program to on the application side, and only requires knowledge of the previous candidate transaction, not the entire mempool or any previous tx's relative position within it.)

Finally, when blockspace is competitive , this simple policy ensures that the per-byte value of every subsequent relayed transaction increases the per-byte value of pending bytes for the next block. This provides a measure of DoS protection and ensures that every relayed byte is more valuable (to the miner/network) than the last.

*The only time that RBF is critical for relay is during full block periods -- if there's not enough transactions to fill a block, using RBF to ensure that a transaction is mined in a timely manner is moot. As such, RBF rules should not be concerned with low-block environments.

### Mempools and Relay The mempool currently serves two masters: the profit motive of the miner and the relay motive of a utxo holder. It is in the interest of a user to send the miner a high per-byte tx, such that it might end up in the next block. It is in the miner's best interest to include the highest per-byte set of transactions in their block template.

There is some conflation here in the current RBF policies between what is in the mempool and what is considered a candidate for the block template. If a miner has already included a more profitable package of txs into their block template than a more valuable per-byte tx that the network has relayed to them, it should be the responsibility of the block template constructor to reject the new proposed tx, not the nodes relaying the transaction to said miner.

This is a policy that the miner can (and should) implement at the level of the template construction, however.

Is it the responsibility of the mempool to provide the best "historical" block opportunity for a miner (e.g. the highest paying block given all txs it's ever seen)? I would say no, that the ability of a utxo owner to re-state the spend condition of a pending transaction is more important, from a use-case perspective, and that the mempool should concern itself solely with relaying increasingly more profitable bytes to miners. Let the miners concern themselves with deciding what the best policy for their own block construction is, and the mempool with relaying the highest value bytes for the network. Net-net, this will benefit everyone as it becomes easier for users to re-submit txs with increasingly greater feerates, creating more active competition for available blockspace as more applications are able to include it as a feature (and it works, reliable, as a relay mechanism).

### Packages and RBF Packages make the increasing per-byte rule less of a guarantee that increasing the per-byte value of a single transaction will net a given miner more fees than including the entire package.

Let's decompose this a bit. It's helpful to think of tx packages as 'composable txs'. Basically when you consider a 'package' it is actually a large tx with sub-components, the individual txs. As a 'composed tx', you can calculate the per-byte feerate of the entire set. This is the number that you, as someone issuing an RBF, would need to beat in order to move your tx up in the pending block queue.

RBF, however, is a transaction level policy: it allows you to replace any *one* component of a package, or tree, with the side effect of possibly invalidating other candidate txs. If the 'composed tx' (aka package) had a net per-byte value that was higher than the new replacement transaction because of a leaf tx that had an outsized per-byte feerate, then it would be more profitable for the miner to have mined the entire package rather than the replacement.

This edge case complicates the picture for the miner. Considered from the viewpoint of the user issuing the RBF, however, it is far simpler. In the ideal case, a person is issuing an RBF because the previous tx tree, even with its high fee sub-component, was not a candidate for the next block. And, in some cases, increasing the sub-component's per-byte feerate will not achieve the goal of moving the tx any closer to being mined. It's only by increasing the feerate above the present feerate of the candidate plus desendents (tx package) that the transaction will advance in the queue.

While not uncomplicated, this is a simple enough metric for a wallet to track, and does not require any knowledge of the current mempool to effectively surpass. It's the wallet's responsibility to track this though; failure to take descendants into account when deciding on the next per-byte feerate for an RBF *will* mean that your RBF will be ineffective at achieving the goal of getting your UTXO spent. Any wallet is incentivized to always provide a higher per-byte feerate than the 'composed tx' (tx and its descendants), so as to ensure an actual improvement in the unconfirmed transaction's position in the block queue, so to speak.

Note that the v2 protocol for channel opens in lightning uses an RBF negotiation that adheres basically to these rules (ea transaction must have a strictly greater per-byte feerate).

We enforce a rate of 65/64th as the required increase in feerate for each subsequent channel open transaction.

https://github.com/lightning/bolts/pull/851/files#diff-ed04ca2c673fd6aabde69389511fa9ee60cb44d6b2ef6c88b549ffaa753d6afeR1154

### RBF and DoS

Admittedly, changing these rules will increase the number of times that any UTXO is eligible to be retransmitted (relayed) via the bitcoin node network. Strictly increasing the per-byte feerate however ensures that this re-relay is increasingly more expensive to the UTXO owner, however.

### in exitus These are the things I've been thinking about with regards to RBF. I hope they can help to highlight the challenges in the RBF design space a bit more clearly, as well as spelling out the case for using a simple heuristic such as "solely increasing per-byte feerate" as a good candidate for the revised RBF policy.

~niftynei

[1] https://basicbitch.software/posts/2018-12-27-Explaining-Replace-By-Fee.html