Removal of reject network messages from Bitcoin Core (BIP61)
22 messages, 6 Mar to 21 Oct 2019. Started by Marco Falke.
Andreas Schildbach (6), Eric Voskuil (3), Dustin Dettmer (2), Aymeric Vitte (2), Gregory Maxwell (2), John Newbery (2), Marco Falke, Sjors Provoost, +3 more
Bitcoin Core may send "reject" messages as response to "tx", "block" or "version" messages from a network peer when the message could not be accepted.
This feature is toggled by the `-enablebip61` command line option and has been disabled by default since Bitcoin Core version 0.18.0 (not yet released as of time of writing). Nodes on the network can not generally be trusted to send valid ("reject") messages, so this should only ever be used when connected to a trusted node. At this time, I am not aware of any software that requires this feature, and I would like to remove if from Bitcoin Core to make the codebase slimmer, easier to understand and maintain. Let us know if your application relies on this feature and you can not use any of the recommended alternatives:
* Testing or debugging of implementations of the Bitcoin P2P network protocol should be done by inspecting the log messages that are produced by a recent version of Bitcoin Core. Bitcoin Core logs debug messages (`-debug=<category>`) to a stream (`-printtoconsole`) or to a file (`-debuglogfile=<debug.log>`).
* Testing the validity of a block can be achieved by specific RPCs:
- `submitblock`
- `getblocktemplate` with `'mode'` set to `'proposal'` for blocks with
potentially invalid POW
* Testing the validity of a transaction can be achieved by specific RPCs: - `sendrawtransaction` - `testmempoolaccept`
* Wallets should not use the absence of "reject" messages to indicate a transaction has propagated the network, nor should wallets use "reject" messages to set transaction fees. Wallets should rather use fee estimation to determine transaction fees and set replace-by-fee if desired. Thus, they could wait until the transaction has confirmed (taking into account the fee target they set (compare the RPC `estimatesmartfee`)) or listen for the transaction announcement by other network peers to check for propagation.
I propose to remove "reject" messages from Bitcoin Core 0.19.0 unless there are valid concerns about its removal.
Marco
The reject message is helpful for figuring out why a tx was rejected.
It’s not useful for determining success, yes. Particularly when doing segwit / newer types of tx’s as there’s always one or more pesky nodes who still don’t support it and send a reject message for perfectly good tx’s.
But after a delay where you haven’t seen your tx propagated on the network, it’s useful to know *why* it failed.
What would be nice is actually expanding this error message. Currently with RBF tx’s “fee too small” is sent for both original transactions as well as replacement transactions. So a bug accidentally sending spent txos (currently in mempool) says “fee too small” instead of something more appropriate like “fee too small to supersede existing unconfirmed transaction.”
On Tue, Mar 5, 2019 at 7:26 PM Marco Falke via bitcoin-dev < bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (57 lines)
> Bitcoin Core may send "reject" messages as response to "tx", "block" or
> "version" messages from a network peer when the message could not be
> accepted.
>
> This feature is toggled by the `-enablebip61` command line option and has
> been
> disabled by default since Bitcoin Core version 0.18.0 (not yet released as
> of
> time of writing). Nodes on the network can not generally be trusted to send
> valid ("reject") messages, so this should only ever be used when connected
> to a
> trusted node. At this time, I am not aware of any software that requires
> this
> feature, and I would like to remove if from Bitcoin Core to make the
> codebase
> slimmer, easier to understand and maintain. Let us know if your application
> relies on this feature and you can not use any of the recommended
> alternatives:
>
> * Testing or debugging of implementations of the Bitcoin P2P network
> protocol
> should be done by inspecting the log messages that are produced by a
> recent
> version of Bitcoin Core. Bitcoin Core logs debug messages
> (`-debug=<category>`) to a stream (`-printtoconsole`) or to a file
> (`-debuglogfile=<debug.log>`).
>
> * Testing the validity of a block can be achieved by specific RPCs:
> - `submitblock`
> - `getblocktemplate` with `'mode'` set to `'proposal'` for blocks with
> potentially invalid POW
>
> * Testing the validity of a transaction can be achieved by specific RPCs:
> - `sendrawtransaction`
> - `testmempoolaccept`
>
> * Wallets should not use the absence of "reject" messages to indicate a
> transaction has propagated the network, nor should wallets use "reject"
> messages to set transaction fees. Wallets should rather use fee
> estimation
> to determine transaction fees and set replace-by-fee if desired. Thus,
> they
> could wait until the transaction has confirmed (taking into account the
> fee
> target they set (compare the RPC `estimatesmartfee`)) or listen for the
> transaction announcement by other network peers to check for propagation.
>
> I propose to remove "reject" messages from Bitcoin Core 0.19.0 unless
> there are
> valid concerns about its removal.
>
> Marco
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation•org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>Reject messages cannot be replaced for debugging user problems. At least unless you plan to make RPC or bitcoind logfiles available via the P2P protocol (both probably not a good idea).
The typical case is, I get mailed a wallet logfile with reject messages and that's all I have. I cannot access the bitcoind logfile(s) of the node(s) that generated the reject message in the first place. Nor can I access their RPC interface.
I strongly suggest re-enabling reject messages by default before 0.18.
On 06/03/2019 01.53, Marco Falke via bitcoin-dev wrote:
Quoted text (40 lines)
> Bitcoin Core may send "reject" messages as response to "tx", "block" or
> "version" messages from a network peer when the message could not be accepted.
>
> This feature is toggled by the `-enablebip61` command line option and has been
> disabled by default since Bitcoin Core version 0.18.0 (not yet released as of
> time of writing). Nodes on the network can not generally be trusted to send
> valid ("reject") messages, so this should only ever be used when connected to a
> trusted node. At this time, I am not aware of any software that requires this
> feature, and I would like to remove if from Bitcoin Core to make the codebase
> slimmer, easier to understand and maintain. Let us know if your application
> relies on this feature and you can not use any of the recommended alternatives:
>
> * Testing or debugging of implementations of the Bitcoin P2P network protocol
> should be done by inspecting the log messages that are produced by a recent
> version of Bitcoin Core. Bitcoin Core logs debug messages
> (`-debug=<category>`) to a stream (`-printtoconsole`) or to a file
> (`-debuglogfile=<debug.log>`).
>
> * Testing the validity of a block can be achieved by specific RPCs:
> - `submitblock`
> - `getblocktemplate` with `'mode'` set to `'proposal'` for blocks with
> potentially invalid POW
>
> * Testing the validity of a transaction can be achieved by specific RPCs:
> - `sendrawtransaction`
> - `testmempoolaccept`
>
> * Wallets should not use the absence of "reject" messages to indicate a
> transaction has propagated the network, nor should wallets use "reject"
> messages to set transaction fees. Wallets should rather use fee estimation
> to determine transaction fees and set replace-by-fee if desired. Thus, they
> could wait until the transaction has confirmed (taking into account the fee
> target they set (compare the RPC `estimatesmartfee`)) or listen for the
> transaction announcement by other network peers to check for propagation.
>
> I propose to remove "reject" messages from Bitcoin Core 0.19.0 unless there are
> valid concerns about its removal.
>
> Marco
>Can you elaborate a bit on what kind of reject messages your users are getting? I assume the users wallet connects directly to the Bitcoin p2p network?
What does the wallet do when a transaction is rejected? Does it forget about it (that seems unsafe) or compose another one (with overlapping inputs)?
Sjors
Quoted text (61 lines)
> Op 6 mrt. 2019, om 17:49 heeft Andreas Schildbach via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> het volgende geschreven:
>
> Reject messages cannot be replaced for debugging user problems. At least
> unless you plan to make RPC or bitcoind logfiles available via the P2P
> protocol (both probably not a good idea).
>
> The typical case is, I get mailed a wallet logfile with reject messages
> and that's all I have. I cannot access the bitcoind logfile(s) of the
> node(s) that generated the reject message in the first place. Nor can I
> access their RPC interface.
>
> I strongly suggest re-enabling reject messages by default before 0.18.
>
>
> On 06/03/2019 01.53, Marco Falke via bitcoin-dev wrote:
>> Bitcoin Core may send "reject" messages as response to "tx", "block" or
>> "version" messages from a network peer when the message could not be accepted.
>>
>> This feature is toggled by the `-enablebip61` command line option and has been
>> disabled by default since Bitcoin Core version 0.18.0 (not yet released as of
>> time of writing). Nodes on the network can not generally be trusted to send
>> valid ("reject") messages, so this should only ever be used when connected to a
>> trusted node. At this time, I am not aware of any software that requires this
>> feature, and I would like to remove if from Bitcoin Core to make the codebase
>> slimmer, easier to understand and maintain. Let us know if your application
>> relies on this feature and you can not use any of the recommended alternatives:
>>
>> * Testing or debugging of implementations of the Bitcoin P2P network protocol
>> should be done by inspecting the log messages that are produced by a recent
>> version of Bitcoin Core. Bitcoin Core logs debug messages
>> (`-debug=<category>`) to a stream (`-printtoconsole`) or to a file
>> (`-debuglogfile=<debug.log>`).
>>
>> * Testing the validity of a block can be achieved by specific RPCs:
>> - `submitblock`
>> - `getblocktemplate` with `'mode'` set to `'proposal'` for blocks with
>> potentially invalid POW
>>
>> * Testing the validity of a transaction can be achieved by specific RPCs:
>> - `sendrawtransaction`
>> - `testmempoolaccept`
>>
>> * Wallets should not use the absence of "reject" messages to indicate a
>> transaction has propagated the network, nor should wallets use "reject"
>> messages to set transaction fees. Wallets should rather use fee estimation
>> to determine transaction fees and set replace-by-fee if desired. Thus, they
>> could wait until the transaction has confirmed (taking into account the fee
>> target they set (compare the RPC `estimatesmartfee`)) or listen for the
>> transaction announcement by other network peers to check for propagation.
>>
>> I propose to remove "reject" messages from Bitcoin Core 0.19.0 unless there are
>> valid concerns about its removal.
>>
>> Marco
>>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation•org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-devYes, I'm talking about P2P connections.
First and foremost, reject messages are an indication that the transaction isn't going to confirm. Without these messages, we'd need to revert to pre-BIP61 behaviour of using a timeout for reception of network confirmations.
Regarding the content, these cases are useful to distinguish:
- Not enough fee - UTXO already spent - Tx validity/standardness (e.g. invalid signature)
While the last one in theority wouldn't be necessary if you produced your software bug-free to begin with, this just isn't how software development works. Developers need any indication they can get.
The first two happen even in the ideal case. Fees are impossible to predict, and unintentional double spends happen because users clone their wallet state.
On 07/03/2019 14.59, Sjors Provoost via bitcoin-dev wrote:
Quoted text (67 lines)
> Can you elaborate a bit on what kind of reject messages your users are getting? I assume the users wallet connects directly to the Bitcoin p2p network?
>
> What does the wallet do when a transaction is rejected? Does it forget about it (that seems unsafe) or compose another one (with overlapping inputs)?
>
> Sjors
>
>> Op 6 mrt. 2019, om 17:49 heeft Andreas Schildbach via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> het volgende geschreven:
>>
>> Reject messages cannot be replaced for debugging user problems. At least
>> unless you plan to make RPC or bitcoind logfiles available via the P2P
>> protocol (both probably not a good idea).
>>
>> The typical case is, I get mailed a wallet logfile with reject messages
>> and that's all I have. I cannot access the bitcoind logfile(s) of the
>> node(s) that generated the reject message in the first place. Nor can I
>> access their RPC interface.
>>
>> I strongly suggest re-enabling reject messages by default before 0.18.
>>
>>
>> On 06/03/2019 01.53, Marco Falke via bitcoin-dev wrote:
>>> Bitcoin Core may send "reject" messages as response to "tx", "block" or
>>> "version" messages from a network peer when the message could not be accepted.
>>>
>>> This feature is toggled by the `-enablebip61` command line option and has been
>>> disabled by default since Bitcoin Core version 0.18.0 (not yet released as of
>>> time of writing). Nodes on the network can not generally be trusted to send
>>> valid ("reject") messages, so this should only ever be used when connected to a
>>> trusted node. At this time, I am not aware of any software that requires this
>>> feature, and I would like to remove if from Bitcoin Core to make the codebase
>>> slimmer, easier to understand and maintain. Let us know if your application
>>> relies on this feature and you can not use any of the recommended alternatives:
>>>
>>> * Testing or debugging of implementations of the Bitcoin P2P network protocol
>>> should be done by inspecting the log messages that are produced by a recent
>>> version of Bitcoin Core. Bitcoin Core logs debug messages
>>> (`-debug=<category>`) to a stream (`-printtoconsole`) or to a file
>>> (`-debuglogfile=<debug.log>`).
>>>
>>> * Testing the validity of a block can be achieved by specific RPCs:
>>> - `submitblock`
>>> - `getblocktemplate` with `'mode'` set to `'proposal'` for blocks with
>>> potentially invalid POW
>>>
>>> * Testing the validity of a transaction can be achieved by specific RPCs:
>>> - `sendrawtransaction`
>>> - `testmempoolaccept`
>>>
>>> * Wallets should not use the absence of "reject" messages to indicate a
>>> transaction has propagated the network, nor should wallets use "reject"
>>> messages to set transaction fees. Wallets should rather use fee estimation
>>> to determine transaction fees and set replace-by-fee if desired. Thus, they
>>> could wait until the transaction has confirmed (taking into account the fee
>>> target they set (compare the RPC `estimatesmartfee`)) or listen for the
>>> transaction announcement by other network peers to check for propagation.
>>>
>>> I propose to remove "reject" messages from Bitcoin Core 0.19.0 unless there are
>>> valid concerns about its removal.
>>>
>>> Marco
>>>
>>
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists•linuxfoundation•org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-devBitcoin-transactions did use this "feature", but does not rely on it any longer since I observed some strange behavior sometimes (no reject message for bad tx, with suprnova for example as far as I remember), then it doublechecks using getdata to see if the tx is in mempool
Indeed you can't trust what a node tells you with or without reject (idem for getdata but more difficult to fake and better than nothing)
Then I don't see any problem to remove it, taking into account also that the error message is too vague to be really helpful https://github.com/bitcoin/bitcoin/issues/11891
Le 06/03/2019 à 01:53, Marco Falke via bitcoin-dev a écrit :
Quoted text (43 lines)
> Bitcoin Core may send "reject" messages as response to "tx", "block" or
> "version" messages from a network peer when the message could not be accepted.
>
> This feature is toggled by the `-enablebip61` command line option and has been
> disabled by default since Bitcoin Core version 0.18.0 (not yet released as of
> time of writing). Nodes on the network can not generally be trusted to send
> valid ("reject") messages, so this should only ever be used when connected to a
> trusted node. At this time, I am not aware of any software that requires this
> feature, and I would like to remove if from Bitcoin Core to make the codebase
> slimmer, easier to understand and maintain. Let us know if your application
> relies on this feature and you can not use any of the recommended alternatives:
>
> * Testing or debugging of implementations of the Bitcoin P2P network protocol
> should be done by inspecting the log messages that are produced by a recent
> version of Bitcoin Core. Bitcoin Core logs debug messages
> (`-debug=<category>`) to a stream (`-printtoconsole`) or to a file
> (`-debuglogfile=<debug.log>`).
>
> * Testing the validity of a block can be achieved by specific RPCs:
> - `submitblock`
> - `getblocktemplate` with `'mode'` set to `'proposal'` for blocks with
> potentially invalid POW
>
> * Testing the validity of a transaction can be achieved by specific RPCs:
> - `sendrawtransaction`
> - `testmempoolaccept`
>
> * Wallets should not use the absence of "reject" messages to indicate a
> transaction has propagated the network, nor should wallets use "reject"
> messages to set transaction fees. Wallets should rather use fee estimation
> to determine transaction fees and set replace-by-fee if desired. Thus, they
> could wait until the transaction has confirmed (taking into account the fee
> target they set (compare the RPC `estimatesmartfee`)) or listen for the
> transaction announcement by other network peers to check for propagation.
>
> I propose to remove "reject" messages from Bitcoin Core 0.19.0 unless there are
> valid concerns about its removal.
>
> Marco
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation•org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-devHi Marco,
Quoted text (3 lines)
> At this time, I am not aware of any software that requires this feature, and I > would like to remove if from Bitcoin Core to make the codebase slimmer, easier > to understand and maintain.
Neutrino[1], a light client implementation that uses BIPs 157 and 158, currently relies on receiving reject messages from its peers when broadcasting a transaction to the network. I've personally gone through the relevant parts of the Bitcoin Core codebase involving reject messages and respectfully disagree that removing it would help much in terms of comprehension and maintainability. IMO, the benefits outweigh this small cost.
Quoted text (2 lines)
> Nodes on the network can not generally be trusted to send valid ("reject")
> messages, so this should only ever be used when connected to a trusted node.Nodes in the network generally rely on the assumption that they are connected to at least one honest peer, so we can actually converge on the set of honest peers and ban/disconnect any who send an invalid reject message for a valid transaction.
Quoted text (2 lines)
> Let us know if your application relies on this feature and you can not use any > of the recommended alternatives:
Unfortunately, none of the recommended alternatives work for our use case. The main thing we want to identify when broadcasting a transaction is whether is it invalid or not. As long as it is valid, reject messages aren't required as the light client can just rebroadcast the transaction upon every new block to ensure it is eventually included in the chain. It can then stop rebroadcasting it once it detects it has confirmed on-chain through its filters. However, if it is invalid, some of the validity checks required cannot be performed by light clients as they do not have a mempool and/or UTXO set.
Reject messages also useful when developing new light clients, as we can get some feedback from the network on why a transaction was rejected, which helps identify potential bugs in their transaction crafting logic. I understand that this can be done by setting up test nodes with the flag enabled, but this justifies that the feature should at least exist and not be completely removed.
Quoted text (3 lines)
> * Testing the validity of a transaction can be achieved by specific RPCs: > - `sendrawtransaction` > - `testmempoolaccept`
These RPCs are not helpful for light clients. Even for full nodes, in the case of `testmempoolaccept`, mempool conditions can quickly change and cause a transaction to be rejected after the fact. One alternative would be for a third-party to set up an endpoint where users can submit their transactions to, but now you're placing your trust solely on them, rather than the network, which doesn't seem like a reasonable or comparable compromise.
With that said, I believe the feature should remain enabled by default in order to aid the light clients of the network. If we disable them by default, no one will bother to enable them manually, and light clients won't be able to realize they are broadcasting invalid transactions.
[1] https://github.com/lightninglabs/neutrino
- Wilmer
Quoted text (5 lines)
>> On Mar 7, 2019, at 19:09, Wilmer Paulino via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> wrote:
>> ...
>> Nodes on the network can not generally be trusted to send valid ("reject") messages, so this should only ever be used when connected to a trusted node.
>
> Nodes in the network generally rely on the assumption that they are connected to at least one honest peer, so we can actually converge on the set of honest peers and ban/disconnect any who send an invalid reject message for a valid transaction.This implies the reject message is valid only when it is expected (i.e. the sender is knowingly sending invalid transactions), which is presumably useful only in a local development environment.
e
On Thu, Mar 7, 2019 at 11:46 PM Andreas Schildbach via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (4 lines)
> First and foremost, reject messages are an indication that the > transaction isn't going to confirm. Without these messages, we'd need to > revert to pre-BIP61 behaviour of using a timeout for reception of > network confirmations.
That is already required because even in the presence of perfectly honest and cooperative hosts reject messages at most can only tell you about first-hop behaviour. It won't even tell you if the transaction was ever even attempted to be sent to a next hop. So alternative handling must be provided and must be reliable for the software to work at all regardless of reject messages.
Quoted text (1 line)
> - Not enough fee
Rejection on low fee (over the static minimum feerate) only happens at the point where the nodes mempool is full, which is already at a point where you might be waiting weeks for confirmation.
Rejection causes were also not stable or reliable because the validity criteria cannot generally be tested independently. For example, if a transaction is queued due to missing a parent it isn't rejected because missing the parent is often a temporary issue, but its feerate cannot be measured without the parent. Later, when the parent is obtained, the transaction can then be rejected due to feerate-- but no reject is sent then.
Output already spend is often completely indistinguishable from a missing parent and can't get rejects generated for it generally.
Similarly, the error state detected for things like invalid signatures are often not very useful. The software knows that script execution returned false, but in the general case _why_ it returned false is not clear, and a straightforward high performance validation implementation doesn't necessarily yield a good way of figuring out and propagating up that information. (I think invalid signatures end up returning a stack-nonempty state from validation currently, as an example of that).
(Posting again, since my previous reply didn't appear)
On 08/03/2019 01.52, Gregory Maxwell via bitcoin-dev wrote:
Quoted text (14 lines)
> That is already required because even in the presence of perfectly > honest and cooperative hosts reject messages at most can only tell you > about first-hop behaviour. It won't even tell you if the transaction > was ever even attempted to be sent to a next hop. So alternative > handling must be provided and must be reliable for the software to > work at all regardless of reject messages. > > Rejection causes were also not stable or reliable because the validity > criteria cannot generally be tested independently. For example, if a > transaction is queued due to missing a parent it isn't rejected > because missing the parent is often a temporary issue, but its feerate > cannot be measured without the parent. Later, when the parent is > obtained, the transaction can then be rejected due to feerate-- but no > reject is sent then.
These two cases are understood and handled by current code. Generally the idea is take reject messages serious, but don't overrate the lack of. Luckily, network confirmations fill the gap. (Yes, a timeout is still useful. But at least it almost never happens.)
Quoted text (8 lines)
> Similarly, the error state detected for things like invalid signatures > are often not very useful. The software knows that script execution > returned false, but in the general case _why_ it returned false is not > clear, and a straightforward high performance validation > implementation doesn't necessarily yield a good way of figuring out > and propagating up that information. (I think invalid signatures end > up returning a stack-nonempty state from validation currently, as an > example of that).
Nevertheless, it has been proven as useful in debugging (just recently when I implemented the witness signature hash in bitcoinj). I think Wilmer Paulino summed up this point quite nicely in his reply to this thread.
On Tue, Mar 12, 2019 at 7:45 PM Andreas Schildbach via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (3 lines)
> These two cases are understood and handled by current code. Generally > the idea is take reject messages serious, but don't overrate the lack > of. Luckily, network confirmations fill the gap. (Yes, a timeout is
I'd like to better understand this, but it would be easier to just read the code than ask a bunch of questions. I tried looking for the handling of reject messages in Android Bitcoin Wallet and BitcoinJ and didn't really find and handling other than logging exceptions. Would you mind giving me a couple pointers to where in the code they're handled?
On 12/03/2019 23.14, Gregory Maxwell via bitcoin-dev wrote:
Quoted text (12 lines)
> On Tue, Mar 12, 2019 at 7:45 PM Andreas Schildbach via bitcoin-dev > <bitcoin-dev@lists•linuxfoundation•org> wrote: >> These two cases are understood and handled by current code. Generally >> the idea is take reject messages serious, but don't overrate the lack >> of. Luckily, network confirmations fill the gap. (Yes, a timeout is > > I'd like to better understand this, but it would be easier to just > read the code than ask a bunch of questions. I tried looking for the > handling of reject messages in Android Bitcoin Wallet and BitcoinJ > and didn't really find and handling other than logging exceptions. > Would you mind giving me a couple pointers to where in the code > they're handled?
It's implemented in bitcoinj's TransactionBroadcast class. Received reject messages are collected and -- if a certain consensus (currently: half of connected peers) is reached -- a RejectedTransactionException is raised.
The handling of that exception in Bitcoin Wallet is extremely rudimentary. I think it still only shows the exception message. But certainly I was hoping to improve on this soon.
Quoted text (6 lines)
> I'd like to better understand this, but it would be easier to just > read the code than ask a bunch of questions. I tried looking for the > handling of reject messages in Android Bitcoin Wallet and BitcoinJ > and didn't really find and handling other than logging exceptions. > Would you mind giving me a couple pointers to where in the code > they're handled?
I’ve solved the same problem in a different way.
1) Submit a transaction 2) Collect all reject messages (that have matching txid in the reject data) 3) Wait 16 seconds after first error message received (chosen semirandomly from trial and error) before processing errors 4) Wait for our txid to be submitted back to us through the mempool, if we get it notify success and delete all pending error events 5) Signal failure with the given reject code if present (after the 16 seconds have elapsed) 6) If no error or success after 20 seconds, signal timeout failure
This works fairly well in testing. Newer transaction types seem to generate reject codes 100% of the time (from at least one node when sending to 4 nodes) so this culling / time delay approach is essentially required.
On a related note: One issue is that RBF attempts with too small a fee and accidental double spends (with enough fee for 1 tx but not a RBF) both generate the same reject code: not enough fee.
A new reject code for RBF based too small of fee would definitely make for a better user experience as I’ve seen this exact problem create confusion for users.
Removing reject codes would make for a much worse user experience. “Your tx failed and we have no idea why” would be the only message and it would require waiting for a full timeout.
On Wed, Mar 13, 2019 at 3:16 PM Oscar Guindzberg via bitcoin-dev < bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (14 lines)
> > I'd like to better understand this, but it would be easier to just > > read the code than ask a bunch of questions. I tried looking for the > > handling of reject messages in Android Bitcoin Wallet and BitcoinJ > > and didn't really find and handling other than logging exceptions. > > Would you mind giving me a couple pointers to where in the code > > they're handled? > > > https://github.com/bitcoinj/bitcoinj/blob/master/core/src/main/java/org/bitcoinj/core/TransactionBroadcast.java#L93-L108 > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >
Apparently I don't have the same experience than others here, what I encountered is no reject message received for wrong txs, but from what I understand here it's not unusual to receive reject message for valid txs, then I don't see how it can be really helpful/relied, given also that the reject messages are unclear and even can be misleading
As it was written already I found it useful only for debugging purposes, at least it can give some kind of ideas about what happened, bitcoin-transactions is implementing the bitcoin protocol but does not act as a node and does not pretend to fake a node behavior waiting for example to get the tx back, is the method of sending a getdata for a given tx to see if it was accepted by a node wrong ? It can't guarantee 100% that it was successful and will propagate but I see that you are doing completely different things
Le 13/03/2019 à 23:30, Dustin Dettmer via bitcoin-dev a écrit :
Quoted text (53 lines)
> I’ve solved the same problem in a different way. > > 1) Submit a transaction > 2) Collect all reject messages (that have matching txid in the reject > data) > 3) Wait 16 seconds after first error message received (chosen > semirandomly from trial and error) before processing errors > 4) Wait for our txid to be submitted back to us through the mempool, > if we get it notify success and delete all pending error events > 5) Signal failure with the given reject code if present (after the 16 > seconds have elapsed) > 6) If no error or success after 20 seconds, signal timeout failure > > This works fairly well in testing. Newer transaction types seem to > generate reject codes 100% of the time (from at least one node when > sending to 4 nodes) so this culling / time delay approach is > essentially required. > > On a related note: One issue is that RBF attempts with too small a fee > and accidental double spends (with enough fee for 1 tx but not a RBF) > both generate the same reject code: not enough fee. > > A new reject code for RBF based too small of fee would definitely make > for a better user experience as I’ve seen this exact problem create > confusion for users. > > Removing reject codes would make for a much worse user experience. > “Your tx failed and we have no idea why” would be the only message and > it would require waiting for a full timeout. > > On Wed, Mar 13, 2019 at 3:16 PM Oscar Guindzberg via bitcoin-dev > <bitcoin-dev@lists•linuxfoundation•org > <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote: > > > I'd like to better understand this, but it would be easier to just > > read the code than ask a bunch of questions. I tried looking for the > > handling of reject messages in Android Bitcoin Wallet and BitcoinJ > > and didn't really find and handling other than logging exceptions. > > Would you mind giving me a couple pointers to where in the code > > they're handled? > > https://github.com/bitcoinj/bitcoinj/blob/master/core/src/main/java/org/bitcoinj/core/TransactionBroadcast.java#L93-L108 > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > <mailto:bitcoin-dev@lists.linuxfoundation.org> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Following discussion on this mailing list, support for BIP 61 REJECT messages was not removed from Bitcoin Core in V0.19. The behaviour in that upcoming release is that REJECT messages are disabled by default and can be enabled using the `-enablebip61` command line option.
Support for REJECT messages will be removed entirely in Bitcoin Core V0.20, expected for release in mid 2020. The PR to remove support was merged into Bitcoin Core's master branch this week.
Adoption of new Bitcoin Core versions across reachable nodes generally takes several months. https://bitnodes.earn.com/dashboard/?days=365 shows that although v0.18 was released in May 2019, there are still several hundred reachable nodes on V0.17, V0.16, V0.15 and earlier software. Software that currently use REJECT messages from public nodes for troubleshooting issues therefore have plenty of time to transition to one of the methods listed by Marco in the email above.
John
On Tue, Mar 5, 2019 at 10:28 PM Marco Falke via bitcoin-dev < bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (57 lines)
> Bitcoin Core may send "reject" messages as response to "tx", "block" or
> "version" messages from a network peer when the message could not be
> accepted.
>
> This feature is toggled by the `-enablebip61` command line option and has
> been
> disabled by default since Bitcoin Core version 0.18.0 (not yet released as
> of
> time of writing). Nodes on the network can not generally be trusted to send
> valid ("reject") messages, so this should only ever be used when connected
> to a
> trusted node. At this time, I am not aware of any software that requires
> this
> feature, and I would like to remove if from Bitcoin Core to make the
> codebase
> slimmer, easier to understand and maintain. Let us know if your application
> relies on this feature and you can not use any of the recommended
> alternatives:
>
> * Testing or debugging of implementations of the Bitcoin P2P network
> protocol
> should be done by inspecting the log messages that are produced by a
> recent
> version of Bitcoin Core. Bitcoin Core logs debug messages
> (`-debug=<category>`) to a stream (`-printtoconsole`) or to a file
> (`-debuglogfile=<debug.log>`).
>
> * Testing the validity of a block can be achieved by specific RPCs:
> - `submitblock`
> - `getblocktemplate` with `'mode'` set to `'proposal'` for blocks with
> potentially invalid POW
>
> * Testing the validity of a transaction can be achieved by specific RPCs:
> - `sendrawtransaction`
> - `testmempoolaccept`
>
> * Wallets should not use the absence of "reject" messages to indicate a
> transaction has propagated the network, nor should wallets use "reject"
> messages to set transaction fees. Wallets should rather use fee
> estimation
> to determine transaction fees and set replace-by-fee if desired. Thus,
> they
> could wait until the transaction has confirmed (taking into account the
> fee
> target they set (compare the RPC `estimatesmartfee`)) or listen for the
> transaction announcement by other network peers to check for propagation.
>
> I propose to remove "reject" messages from Bitcoin Core 0.19.0 unless
> there are
> valid concerns about its removal.
>
> Marco
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation•org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>On 16/10/2019 18.43, John Newbery via bitcoin-dev wrote:
Quoted text (4 lines)
> Following discussion on this mailing list, support for BIP 61 REJECT > messages was not removed from Bitcoin Core in V0.19. The behaviour in > that upcoming release is that REJECT messages are disabled by default > and can be enabled using the `-enablebip61` command line option.
Is there a NODE_* bit we can use to pick peers that support this (useful!) feature?
As this is a P2P protocol change it should be exposed as a version increment (and a BIP), not just as a conditional service. If the intent is to retain this protocol indefinitely, exposing it conditionally, then a service bit would make sense, but it remains a protocol change.
BIP61 is explicit:
“All implementations of the P2P protocol version 70,002 and later should support the reject message.“
e
Quoted text (17 lines)
> On Oct 17, 2019, at 12:54, Andreas Schildbach via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> wrote: > > On 16/10/2019 18.43, John Newbery via bitcoin-dev wrote: > >> Following discussion on this mailing list, support for BIP 61 REJECT >> messages was not removed from Bitcoin Core in V0.19. The behaviour in >> that upcoming release is that REJECT messages are disabled by default >> and can be enabled using the `-enablebip61` command line option. > > Is there a NODE_* bit we can use to pick peers that support this > (useful!) feature? > > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Quoted text (1 line)
> Is there a NODE_* bit we can use to pick peers that support this (useful!)
feature?
No. BIP 61 has no mechanism for advertising that a node will send REJECT messages.
On Wed, Oct 16, 2019 at 12:43 PM John Newbery <john@johnnewbery•com> wrote:
Quoted text (83 lines)
> Following discussion on this mailing list, support for BIP 61 REJECT > messages was not removed from Bitcoin Core in V0.19. The behaviour in that > upcoming release is that REJECT messages are disabled by default and can be > enabled using the `-enablebip61` command line option. > > Support for REJECT messages will be removed entirely in Bitcoin Core > V0.20, expected for release in mid 2020. The PR to remove support was > merged into Bitcoin Core's master branch this week. > > Adoption of new Bitcoin Core versions across reachable nodes generally > takes several months. https://bitnodes.earn.com/dashboard/?days=365 shows > that although v0.18 was released in May 2019, there are still several > hundred reachable nodes on V0.17, V0.16, V0.15 and earlier software. > Software that currently use REJECT messages from public nodes for > troubleshooting issues therefore have plenty of time to transition to one > of the methods listed by Marco in the email above. > > John > > On Tue, Mar 5, 2019 at 10:28 PM Marco Falke via bitcoin-dev < > bitcoin-dev@lists•linuxfoundation•org> wrote: > >> Bitcoin Core may send "reject" messages as response to "tx", "block" or >> "version" messages from a network peer when the message could not be >> accepted. >> >> This feature is toggled by the `-enablebip61` command line option and has >> been >> disabled by default since Bitcoin Core version 0.18.0 (not yet released >> as of >> time of writing). Nodes on the network can not generally be trusted to >> send >> valid ("reject") messages, so this should only ever be used when >> connected to a >> trusted node. At this time, I am not aware of any software that requires >> this >> feature, and I would like to remove if from Bitcoin Core to make the >> codebase >> slimmer, easier to understand and maintain. Let us know if your >> application >> relies on this feature and you can not use any of the recommended >> alternatives: >> >> * Testing or debugging of implementations of the Bitcoin P2P network >> protocol >> should be done by inspecting the log messages that are produced by a >> recent >> version of Bitcoin Core. Bitcoin Core logs debug messages >> (`-debug=<category>`) to a stream (`-printtoconsole`) or to a file >> (`-debuglogfile=<debug.log>`). >> >> * Testing the validity of a block can be achieved by specific RPCs: >> - `submitblock` >> - `getblocktemplate` with `'mode'` set to `'proposal'` for blocks with >> potentially invalid POW >> >> * Testing the validity of a transaction can be achieved by specific RPCs: >> - `sendrawtransaction` >> - `testmempoolaccept` >> >> * Wallets should not use the absence of "reject" messages to indicate a >> transaction has propagated the network, nor should wallets use "reject" >> messages to set transaction fees. Wallets should rather use fee >> estimation >> to determine transaction fees and set replace-by-fee if desired. Thus, >> they >> could wait until the transaction has confirmed (taking into account the >> fee >> target they set (compare the RPC `estimatesmartfee`)) or listen for the >> transaction announcement by other network peers to check for >> propagation. >> >> I propose to remove "reject" messages from Bitcoin Core 0.19.0 unless >> there are >> valid concerns about its removal. >> >> Marco >> _______________________________________________ >> bitcoin-dev mailing list >> bitcoin-dev@lists•linuxfoundation•org >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >> >
On Thu, Oct 17, 2019 at 01:16:47PM -0700, Eric Voskuil via bitcoin-dev wrote:
Quoted text (7 lines)
> As this is a P2P protocol change it should be exposed as a version > increment (and a BIP) [...] > > BIP61 is explicit: > > “All implementations of the P2P protocol version 70,002 and later > should support the reject message.“
I don't think a new BIP or a version number increment is necessary.
1. "Should support" isn't the same as "must support". See https://tools.ietf.org/html/rfc2119 ; by that reading, implementations with protocol versions above 70,002 are not required to support the reject message.
2. If you don't implement a BIP, as Bitcoin Core explicitly doesn't any more for BIP61[1], you're not bound by its conditions.
-Dave
[1] https://github.com/bitcoin/bitcoin/blob/master/doc/bips.md "BIP61 [...] Support was removed in v0.20.0"
I agree, thanks.
FWIW I’ve never been a fan of the ‘reject’ message, or its implementation.
https://github.com/bitcoin/bips/wiki/Comments:BIP-0061
e
Quoted text (25 lines)
> On Oct 18, 2019, at 18:46, David A. Harding <dave@dtrt•org> wrote: > > On Thu, Oct 17, 2019 at 01:16:47PM -0700, Eric Voskuil via bitcoin-dev wrote: >> As this is a P2P protocol change it should be exposed as a version >> increment (and a BIP) [...] >> >> BIP61 is explicit: >> >> “All implementations of the P2P protocol version 70,002 and later >> should support the reject message.“ > > I don't think a new BIP or a version number increment is necessary. > > 1. "Should support" isn't the same as "must support". See > https://tools.ietf.org/html/rfc2119 ; by that reading, > implementations with protocol versions above 70,002 are not required > to support the reject message. > > 2. If you don't implement a BIP, as Bitcoin Core explicitly doesn't any > more for BIP61[1], you're not bound by its conditions. > > -Dave > > [1] https://github.com/bitcoin/bitcoin/blob/master/doc/bips.md "BIP61 > [...] Support was removed in v0.20.0"
I guess then the best way to discover nodes that have reject messages enabled is connecting/disconnecting to random nodes and send them invalid transactions and keep the ones which reply with a reject message.
On 18/10/2019 22.53, John Newbery via bitcoin-dev wrote:
Quoted text (106 lines)
>> Is there a NODE_* bit we can use to pick peers that support this > (useful!) feature? > > No. BIP 61 has no mechanism for advertising that a node will send REJECT > messages. > > On Wed, Oct 16, 2019 at 12:43 PM John Newbery <john@johnnewbery•com > <mailto:john@johnnewbery.com>> wrote: > > Following discussion on this mailing list, support for BIP 61 REJECT > messages was not removed from Bitcoin Core in V0.19. The behaviour > in that upcoming release is that REJECT messages are disabled by > default and can be enabled using the `-enablebip61` command line option. > > Support for REJECT messages will be removed entirely in Bitcoin Core > V0.20, expected for release in mid 2020. The PR to remove support > was merged into Bitcoin Core's master branch this week. > > Adoption of new Bitcoin Core versions across reachable nodes > generally takes several months. > https://bitnodes.earn.com/dashboard/?days=365 shows that although > v0.18 was released in May 2019, there are still several hundred > reachable nodes on V0.17, V0.16, V0.15 and earlier software. > Software that currently use REJECT messages from public nodes for > troubleshooting issues therefore have plenty of time to transition > to one of the methods listed by Marco in the email above. > > John > > On Tue, Mar 5, 2019 at 10:28 PM Marco Falke via bitcoin-dev > <bitcoin-dev@lists•linuxfoundation•org > <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote: > > Bitcoin Core may send "reject" messages as response to "tx", > "block" or > "version" messages from a network peer when the message could > not be accepted. > > This feature is toggled by the `-enablebip61` command line > option and has been > disabled by default since Bitcoin Core version 0.18.0 (not yet > released as of > time of writing). Nodes on the network can not generally be > trusted to send > valid ("reject") messages, so this should only ever be used when > connected to a > trusted node. At this time, I am not aware of any software that > requires this > feature, and I would like to remove if from Bitcoin Core to make > the codebase > slimmer, easier to understand and maintain. Let us know if your > application > relies on this feature and you can not use any of the > recommended alternatives: > > * Testing or debugging of implementations of the Bitcoin P2P > network protocol > should be done by inspecting the log messages that are > produced by a recent > version of Bitcoin Core. Bitcoin Core logs debug messages > (`-debug=<category>`) to a stream (`-printtoconsole`) or to a file > (`-debuglogfile=<debug.log>`). > > * Testing the validity of a block can be achieved by specific RPCs: > - `submitblock` > - `getblocktemplate` with `'mode'` set to `'proposal'` for > blocks with > potentially invalid POW > > * Testing the validity of a transaction can be achieved by > specific RPCs: > - `sendrawtransaction` > - `testmempoolaccept` > > * Wallets should not use the absence of "reject" messages to > indicate a > transaction has propagated the network, nor should wallets use > "reject" > messages to set transaction fees. Wallets should rather use > fee estimation > to determine transaction fees and set replace-by-fee if > desired. Thus, they > could wait until the transaction has confirmed (taking into > account the fee > target they set (compare the RPC `estimatesmartfee`)) or > listen for the > transaction announcement by other network peers to check for > propagation. > > I propose to remove "reject" messages from Bitcoin Core 0.19.0 > unless there are > valid concerns about its removal. > > Marco > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > <mailto:bitcoin-dev@lists.linuxfoundation.org> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >