Trying to patch Core ZMQ "rawtx" topic to only publish unconfirmed transactions: How?
5 messages, 26 to 29 Nov 2021. Started by Ali Sherief.
Ali Sherief (2), Prayank, 0xB10C, LORD HIS EXCELLENCY JAMES HRMH
This has also been posted on Bitcointalk forum: https://bitcointalk.org/index.php?topic=5373341.msg58539261#msg58539261 I have republished it here hoping someone more knowledgeable can post some insight about this. ---- It appears that the ZeroMQ topic I'm listening to, "rawtx", not only emits a raw transaction when it appears on the mempool, but once it's already confirmed too.
This messes with my software, causing it to add txids, addresses, etc. a second time inside arrays (this means that the same transaction is received twice in total).
Array de-duping is not a viable solution long-term (because the array will quickly grow to be big eventually and then this has to happen every time a new element is added), so I'm trying to nip the problem from the source by instructing Core to only publish unconfirmed bitcoin transactions.
According to https://bitcoin.stackexchange.com/questions/52848/is-it-possible-to-configure-the-bitcoin-daemon-to-only-broadcast-unconfirmed-tra , it is not possible to configure this from a configuration or command-line option. The source code must directly be edited. But since the codebase has changed greatly, the proposed solution no longer works.
----
So basically, I know that something inside src/zmq/zmqnotificationinterface.cpp needs to be patched, but I'm not sure which function, or how to do it. Because I only need unconfirmed transactions to be published on ZeroMQ rawtx and not confirmed ones, it's one of the following functions that I need to patch for my own build:
CZMQNotificationInterface::TransactionRemovedFromMempool void CZMQNotificationInterface::BlockDisconnected
Both of these call NotifyTransaction() method which I assume fires a message on "rawtx" channel.
In the Stack Exchange question I linked above, Jonas Schnelli suggested adding an `if (!pblock)` check, but that was several years ago and the function he was referencing no longer exists.
But I still wonder if the pblock check is still applicable in the present day (i.e. if it's indicating a block the transaction is inside).
- Ali Sherief
Hi Ali,
Not sure if this is exactly what you are looking for but maybe trying to solve this I might also learn few things:
Save zmqpubsequence=tcp://127.0.0.1:28332 in bitcoin.conf
Run bitcoind
Run this python script: https://pastebin.com/raw/tNp2x5y3
You will see results like this: https://i.imgur.com/xKzFJbl.png https://i.imgur.com/gpsTTHZ.png
A - Accepted, C- Connect (block) and R- Removal in the above screenshots
If you are looking for unconfirmed transactions printed in sequence I think this should help. Since transactions can be printed twice (accept,remove) in this case as well, python script can be modified to manage this IMO.
Other alternatives can be debug=mempool and reading debug.log for changes without polling.
Hi Ali,
I've run into this multiple times myself. I've opened a draft PR [0] adding a rawmempooltx publisher.
You're right. In zmq/zmqnotificationinterface.cpp the CZMQNotificationInterface is notified about TransactionAddedToMempool. Currently, this calls NotifyTransaction() (the publisher with the rawtx topic) and NotifyTransactionAcceptance() (the publisher with the sequence topic)[1]. I've added a call to a new NotifyMempoolTransaction() function (the publisher with the rawmempooltx topic).
I'd find a mempool transaction publisher with both the raw transaction and transaction fee useful too. However, this requires changes to the chain notifications in interfaces/chain.h.
[0]: https://github.com/bitcoin/bitcoin/pull/23624 [1]: https://github.com/bitcoin/bitcoin/pull/23624/files#diff-ac4b2d3a8de2c4dd41ad9d75505ea6ce4dc87a476710a9ebee8acf9bebf5cca2L146-L148
Best, 0xB10C
On 11/26/21 5:56 PM, Ali Sherief via bitcoin-dev wrote:
Quoted text (56 lines)
> > This has also been posted on Bitcointalk > forum: https://bitcointalk.org/index.php?topic=5373341.msg58539261#msg58539261 > <https://bitcointalk.org/index.php?topic=5373341.msg58539261#msg58539261> I > have republished it here hoping someone more knowledgeable can post > some insight about this. > ---- > It appears that the ZeroMQ topic I'm listening to, "rawtx", not only > emits a raw transaction when it appears on the mempool, but once it's > already confirmed too. > > This messes with my software, causing it to add txids, addresses, etc. > a second time inside arrays (this means that the same transaction is > received twice in total). > > Array de-duping is not a viable solution long-term (because the array > will quickly grow to be big eventually and then this has to happen > every time a new element is added), so I'm trying to nip the problem > from the source by instructing Core to only publish unconfirmed > bitcoin transactions. > > According to > https://bitcoin.stackexchange.com/questions/52848/is-it-possible-to-configure-the-bitcoin-daemon-to-only-broadcast-unconfirmed-tra > <https://bitcoin.stackexchange.com/questions/52848/is-it-possible-to-configure-the-bitcoin-daemon-to-only-broadcast-unconfirmed-tra> > , it is not possible to configure this from a configuration or > command-line option. The source code must directly be edited. But > since the codebase has changed greatly, the proposed solution no > longer works. > > ---- > > So basically, I know that something inside > src/zmq/zmqnotificationinterface.cpp needs to be patched, but I'm not > sure which function, or how to do it. Because I only need unconfirmed > transactions to be published on ZeroMQ rawtx and not confirmed ones, > it's one of the following functions that I need to patch for my own build: > > CZMQNotificationInterface::TransactionRemovedFromMempool > void CZMQNotificationInterface::BlockDisconnected > > Both of these call NotifyTransaction() method which I assume fires a > message on "rawtx" channel. > > In the Stack Exchange question I linked above, Jonas Schnelli > suggested adding an `if (!pblock)` check, but that was several years > ago and the function he was referencing no longer exists. > > But I still wonder if the pblock check is still applicable in the > present day (i.e. if it's indicating a block the transaction is inside). > > - Ali Sherief > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Wasn't this already not a problem because you can check if it was confirmed? The transaction is not finalised in the mempool it is just speculation of a transaction, so it makes sense to emit when the transaction is confirmed. Just already check..
Quoted text (3 lines)
> It appears that the ZeroMQ topic I'm listening to, "rawtx", not only > emits a raw transaction when it appears on the mempool, but once it's > already confirmed too.
KING JAMES HRMH Great British Empire
Regards, The Australian LORD HIS EXCELLENCY JAMES HRMH (& HMRH) of Hougun Manor & Glencoe & British Empire MR. Damian A. James Williamson Wills
et al.
Willtech www.willtech.com.au www.go-overt.com duigco.org DUIGCO API and other projects
m. 0487135719 f. +61261470192
This email does not constitute a general advice. Please disregard this email if misdelivered. ________________________________ From: bitcoin-dev <bitcoin-dev-bounces@lists•linuxfoundation•org> on behalf of 0xB10C via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> Sent: Monday, 29 November 2021 8:32 PM To: Ali Sherief <ali@notatether•com>; Bitcoin Protocol Discussion <bitcoin-dev@lists•linuxfoundation•org> Subject: Re: [bitcoin-dev] Trying to patch Core ZMQ "rawtx" topic to only publish unconfirmed transactions: How?
Hi Ali,
I've run into this multiple times myself. I've opened a draft PR [0] adding a rawmempooltx publisher.
You're right. In zmq/zmqnotificationinterface.cpp the CZMQNotificationInterface is notified about TransactionAddedToMempool. Currently, this calls NotifyTransaction() (the publisher with the rawtx topic) and NotifyTransactionAcceptance() (the publisher with the sequence topic)[1]. I've added a call to a new NotifyMempoolTransaction() function (the publisher with the rawmempooltx topic).
I'd find a mempool transaction publisher with both the raw transaction and transaction fee useful too. However, this requires changes to the chain notifications in interfaces/chain.h.
[0]: https://github.com/bitcoin/bitcoin/pull/23624 [1]: https://github.com/bitcoin/bitcoin/pull/23624/files#diff-ac4b2d3a8de2c4dd41ad9d75505ea6ce4dc87a476710a9ebee8acf9bebf5cca2L146-L148
Best, 0xB10C
On 11/26/21 5:56 PM, Ali Sherief via bitcoin-dev wrote:
Quoted text (56 lines)
> > This has also been posted on Bitcointalk > forum: https://bitcointalk.org/index.php?topic=5373341.msg58539261#msg58539261 > <https://bitcointalk.org/index.php?topic=5373341.msg58539261#msg58539261> I > have republished it here hoping someone more knowledgeable can post > some insight about this. > ---- > It appears that the ZeroMQ topic I'm listening to, "rawtx", not only > emits a raw transaction when it appears on the mempool, but once it's > already confirmed too. > > This messes with my software, causing it to add txids, addresses, etc. > a second time inside arrays (this means that the same transaction is > received twice in total). > > Array de-duping is not a viable solution long-term (because the array > will quickly grow to be big eventually and then this has to happen > every time a new element is added), so I'm trying to nip the problem > from the source by instructing Core to only publish unconfirmed > bitcoin transactions. > > According to > https://bitcoin.stackexchange.com/questions/52848/is-it-possible-to-configure-the-bitcoin-daemon-to-only-broadcast-unconfirmed-tra > <https://bitcoin.stackexchange.com/questions/52848/is-it-possible-to-configure-the-bitcoin-daemon-to-only-broadcast-unconfirmed-tra> > , it is not possible to configure this from a configuration or > command-line option. The source code must directly be edited. But > since the codebase has changed greatly, the proposed solution no > longer works. > > ---- > > So basically, I know that something inside > src/zmq/zmqnotificationinterface.cpp needs to be patched, but I'm not > sure which function, or how to do it. Because I only need unconfirmed > transactions to be published on ZeroMQ rawtx and not confirmed ones, > it's one of the following functions that I need to patch for my own build: > > CZMQNotificationInterface::TransactionRemovedFromMempool > void CZMQNotificationInterface::BlockDisconnected > > Both of these call NotifyTransaction() method which I assume fires a > message on "rawtx" channel. > > In the Stack Exchange question I linked above, Jonas Schnelli > suggested adding an `if (!pblock)` check, but that was several years > ago and the function he was referencing no longer exists. > > But I still wonder if the pblock check is still applicable in the > present day (i.e. if it's indicating a block the transaction is inside). > > - Ali Sherief > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Theoretically that would be the desired outcome for me but this change is going to be implemented as part of a casino which must display the status of new deposits that are made, even when they are unconfirmed to now. Hence why I need to receive the unconfirmed messages.
- Ali Sherief
On Mon, Nov 29, 2021 at 5:10 PM, LORD HIS EXCELLENCY JAMES HRMH <willtech@live•com•au> wrote:
Quoted text (121 lines)
> Wasn't this already not a problem because you can check if it was confirmed? The transaction is not finalised in the mempool it is just speculation of a transaction, so it makes sense to emit when the transaction is confirmed. Just already check.. > >> It appears that the ZeroMQ topic I'm listening to, "rawtx", not only >> emits a raw transaction when it appears on the mempool, but once it's >> already confirmed too. > > KING JAMES HRMH > Great British Empire > > Regards, > The Australian > LORD HIS EXCELLENCY JAMES HRMH (& HMRH) > of Hougun Manor & Glencoe & British Empire > MR. Damian A. James Williamson > Wills > > et al. > > Willtech > www.willtech.com.au > www.go-overt.com > duigco.org DUIGCO API > and other projects > > m. 0487135719 > f. +61261470192 > > This email does not constitute a general advice. Please disregard this email if misdelivered. > --------------------------------------------------------------- > > From: bitcoin-dev <bitcoin-dev-bounces@lists•linuxfoundation•org> on behalf of 0xB10C via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> > Sent: Monday, 29 November 2021 8:32 PM > To: Ali Sherief <ali@notatether•com>; Bitcoin Protocol Discussion <bitcoin-dev@lists•linuxfoundation•org> > Subject: Re: [bitcoin-dev] Trying to patch Core ZMQ "rawtx" topic to only publish unconfirmed transactions: How? > > Hi Ali, > > I've run into this multiple times myself. I've opened a draft PR [0] > adding a rawmempooltx publisher. > > You're right. In zmq/zmqnotificationinterface.cpp the > CZMQNotificationInterface is notified about TransactionAddedToMempool. > Currently, this calls NotifyTransaction() (the publisher with the rawtx > topic) and NotifyTransactionAcceptance() (the publisher with the > sequence topic)[1]. I've added a call to a new > NotifyMempoolTransaction() function (the publisher with the rawmempooltx > topic). > > I'd find a mempool transaction publisher with both the raw transaction > and transaction fee useful too. However, this requires changes to the > chain notifications in interfaces/chain.h. > > [0]: https://github.com/bitcoin/bitcoin/pull/23624 > [1]: > https://github.com/bitcoin/bitcoin/pull/23624/files#diff-ac4b2d3a8de2c4dd41ad9d75505ea6ce4dc87a476710a9ebee8acf9bebf5cca2L146-L148 > > Best, > 0xB10C > > On 11/26/21 5:56 PM, Ali Sherief via bitcoin-dev wrote: >> >> This has also been posted on Bitcointalk >> forum: https://bitcointalk.org/index.php?topic=5373341.msg58539261#msg58539261 >> <https://bitcointalk.org/index.php?topic=5373341.msg58539261#msg58539261> I >> have republished it here hoping someone more knowledgeable can post >> some insight about this. >> ---- >> It appears that the ZeroMQ topic I'm listening to, "rawtx", not only >> emits a raw transaction when it appears on the mempool, but once it's >> already confirmed too. >> >> This messes with my software, causing it to add txids, addresses, etc. >> a second time inside arrays (this means that the same transaction is >> received twice in total). >> >> Array de-duping is not a viable solution long-term (because the array >> will quickly grow to be big eventually and then this has to happen >> every time a new element is added), so I'm trying to nip the problem >> from the source by instructing Core to only publish unconfirmed >> bitcoin transactions. >> >> According to >> https://bitcoin.stackexchange.com/questions/52848/is-it-possible-to-configure-the-bitcoin-daemon-to-only-broadcast-unconfirmed-tra >> <https://bitcoin.stackexchange.com/questions/52848/is-it-possible-to-configure-the-bitcoin-daemon-to-only-broadcast-unconfirmed-tra> >> , it is not possible to configure this from a configuration or >> command-line option. The source code must directly be edited. But >> since the codebase has changed greatly, the proposed solution no >> longer works. >> >> ---- >> >> So basically, I know that something inside >> src/zmq/zmqnotificationinterface.cpp needs to be patched, but I'm not >> sure which function, or how to do it. Because I only need unconfirmed >> transactions to be published on ZeroMQ rawtx and not confirmed ones, >> it's one of the following functions that I need to patch for my own build: >> >> CZMQNotificationInterface::TransactionRemovedFromMempool >> void CZMQNotificationInterface::BlockDisconnected >> >> Both of these call NotifyTransaction() method which I assume fires a >> message on "rawtx" channel. >> >> In the Stack Exchange question I linked above, Jonas Schnelli >> suggested adding an `if (!pblock)` check, but that was several years >> ago and the function he was referencing no longer exists. >> >> But I still wonder if the pblock check is still applicable in the >> present day (i.e. if it's indicating a block the transaction is inside). >> >> - Ali Sherief >> >> _______________________________________________ >> bitcoin-dev mailing list >> bitcoin-dev@lists•linuxfoundation•org >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev