Do we really need a mempool? (for relay nodes)
4 messages, 18 to 20 Jul 2015. Started by Peter Todd.
Peter Todd, Patrick Strateman, Odinn Cyberguerrilla, Jorge Timón
As in, do relay nodes need to keep a record of the transactions they've relayed? Strictly speaking, the answer is no: one a tx is relayed modulo DoS concerns the entire thing can be discarded by the node. (unconfirmed txs spending other unconfirmed txs can be handled by creating packages of transactions, evaluated as a whole)
To mitigate DoS concerns, we of course have to have some per-UTXO limit on bandwidth relayed, but that task can be accomplished by simply maintaining some kind of per-UTXO record of bandwidth used. For instance if the weighted fee and fee/KB were recorded, and forced to - say - double for each additional tx relayed that spent a given UTXO you would have a clear and simple upper limit of lifetime bandwidth. Equally it's easy to limit bandwidth moment to moment by asking peers for highest fee/KB transactions they advertise first, stopping when our bandwidth limit is reached.
You probably could even remove IsStandard() pretty much entirely with the right increasingly expensive "replacement" policy, relying on it alone to provide anti-DoS. Obviously this would simplify some of the debates around mining policy! This could even be re-used for scalable a general-purpose messaging network paid by coin ownership if the UTXO set is split up, and some kind of expiration over time policy is implemented.
Miners of course would still want to have a mempool, but that codebase may prove simpler if it doesn't have to work double-duty for relaying as well.
Relay nodes do not need a mempool, but do need some mechanism to avoid DoS issues.
Wallet nodes can use the mempool for fee estimation (in addition to looking at past blocks).
On 07/18/2015 11:52 AM, Peter Todd via bitcoin-dev wrote:
Quoted text (34 lines)
> As in, do relay nodes need to keep a record of the transactions they've > relayed? Strictly speaking, the answer is no: one a tx is relayed modulo > DoS concerns the entire thing can be discarded by the node. (unconfirmed > txs spending other unconfirmed txs can be handled by creating packages > of transactions, evaluated as a whole) > > To mitigate DoS concerns, we of course have to have some per-UTXO limit > on bandwidth relayed, but that task can be accomplished by simply > maintaining some kind of per-UTXO record of bandwidth used. For instance > if the weighted fee and fee/KB were recorded, and forced to - say - > double for each additional tx relayed that spent a given UTXO you would > have a clear and simple upper limit of lifetime bandwidth. Equally it's > easy to limit bandwidth moment to moment by asking peers for highest > fee/KB transactions they advertise first, stopping when our bandwidth > limit is reached. > > You probably could even remove IsStandard() pretty much entirely with > the right increasingly expensive "replacement" policy, relying on it > alone to provide anti-DoS. Obviously this would simplify some of the > debates around mining policy! This could even be re-used for scalable a > general-purpose messaging network paid by coin ownership if the UTXO set > is split up, and some kind of expiration over time policy is > implemented. > > Miners of course would still want to have a mempool, but that codebase > may prove simpler if it doesn't have to work double-duty for relaying as > well. > > > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Some brief thoughts, adding here a suggestion for a dynamic approach to the issue. (e.g. each additional tx relayed has some thing associated with it, that is, a "doubling" for each additional tx relayed that spends a given UTXO, doesn't sound like it would be the most dynamic approach to the issue; considering that full nodes use the (UTXOs) to establish if transactions are valid – all inputs to a transaction must be in the UTXO database for it to be valid, but rather, would end up ratcheting upward the fee/kB for each additional tx relayed, as proposed).
A more dynamic fee approach would be a better one, imho, but how is this to occur?
Quoting from Gavin Andresen's http://gavinandresen.ninja/utxo-uhoh, "The entire UTXO set doesn’t have to be in RAM; it can be stored on an SSD or spinning hard disk. The access pattern to the UTXO is not random; outputs that were spent recently are more likely to be re-spent than outputs that have not been spent in a long time. Bitcoin Core already has a multi-level UTXO cache, thanks to the hard work of Pieter Wuille."
The relay nodes would, in this scenario that is proposed here in this message, be confirming and discarding; the wallet nodes, if I understand properly, in this scenario, as proposed, should be retaining (keeping a record of the transactions they've relayed and using a mempool).
There are some questions here:
- - How is the mempool to be limited? - - What is the mechanism by which the UTXO set is stored (or proposed to be stored)? - - How would dynamic fee determinations be calculated? - - Please describe more the general purpose messaging network?
Thank you
On 07/18/2015 12:46 PM, Patrick Strateman via bitcoin-dev wrote:
Quoted text (49 lines)
> Relay nodes do not need a mempool, but do need some mechanism to > avoid DoS issues. > > Wallet nodes can use the mempool for fee estimation (in addition > to looking at past blocks). > > On 07/18/2015 11:52 AM, Peter Todd via bitcoin-dev wrote: >> As in, do relay nodes need to keep a record of the transactions >> they've relayed? Strictly speaking, the answer is no: one a tx is >> relayed modulo DoS concerns the entire thing can be discarded by >> the node. (unconfirmed txs spending other unconfirmed txs can be >> handled by creating packages of transactions, evaluated as a >> whole) >> >> To mitigate DoS concerns, we of course have to have some per-UTXO >> limit on bandwidth relayed, but that task can be accomplished by >> simply maintaining some kind of per-UTXO record of bandwidth >> used. For instance if the weighted fee and fee/KB were recorded, >> and forced to - say - double for each additional tx relayed that >> spent a given UTXO you would have a clear and simple upper limit >> of lifetime bandwidth. Equally it's easy to limit bandwidth >> moment to moment by asking peers for highest fee/KB transactions >> they advertise first, stopping when our bandwidth limit is >> reached. >> >> You probably could even remove IsStandard() pretty much entirely >> with the right increasingly expensive "replacement" policy, >> relying on it alone to provide anti-DoS. Obviously this would >> simplify some of the debates around mining policy! This could >> even be re-used for scalable a general-purpose messaging network >> paid by coin ownership if the UTXO set is split up, and some kind >> of expiration over time policy is implemented. >> >> Miners of course would still want to have a mempool, but that >> codebase may prove simpler if it doesn't have to work double-duty >> for relaying as well. >> >> >> >> _______________________________________________ bitcoin-dev >> mailing list bitcoin-dev@lists•linuxfoundation•org >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > > > > _______________________________________________ bitcoin-dev mailing > list bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >
- -- http://abis.io ~ "a protocol concept to enable decentralization and expansion of a giving economy, and a new social good" https://keybase.io/odinn
On Sat, Jul 18, 2015 at 9:46 PM, Patrick Strateman via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (5 lines)
> Relay nodes do not need a mempool, but do need some mechanism to avoid DoS > issues. > > Wallet nodes can use the mempool for fee estimation (in addition to looking > at past blocks).
Exactly, so an anti-DoS mechanism that would be sufficient for a non-mempool node would be also useful for small values in -maxmempool. I think a simple cache for transaction validations should be enough. Please, review a draft for that in the newest #6448.
https://github.com/bitcoin/bitcoin/pull/6448
I would be happy to rebase it back to 0.11 and even 0.10.