Mempool "Expected Byte Stay" policy

Spammers out there are being very disrepectful of my fullnode resources these days! I'm making some changes. In case others are interested, here's a description:

There is now a maximum size for the memory pool. Space is allocated with a pretty simple rule. For each tx, I calculate MY COST of continuing to hold it in the mempool. I measure the cost to me by "expected byte stay":

expectedByteStay = sizeBytes * expectedBlocksToConfirm(feeRate)

Rule 1: When there's not enough space for a new tx, I try to make space by evicting txes with expectedByteStay higher than tx.

I'm NOT worrying about
  - Fees
    EXCEPT via their effect on confirmation time

  - Coin age
    You already made money on your old coins.  Pay up.

  - CPFP
    Child's expectedBlocksToConfirm is max'ed with its
    parent, then parent expectedByteStay is ADDED to child's

  - Replacement
    You'll get another chance in 2 hours (see below).

Rule 2: A transaction and its dependents are evicted on its 2-hour anniversary, whether space is required or not

The latest expectedBlocksToConfirm(feeRate) table is applied to the entire mempool periodically.

What do you think? I'll let you know how it works out. I'm putting a lot of faith in the new fee estimation (particularly its size independence). Another possibility is clog-ups by transactions that look like they'll confirm next block, but don't because of factors other than fees (other people's blacklists?)

On Tuesday 14. July 2015 17.24.23 Tom Harding via bitcoin-dev wrote:

> Rule 2: A transaction and its dependents are evicted on its 2-hour
> anniversary, whether space is required or not

Instead of 2 hours, why not a number of blocks?

On 7/15/2015 12:18 PM, Thomas Zander via bitcoin-dev wrote:

> On Tuesday 14. July 2015 17.24.23 Tom Harding via bitcoin-dev wrote:
>> Rule 2: A transaction and its dependents are evicted on its 2-hour
>> anniversary, whether space is required or not
> Instead of 2 hours, why not a number of blocks?

So users/wallets can know when they should rebroadcast and consider increasing the fee.

Using 12 blocks, there is a 5% chance he has to wait 3 hours.*

Using 120 minutes, there is only a .23% chance that fewer than 4 blocks have occurred.**

* Table[{x, 1 - CDF[ErlangDistribution[12, 1/10], x]}, {x, 0, 240, 10}] //N //TableForm 0. 1. 10. 1. 20. 0.999999 30. 0.999929 40. 0.999085 50. 0.994547 60. 0.979908 70. 0.94665 80. 0.888076 90. 0.803008 100. 0.696776 110. 0.579267 120. 0.461597 130. 0.353165 140. 0.26004 150. 0.184752 160. 0.126993 170. 0.0846691 180. 0.0548874 190. 0.0346726 200. 0.0213868 210. 0.0129048 220. 0.00762994 230. 0.00442702 240. 0.00252413

** Table[{x, CDF[PoissonDistribution[1/10 * 120], x]}, {x, 0, 20}] //N //TableForm 0. 6.14421*10^-6 1. 0.0000798748 2. 0.000522258 3. 0.00229179 4. 0.00760039 5. 0.020341 6. 0.0458223 7. 0.0895045 8. 0.155028 9. 0.242392 10. 0.347229 11. 0.461597 12. 0.575965 13. 0.681536 14. 0.772025 15. 0.844416 16. 0.898709 17. 0.937034 18. 0.962584 19. 0.97872 20. 0.988402

On Wednesday 15. July 2015 16.15.24 Tom Harding via bitcoin-dev wrote:

> On 7/15/2015 12:18 PM, Thomas Zander via bitcoin-dev wrote:
> > On Tuesday 14. July 2015 17.24.23 Tom Harding via bitcoin-dev wrote:
> >> Rule 2: A transaction and its dependents are evicted on its 2-hour
> >> anniversary, whether space is required or not
> >
> > Instead of 2 hours, why not a number of blocks?
>
> So users/wallets can know when they should rebroadcast and consider
> increasing the fee.
>
>
> Using 12 blocks, there is a 5% chance he has to wait 3 hours.*
>
> Using 120 minutes, there is only a .23% chance that fewer than 4 blocks
> have occurred.**

Using the good old saying; results in the past are no indication of the future. I see a logic error in your thinking.

Your assumption that time is a better indicator is false. Naturally time itself is universal, but blocks are known by wallets too. Its just as good.

This assumption of yours leans heavily on block mining times, and that is not guaranteed in the future. Imagine one day half the miners dropping and blocks take much longer for a week or so. Your assumptions just broke the mempool.

On 7/16/2015 2:38 AM, Thomas Zander via bitcoin-dev wrote:

> On Wednesday 15. July 2015 16.15.24 Tom Harding via bitcoin-dev wrote:
>> On 7/15/2015 12:18 PM, Thomas Zander via bitcoin-dev wrote:
>>> On Tuesday 14. July 2015 17.24.23 Tom Harding via bitcoin-dev wrote:
>>>> Rule 2: A transaction and its dependents are evicted on its 2-hour
>>>> anniversary, whether space is required or not
>>> Instead of 2 hours, why not a number of blocks?
>> So users/wallets can know when they should rebroadcast and consider
>> increasing the fee.
>>
>>
>> Using 12 blocks, there is a 5% chance he has to wait 3 hours.*
>>
>> Using 120 minutes, there is only a .23% chance that fewer than 4 blocks
>> have occurred.**
> Using the good old saying; results in the past are no indication of the
> future.
> I see a logic error in your thinking.
>
> Your assumption that time is a better indicator is false. Naturally time
> itself is universal, but blocks are known by wallets too. Its just as good.
>
> This assumption of yours leans heavily on block mining times, and that is
> not guaranteed in the future.  Imagine one day half the miners dropping and
> blocks take much longer for a week or so.  Your assumptions just broke the
> mempool.
>

It's not a question of right vs. wrong. Either method has consequences for user expectations and behavior.

With fixed-block mempool expiration, the expiration time is variable. User can get an alert, but at an unpredictable time.

With fixed-timeout, the likelihood of expiration is more variable (expiration occurrence is unpredictable regardless), but any expiration will occur at the timeout.