Large backlog of transactions building up?

Has anyone got long term longs that contain the pool size and timestamps?

Unfortunately I forgot to enable timestamps in the logs for my own nodes (the privacy benefit of disabling this by default is questionable, imho). But just looking at the general trends and cross-checking against my own memory it definitely seems that there are more and more pending transactions that don't get cleared into blocks.

One of my nodes now routinely has 4000 transactions in the mempool. Blocks typically clear only a few hundred at most, which is what you'd expect given current transaction rates (around 300 per ten minute interval). So what are the other pending transactions doing and why aren't they getting drained out of the mempool?

On Sun, Sep 23, 2012 at 8:12 AM, Mike Hearn <mike@plan99•net> wrote:

> Has anyone got long term longs that contain the pool size and timestamps?
>
> Unfortunately I forgot to enable timestamps in the logs for my own
> nodes (the privacy benefit of disabling this by default is
> questionable, imho). But just looking at the general trends and
> cross-checking against my own memory it definitely seems that there
> are more and more pending transactions that don't get cleared into
> blocks.
>
> One of my nodes now routinely has 4000 transactions in the mempool.
> Blocks typically clear only a few hundred at most, which is what you'd
> expect given current transaction rates (around 300 per ten minute
> interval). So what are the other pending transactions doing and why
> aren't they getting drained out of the mempool?

Yeah, my public nodes currently have 2200+ Over time, it gets cluttered naturally due to the disconnect between what miners mine and what relayers relay.

I've long argued that all mempool implementations should limit the lifetime of any TX to a specific number of blocks. Rationale: - bitcoin clients retransmit until TX is confirmed - provides a deterministic lifetime for a TX; if you KNOW a TX will disappear 144 blocks (24 hours) after you stop transmitting, then it is probably safe to initiate recovery procedures and perhaps revise the transaction - prevents zombie TXs from littering memory... they hang around, wasting resources, but never get confirmed

No one has strenuously argued against this, so I suppose it is down to writing a patch, and coming up with a good number we (as a network) can agree upon.

On Sun, Sep 23, 2012 at 4:30 PM, Jeff Garzik <jgarzik@exmulti•com> wrote:

> Yeah, my public nodes currently have 2200+  Over time, it gets
> cluttered naturally due to the disconnect between what miners mine and
> what relayers relay.

Right, this disconnect is why simple scalar measures of mempool size aren't terribly informative.

There are bursts of weird transactions (e.g. someone was flooding zero value txn a few weeks ago; before that there were some enormous series of double-spend induced orphans), and other sustained loads that quite a few miners are intentionally excluding.

> No one has strenuously argued against this, so I suppose it is down to
> writing a patch, and coming up with a good number we (as a network)
> can agree upon.

Sounds good— my only concern is that nodes will repeat their own transactions but not the unconfirmed parents. So being more aggressive can turn otherwise valid transactions into orphans.

Would there be value in an archive-mempool which is only checked when you receive an orphan transaction?

I would point out that you can't _KNOW_ a txn will disappear. Someone else could happily reannounce it. (I know you know this; but it's good to be clear on that point when we talk about it!)

> Sounds good— my only concern is that nodes will repeat their own
> transactions but not the unconfirmed parents.

Nodes repeat wallet transactions and any previous transactions that are not yet included in the chain (see CWalletTx::RelayWalletTransaction). So I don't think it's an issue.

(ok, bitcoinj clients don't do that, they just announce their own transactions)

On 9/23/12, Jeff Garzik <jgarzik@exmulti•com> wrote:

> - provides a deterministic lifetime for a TX; if you KNOW a TX will
> disappear 144 blocks (24 hours) after you stop transmitting, then it
> is probably safe to initiate recovery procedures and perhaps revise
> the transaction
> - prevents zombie TXs from littering memory... they hang around,
> wasting resources, but never get confirmed

I don't understand. Can the chain enforce this number? Why can't clients delete all those transactions right now?

On 9/23/12, Gregory Maxwell <gmaxwell@gmail•com> wrote:

> There are bursts of weird transactions (e.g. someone was flooding zero
> value txn a few weeks ago; before that there were some enormous series
> of double-spend induced orphans), and other sustained loads that quite
> a few miners are intentionally excluding.

Why clients store transactions that don't obey the current rules of the chain at all?

On Tue, Sep 25, 2012 at 1:34 PM, Jorge Timón <timon.elviejo@gmail•com> wrote:

> On 9/23/12, Jeff Garzik <jgarzik@exmulti•com> wrote:
>> - provides a deterministic lifetime for a TX; if you KNOW a TX will
>> disappear 144 blocks (24 hours) after you stop transmitting, then it
>> is probably safe to initiate recovery procedures and perhaps revise
>> the transaction
>> - prevents zombie TXs from littering memory... they hang around,
>> wasting resources, but never get confirmed
>
> I don't understand. Can the chain enforce this number?
> Why can't clients delete all those transactions right now?

This is discussion about transactions which are not in the chain yet.

> On 9/23/12, Gregory Maxwell <gmaxwell@gmail•com> wrote:
>> There are bursts of weird transactions (e.g. someone was flooding zero
>> value txn a few weeks ago; before that there were some enormous series
>> of double-spend induced orphans), and other sustained loads that quite
>> a few miners are intentionally excluding.
>
> Why clients store transactions that don't obey the current rules of
> the chain at all?

The double spending transaction is not stored— which is, in fact, the problem which creates these huge chain. When a transaction depending on the doublespend is received we do not know its parent (because we dropped it because it was a rule violation) so we keep it around as an orphan hoping its parent arrives.

The software could maintain a cache of rejected txids to consult for orphan txn's parents, but it would need to be dropped any time there is a reorg so I don't know how useful it would be.