Memory leaks?
26 messages, 13 to 23 Oct 2015. Started by Jonathan Toomim.
Jonathan Toomim (8), Multipool Admin (4), Odinn Cyberguerrilla (3), Dave Scotese (2), Tom Zander (2), Mike Hearn (2), Pavel Janík, Wladimir J. van der Laan, +3 more
I just noticed that several of my running bitcoind processes were using around 3+ GB of RAM, even though the mempool itself seemed to be under control.
XXXX@prime:~/bin$ ./bitcoin-cli getmempoolinfo
{
"size" : 1896,
"bytes" : 37341328
}
[total memory usage not shown -- I restarted bitcoind as soon as I noticed, and didn't copy it down from top]
37 MB mempool, >3 GB RAM usage. Normally, when there aren't a lot of unconfirmed txns floating around the network, memory usage is around 600 MB, so this is quite unusual.
After restarting the process and letting it run for a few minutes, I get:
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command [###] [XXXX] 20 0 1402M 317M 49836 S 1.0 8.2 0:41.71 ./bitcoind -daemon
XXXX@prime:~/bin$ ./bitcoin-cli getmempoolinfo
{
"size" : 1072,
"bytes" : 670000
}
0.67 MB mempool, 317 MB RAM usage. Much more reasonable.
Here's another node I'm running that has been online longer, before restarting:
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command [###] [XXXX] 20 0 4961M 3540M 11080 S 2.8 45.3 8h20:11 bin/bitcoind -daemon
XXXX@feather:~$ bin/bitcoin-cli getmempoolinfo
{
"size" : 3045,
"bytes" : 39656126
}
39 MB mempool, 3540 MB total memory usage. After restarting bitcoind, I see:
[XXXX]@feather:~$ bin/bitcoin-cli stop
Bitcoin server stopping
[XXXX]@feather:~$ bin/bitcoind -daemon
Bitcoin server starting
[XXXX]@feather:~$ sleep 10; bin/bitcoin-cli getmempoolinfo
{
"size" : 39,
"bytes" : 47037
}
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command [###] [XXXX] 20 0 1640M 247M 67960 S 0.0 3.2 0:05.17 bin/bitcoind -daemon
Does anybody have any guesses where we might be leaking memory, or what is using the additional 2.4 GB? I've been using minrelaytxfee=0.00003 or similar on my nodes. Maybe there's a leak in the minrelaytxfee code path? Has anyone else seen something similar?
This issue appears to happen both with Bitcoin Core 0.10.1 and with Bitcoin XT 0.11B.
You should compare this to having set up a node on a completely clean computer. It would also help to know what operating system(s) you are using for both the oldie and the freshie.
Also, dump your XT, is poo. Then try again, look at Core nodes on your oldie and freshie. Watch them for a bit.
Cheers,
O
Jonathan Toomim (Toomim Bros) via bitcoin-dev:
Quoted text (68 lines)
> I just noticed that several of my running bitcoind processes were
> using around 3+ GB of RAM, even though the mempool itself seemed to
> be under control.
>
> XXXX@prime:~/bin$ ./bitcoin-cli getmempoolinfo { "size" : 1896,
> "bytes" : 37341328 }
>
> [total memory usage not shown -- I restarted bitcoind as soon as I
> noticed, and didn't copy it down from top]
>
> 37 MB mempool, >3 GB RAM usage. Normally, when there aren't a lot
> of unconfirmed txns floating around the network, memory usage is
> around 600 MB, so this is quite unusual.
>
> After restarting the process and letting it run for a few minutes,
> I get:
>
> PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+
> Command [###] [XXXX] 20 0 1402M 317M 49836 S 1.0 8.2
> 0:41.71 ./bitcoind -daemon
>
> XXXX@prime:~/bin$ ./bitcoin-cli getmempoolinfo { "size" : 1072,
> "bytes" : 670000 }
>
> 0.67 MB mempool, 317 MB RAM usage. Much more reasonable.
>
>
> Here's another node I'm running that has been online longer, before
> restarting:
>
> PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+
> Command [###] [XXXX] 20 0 4961M 3540M 11080 S 2.8 45.3
> 8h20:11 bin/bitcoind -daemon
>
> XXXX@feather:~$ bin/bitcoin-cli getmempoolinfo { "size" : 3045,
> "bytes" : 39656126 }
>
> 39 MB mempool, 3540 MB total memory usage. After restarting
> bitcoind, I see:
>
> [XXXX]@feather:~$ bin/bitcoin-cli stop Bitcoin server stopping
> [XXXX]@feather:~$ bin/bitcoind -daemon Bitcoin server starting
> [XXXX]@feather:~$ sleep 10; bin/bitcoin-cli getmempoolinfo { "size"
> : 39, "bytes" : 47037 }
>
>
> PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+
> Command [###] [XXXX] 20 0 1640M 247M 67960 S 0.0 3.2
> 0:05.17 bin/bitcoind -daemon
>
>
>
>
> Does anybody have any guesses where we might be leaking memory, or
> what is using the additional 2.4 GB? I've been using
> minrelaytxfee=0.00003 or similar on my nodes. Maybe there's a leak
> in the minrelaytxfee code path? Has anyone else seen something
> similar?
>
> This issue appears to happen both with Bitcoin Core 0.10.1 and with
> Bitcoin XT 0.11B.
>
>
>
> _______________________________________________ bitcoin-dev mailing
> list bitcoin-dev@lists•linuxfoundation•org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>- -- http://abis.io ~ "a protocol concept to enable decentralization and expansion of a giving economy, and a new social good" https://keybase.io/odinn
On Oct 13, 2015, at 3:49 PM, odinn <odinn.cyberguerrilla@riseup•net> wrote:
Quoted text (3 lines)
> Signed PGP part > It would also help to know what operating system(s) you are > using for both the oldie and the freshie.
Linux feather 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04) x86_64 GNU/Linux Linux server 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux Linux prime 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2+deb7u2 x86_64 GNU/Linux
This excessive memory consumption was seen on 3 machines, all of which run Debian. All three machines run p2pool as well as bitcoind. Two run XT, one runs Core.
Quoted text (3 lines)
> > You should compare this to having set up a node on a completely clean > computer.
I can't afford to do that. All of the servers I have are being used for something. Also, I'm not sure what it is you're trying to test for with that suggestion. The numbers I'm reporting are for bitcoind's resident set, not for the whole server's memory usage. I don't see how other processes running on the same machine are relevant unless you are suggesting that RPC calls (e.g. getblocktemplate) might be somehow responsible.
Quoted text (2 lines)
> > Also, dump your XT, is poo.
Not relevant. I addressed this message to both the Core and XT lists because the issue appears to affect both forks. Let's keep blocksize and governance debates to their own threads, please.
Repeating request: Has anyone else seen something similar? Can you report your mempool size and total bitcoind resident set size for your running full nodes?
{ "size" : 1085, "bytes" : 16151768 }
It has been running about a day. I'll report tomorrow too. This is a Windows 8.1 box.
16 million divided by 1085 transactions is almost 15Kb per transaction = unlikely, right?
On Tue, Oct 13, 2015 at 4:14 PM, Jonathan Toomim (Toomim Bros) <j@toom•im> wrote:
Quoted text (47 lines)
> > On Oct 13, 2015, at 3:49 PM, odinn <odinn.cyberguerrilla@riseup•net> > wrote: > > Signed PGP part > It would also help to know what operating system(s) you are > using for both the oldie and the freshie. > > > Linux feather 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3 > (2015-08-04) x86_64 GNU/Linux > Linux server 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux > Linux prime 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2+deb7u2 x86_64 GNU/Linux > > This excessive memory consumption was seen on 3 machines, all of which run > Debian. All three machines run p2pool as well as bitcoind. Two run XT, one > runs Core. > > You should compare this to having set up a node on a completely clean > computer. > > > I can't afford to do that. All of the servers I have are being used for > something. Also, I'm not sure what it is you're trying to test for with > that suggestion. The numbers I'm reporting are for bitcoind's resident set, > not for the whole server's memory usage. I don't see how other processes > running on the same machine are relevant unless you are suggesting that RPC > calls (e.g. getblocktemplate) might be somehow responsible. > > Also, dump your XT, is poo. > > > Not relevant. I addressed this message to both the Core and XT lists > because the issue appears to affect both forks. Let's keep blocksize and > governance debates to their own threads, please. > > Repeating request: Has anyone else seen something similar? Can you report > your mempool size and total bitcoind resident set size for your running > full nodes? > > -- > You received this message because you are subscribed to the Google Groups > "bitcoin-xt" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to bitcoin-xt+unsubscribe@googlegroups•com. > For more options, visit https://groups.google.com/d/optout. >
You wanted advice... you got it
Jonathan Toomim (Toomim Bros):
Quoted text (41 lines)
> > On Oct 13, 2015, at 3:49 PM, odinn > <odinn.cyberguerrilla@riseup•net> wrote: > >> Signed PGP part It would also help to know what operating >> system(s) you are using for both the oldie and the freshie. > > Linux feather 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3 > (2015-08-04) x86_64 GNU/Linux Linux server 3.2.0-4-amd64 #1 SMP > Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux Linux prime 3.2.0-4-amd64 > #1 SMP Debian 3.2.63-2+deb7u2 x86_64 GNU/Linux > > This excessive memory consumption was seen on 3 machines, all of > which run Debian. All three machines run p2pool as well as > bitcoind. Two run XT, one runs Core. > >> >> You should compare this to having set up a node on a completely >> clean computer. > > > I can't afford to do that. All of the servers I have are being used > for something. Also, I'm not sure what it is you're trying to test > for with that suggestion. The numbers I'm reporting are for > bitcoind's resident set, not for the whole server's memory usage. I > don't see how other processes running on the same machine are > relevant unless you are suggesting that RPC calls (e.g. > getblocktemplate) might be somehow responsible. > >> >> Also, dump your XT, is poo. > > > Not relevant. I addressed this message to both the Core and XT > lists because the issue appears to affect both forks. Let's keep > blocksize and governance debates to their own threads, please. > > Repeating request: Has anyone else seen something similar? Can you > report your mempool size and total bitcoind resident set size for > your running full nodes? >
- -- http://abis.io ~ "a protocol concept to enable decentralization and expansion of a giving economy, and a new social good" https://keybase.io/odinn
Quoted text (1 line)
> 16 million divided by 1085 transactions is almost 15Kb per transaction = unlikely, right?
The recent spam was about 15 kB per transaction, so that part sounds right.
The anomalous thing that I saw was that the total bitcoind process usage was about 50-100x higher than I would have expected if the mempool was the main determinant of memory usage scaling. Can you tell me how much memory Task Manager is reporting your bitcoin process as using both today and tomorrow?
On Oct 13, 2015, at 4:52 PM, Dave Scotese via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (8 lines)
>
>
> {
> "size" : 1085,
> "bytes" : 16151768
> }
> It has been running about a day. I'll report tomorrow too. This is a Windows 8.1 box.
> 16 million divided by 1085 transactions is almost 15Kb per transaction = unlikely, right?It was about 360MB (30 minutes ago?), but is now about 460MB. I'm sure it won't keep going up that fast. { "size" : 3413, "bytes" : 41892350 }
On Tue, Oct 13, 2015 at 5:08 PM, Jonathan Toomim (Toomim Bros) <j@toom•im> wrote:
Quoted text (33 lines)
> 16 million divided by 1085 transactions is almost 15Kb per transaction =
> unlikely, right?
>
>
> The recent spam was about 15 kB per transaction, so that part sounds right.
>
> The anomalous thing that I saw was that the total bitcoind process usage
> was about 50-100x higher than I would have expected if the mempool was the
> main determinant of memory usage scaling. Can you tell me how much memory
> Task Manager is reporting your bitcoin process as using both today and
> tomorrow?
>
> On Oct 13, 2015, at 4:52 PM, Dave Scotese via bitcoin-dev <
> bitcoin-dev@lists•linuxfoundation•org> wrote:
>
>
> {
> "size" : 1085,
> "bytes" : 16151768
> }
> It has been running about a day. I'll report tomorrow too. This is a
> Windows 8.1 box.
> 16 million divided by 1085 transactions is almost 15Kb per transaction =
> unlikely, right?
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "bitcoin-xt" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bitcoin-xt+unsubscribe@googlegroups•com.
> For more options, visit https://groups.google.com/d/optout.
>Quoted text (9 lines)
> I just noticed that several of my running bitcoind processes were using around 3+ GB of RAM, even though the mempool itself seemed to be under control.
>
> XXXX@prime:~/bin$ ./bitcoin-cli getmempoolinfo
> {
> "size" : 1896,
> "bytes" : 37341328
> }
>
> [total memory usage not shown -- I restarted bitcoind as soon as I noticed, and didn't copy it down from top]The newer versions contain much more accurate reporting of memorypool usage that reflect the real memory usage:
{
"size": 58536,
"bytes": 770044257,
"usage": 1652554832
}
See PR #6410 (Implement accurate memory accounting for mempool). -- Pavel Janík
On Tuesday 13 Oct 2015 14:56:08 Jonathan Toomim via bitcoin-dev wrote:
Quoted text (4 lines)
> Does anybody have any guesses where we might be leaking memory, or what is > using the additional 2.4 GB? I've been using minrelaytxfee=0.00003 or > similar on my nodes. Maybe there's a leak in the minrelaytxfee code path? > Has anyone else seen something similar?
I suggest running it in valgrind with --leak-check=full for 10 minutes.
valgrind --leak-check=full src/bitcoind 2>&1 | tee out
This at least will show you any memory leaks at exit. Naturally, the leaks you observe may just be design issues where cache can grow to much and when the cache is cleaned on shutdown you won't see it in the valgrind output.
Leaks are not the only explanation possible. Caches and fragmentation can also give this sort of effect. Unfortunately the tools to debug this aren't great. You could try a build with tcmalloc and use it to investigate heap stats.
Odinn, trolling like a 3 year old will get you swiftly banned. Last warning. On 14 Oct 2015 9:58 am, "Tom Zander" <tomz@freedommail•ch> wrote:
Quoted text (24 lines)
> On Tuesday 13 Oct 2015 14:56:08 Jonathan Toomim via bitcoin-dev wrote: > > Does anybody have any guesses where we might be leaking memory, or what > is > > using the additional 2.4 GB? I've been using minrelaytxfee=0.00003 or > > similar on my nodes. Maybe there's a leak in the minrelaytxfee code path? > > Has anyone else seen something similar? > > I suggest running it in valgrind with --leak-check=full for 10 minutes. > > valgrind --leak-check=full src/bitcoind 2>&1 | tee out > > This at least will show you any memory leaks at exit. > Naturally, the leaks you observe may just be design issues where cache can > grow to much and when the cache is cleaned on shutdown you won't see it in > the > valgrind output. > > -- > You received this message because you are subscribed to the Google Groups > "bitcoin-xt" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to bitcoin-xt+unsubscribe@googlegroups•com. > For more options, visit https://groups.google.com/d/optout. >
On Tue, Oct 13, 2015 at 02:56:08PM -0700, Jonathan Toomim (Toomim Bros) via bitcoin-dev wrote:
Quoted text (9 lines)
> I just noticed that several of my running bitcoind processes were using around 3+ GB of RAM, even though the mempool itself seemed to be under control.
>
> XXXX@prime:~/bin$ ./bitcoin-cli getmempoolinfo
> {
> "size" : 1896,
> "bytes" : 37341328
> }
>
> [total memory usage not shown -- I restarted bitcoind as soon as I noticed, and didn't copy it down from top]This is *most likely* the mempool, but is just not reported correctly.
In the meantime you can mitigate the mempool growth by setting `-mintxfee`, see https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
This will be the default in 0.11.1 and 0.10.3. 0.12.0 will have better mempool management, as well as more precise reporting (as paveljanik already mentions.)
Wladimir
For the record, Mr. Hearn, you do not own this list. I submit to you that you have very little to say on this matter at this stage and your idle threats to "ban people" based on their preferences, suggestions, or characterizations of your chosen software project are at best very silly.
I am not and never was on bitcoin-xt@googlegroups•com and thus have clipped that from the reply-to.
Mike Hearn via bitcoin-dev:
Quoted text (42 lines)
> Leaks are not the only explanation possible. Caches and > fragmentation can also give this sort of effect. Unfortunately the > tools to debug this aren't great. You could try a build with > tcmalloc and use it to investigate heap stats. > > Odinn, trolling like a 3 year old will get you swiftly banned. Last > warning. On 14 Oct 2015 9:58 am, "Tom Zander" <tomz@freedommail•ch> > wrote: > >> On Tuesday 13 Oct 2015 14:56:08 Jonathan Toomim via bitcoin-dev >> wrote: >>> Does anybody have any guesses where we might be leaking memory, >>> or what >> is >>> using the additional 2.4 GB? I've been using >>> minrelaytxfee=0.00003 or similar on my nodes. Maybe there's a >>> leak in the minrelaytxfee code path? Has anyone else seen >>> something similar? >> >> I suggest running it in valgrind with --leak-check=full for 10 >> minutes. >> >> valgrind --leak-check=full src/bitcoind 2>&1 | tee out >> >> This at least will show you any memory leaks at exit. Naturally, >> the leaks you observe may just be design issues where cache can >> grow to much and when the cache is cleaned on shutdown you won't >> see it in the valgrind output. >> >> -- You received this message because you are subscribed to the >> Google Groups "bitcoin-xt" group. To unsubscribe from this group >> and stop receiving emails from it, send an email to >> bitcoin-xt+unsubscribe@googlegroups•com. For more options, visit >> https://groups.google.com/d/optout. >> > > > > _______________________________________________ bitcoin-dev mailing > list bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >
- -- http://abis.io ~ "a protocol concept to enable decentralization and expansion of a giving economy, and a new social good" https://keybase.io/odinn
I can do better than that:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10625 ******** 20 0 41.310g 0.035t 51172 S 1.0 57.6 47:11.17 bitcoind
Any idea what's going on here?
$ cat /proc/10625/status|grep -i VM VmPeak: 43647420 kB VmSize: 43319932 kB VmLck: 60 kB VmPin: 0 kB VmHWM: 38286252 kB VmRSS: 38014300 kB VmData: 43103812 kB VmStk: 236 kB VmExe: 8844 kB VmLib: 194656 kB VmPTE: 79932 kB VmSwap: 0 kB
On Tuesday, October 13, 2015 at 2:56:15 PM UTC-7, Jonathan Toomim (Toomim Bros) wrote:
Quoted text (75 lines)
>
> I just noticed that several of my running bitcoind processes were using
> around 3+ GB of RAM, even though the mempool itself seemed to be under
> control.
>
> XXXX@prime:~/bin$ ./bitcoin-cli getmempoolinfo
> {
> "size" : 1896,
> "bytes" : 37341328
> }
>
> [total memory usage not shown -- I restarted bitcoind as soon as I
> noticed, and didn't copy it down from top]
>
> 37 MB mempool, >3 GB RAM usage. Normally, when there aren't a lot of
> unconfirmed txns floating around the network, memory usage is around 600
> MB, so this is quite unusual.
>
> After restarting the process and letting it run for a few minutes, I get:
>
> PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
> [###] [XXXX] 20 0 1402M 317M 49836 S 1.0 8.2 0:41.71 ./bitcoind
> -daemon
>
> XXXX@prime:~/bin$ ./bitcoin-cli getmempoolinfo
> {
> "size" : 1072,
> "bytes" : 670000
> }
>
> 0.67 MB mempool, 317 MB RAM usage. Much more reasonable.
>
>
> Here's another node I'm running that has been online longer, before
> restarting:
>
> PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
> [###] [XXXX] 20 0 4961M 3540M 11080 S 2.8 45.3 8h20:11
> bin/bitcoind -daemon
>
> XXXX@feather:~$ bin/bitcoin-cli getmempoolinfo
> {
> "size" : 3045,
> "bytes" : 39656126
> }
>
> 39 MB mempool, 3540 MB total memory usage. After restarting bitcoind, I
> see:
>
> [XXXX]@feather:~$ bin/bitcoin-cli stop
> Bitcoin server stopping
> [XXXX]@feather:~$ bin/bitcoind -daemon
> Bitcoin server starting
> [XXXX]@feather:~$ sleep 10; bin/bitcoin-cli getmempoolinfo
> {
> "size" : 39,
> "bytes" : 47037
> }
>
>
> PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
> [###] [XXXX] 20 0 1640M 247M 67960 S 0.0 3.2 0:05.17
> bin/bitcoind -daemon
>
>
>
>
> Does anybody have any guesses where we might be leaking memory, or what is
> using the additional 2.4 GB? I've been using minrelaytxfee=0.00003 or
> similar on my nodes. Maybe there's a leak in the minrelaytxfee code path?
> Has anyone else seen something similar?
>
> This issue appears to happen both with Bitcoin Core 0.10.1 and with
> Bitcoin XT 0.11B.
>On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail•com> wrote:
Quoted text (1 line)
> This is *most likely* the mempool, but is just not reported correctly.
I did some testing with PR #6410's better mempool reporting. The improved reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is about 2.5x to 3x higher than the serialized transaction sizes ("bytes":). The excess memory usage that I'm seeing is on the order of 100x higher than the mempool "bytes": value. As such, I think it's unlikely that this is the mempool, or at least not normal/correct mempool behavior.
Another user (admin@multipool•us) reported 35 GB of RSS usage. I'm guessing his bitcoind has been running longer than any of mine. His server definitely has more RAM. I don't know which email list he is subscribed to (probably XT), so I'm sharing it with both lists to make sure you're all aware of how big an issue this can be.
Quoted text (2 lines)
> In the meantime you can mitigate the mempool growth by setting `-mintxfee`, see > https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
I have mintxfee and minrelaytxfee set to about 0.00003, which is high enough to exclude essentially all of the of the 14700-14800 byte flood transactions. My nodes' mempools only contain about one or two blocks' worth of transactions. So I don't think this is correct either.
Some additional notes on this issue:
1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a Core bitcoind process (commit d78a880) overnight with no mining connected to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes, and RSS climbed to 1180 MB. An hour after that with more getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.] getmempoolinfo is still showing "usage" around 25MB or less.
I'll do some more testing with this and see if I can make it repeatable, and record the results more carefully. Expect a follow-up from me in a day or two.
2. valgrind did not show anything super promising. It did report this:
==6880== LEAK SUMMARY: ==6880== definitely lost: 0 bytes in 0 blocks ==6880== indirectly lost: 0 bytes in 0 blocks ==6880== possibly lost: 288 bytes in 1 blocks ==6880== still reachable: 10,552 bytes in 39 blocks ==6880== suppressed: 0 bytes in 0 blocks (Bitcoin Core commit d78a880)
and this: ==6778== LEAK SUMMARY: ==6778== definitely lost: 0 bytes in 0 blocks ==6778== indirectly lost: 0 bytes in 0 blocks ==6778== possibly lost: 320 bytes in 1 blocks ==6778== still reachable: 10,080 bytes in 32 blocks ==6778== suppressed: 0 bytes in 0 blocks (Bitcoin XT commit fe446d)
I haven't found anything in there yet that I think would produce the multi-GB memory usage after running for a few days, but I could be missing it. Email me if you want the full log.
I did not try running getblocktemplate while valgrind was running. I'll have to try that. I also have not let valgrind run for more than an hour.
P.S.: Sorry for all the cross-post confusion and consequent flamewar fallout. While it's probably too late for this thread, I'll make sure to post in a manner that keeps the threads clearly separate in the future (e.g. different subject lines).
My nodes are continuously running getblocktemplate and getinfo, and I also suspected the issue is in either gbt or the rpc server.
The instance only takes a few hours to get up to that memory usage. On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" < bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (87 lines)
> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail•com>
> wrote:
>
> This is *most likely* the mempool, but is just not reported correctly.
>
>
> I did some testing with PR #6410's better mempool reporting. The improved
> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
> The excess memory usage that I'm seeing is on the order of 100x higher than
> the mempool "bytes": value. As such, I think it's unlikely that this is the
> mempool, or at least not normal/correct mempool behavior.
>
> Another user (admin@multipool•us) reported 35 GB of RSS usage. I'm
> guessing his bitcoind has been running longer than any of mine. His server
> definitely has more RAM. I don't know which email list he is subscribed to
> (probably XT), so I'm sharing it with both lists to make sure you're all
> aware of how big an issue this can be.
>
> In the meantime you can mitigate the mempool growth by setting
> `-mintxfee`, see
>
> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>
>
> I have mintxfee and minrelaytxfee set to about 0.00003, which is high
> enough to exclude essentially all of the of the 14700-14800 byte flood
> transactions. My nodes' mempools only contain about one or two blocks'
> worth of transactions. So I don't think this is correct either.
>
>
>
> Some additional notes on this issue:
>
> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
> Core bitcoind process (commit d78a880) overnight with no mining connected
> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
> getmempoolinfo is still showing "usage" around 25MB or less.
>
> I'll do some more testing with this and see if I can make it repeatable,
> and record the results more carefully. Expect a follow-up from me in a day
> or two.
>
> 2. valgrind did not show anything super promising. It did report this:
>
> ==6880== LEAK SUMMARY:
> ==6880== definitely lost: 0 bytes in 0 blocks
> ==6880== indirectly lost: 0 bytes in 0 blocks
> ==6880== possibly lost: 288 bytes in 1 blocks
> ==6880== still reachable: 10,552 bytes in 39 blocks
> ==6880== suppressed: 0 bytes in 0 blocks
> (Bitcoin Core commit d78a880)
>
> and this:
> ==6778== LEAK SUMMARY:
> ==6778== definitely lost: 0 bytes in 0 blocks
> ==6778== indirectly lost: 0 bytes in 0 blocks
> ==6778== possibly lost: 320 bytes in 1 blocks
> ==6778== still reachable: 10,080 bytes in 32 blocks
> ==6778== suppressed: 0 bytes in 0 blocks
> (Bitcoin XT commit fe446d)
>
> I haven't found anything in there yet that I think would produce the
> multi-GB memory usage after running for a few days, but I could be missing
> it. Email me if you want the full log.
>
> I did not try running getblocktemplate while valgrind was running. I'll
> have to try that. I also have not let valgrind run for more than an hour.
>
>
>
> P.S.: Sorry for all the cross-post confusion and consequent flamewar
> fallout. While it's probably too late for this thread, I'll make sure to
> post in a manner that keeps the threads clearly separate in the future
> (e.g. different subject lines).
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation•org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>I should also mention that this is definitely not an attack coming from connected nodes. My node experiencing the issue is only connected to 3 other nodes, all of which I control (via connect=).
--Adam
On Mon, Oct 19, 2015 at 12:17 PM, Multipool Admin <admin@multipool•us> wrote:
Quoted text (94 lines)
> My nodes are continuously running getblocktemplate and getinfo, and I also
> suspected the issue is in either gbt or the rpc server.
>
> The instance only takes a few hours to get up to that memory usage.
> On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <
> bitcoin-dev@lists•linuxfoundation•org> wrote:
>
>> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail•com>
>> wrote:
>>
>> This is *most likely* the mempool, but is just not reported correctly.
>>
>>
>> I did some testing with PR #6410's better mempool reporting. The improved
>> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
>> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
>> The excess memory usage that I'm seeing is on the order of 100x higher than
>> the mempool "bytes": value. As such, I think it's unlikely that this is the
>> mempool, or at least not normal/correct mempool behavior.
>>
>> Another user (admin@multipool•us) reported 35 GB of RSS usage. I'm
>> guessing his bitcoind has been running longer than any of mine. His server
>> definitely has more RAM. I don't know which email list he is subscribed to
>> (probably XT), so I'm sharing it with both lists to make sure you're all
>> aware of how big an issue this can be.
>>
>> In the meantime you can mitigate the mempool growth by setting
>> `-mintxfee`, see
>>
>> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>>
>>
>> I have mintxfee and minrelaytxfee set to about 0.00003, which is high
>> enough to exclude essentially all of the of the 14700-14800 byte flood
>> transactions. My nodes' mempools only contain about one or two blocks'
>> worth of transactions. So I don't think this is correct either.
>>
>>
>>
>> Some additional notes on this issue:
>>
>> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
>> Core bitcoind process (commit d78a880) overnight with no mining connected
>> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
>> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
>> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
>> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
>> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
>> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
>> getmempoolinfo is still showing "usage" around 25MB or less.
>>
>> I'll do some more testing with this and see if I can make it repeatable,
>> and record the results more carefully. Expect a follow-up from me in a day
>> or two.
>>
>> 2. valgrind did not show anything super promising. It did report this:
>>
>> ==6880== LEAK SUMMARY:
>> ==6880== definitely lost: 0 bytes in 0 blocks
>> ==6880== indirectly lost: 0 bytes in 0 blocks
>> ==6880== possibly lost: 288 bytes in 1 blocks
>> ==6880== still reachable: 10,552 bytes in 39 blocks
>> ==6880== suppressed: 0 bytes in 0 blocks
>> (Bitcoin Core commit d78a880)
>>
>> and this:
>> ==6778== LEAK SUMMARY:
>> ==6778== definitely lost: 0 bytes in 0 blocks
>> ==6778== indirectly lost: 0 bytes in 0 blocks
>> ==6778== possibly lost: 320 bytes in 1 blocks
>> ==6778== still reachable: 10,080 bytes in 32 blocks
>> ==6778== suppressed: 0 bytes in 0 blocks
>> (Bitcoin XT commit fe446d)
>>
>> I haven't found anything in there yet that I think would produce the
>> multi-GB memory usage after running for a few days, but I could be missing
>> it. Email me if you want the full log.
>>
>> I did not try running getblocktemplate while valgrind was running. I'll
>> have to try that. I also have not let valgrind run for more than an hour.
>>
>>
>>
>> P.S.: Sorry for all the cross-post confusion and consequent flamewar
>> fallout. While it's probably too late for this thread, I'll make sure to
>> post in a manner that keeps the threads clearly separate in the future
>> (e.g. different subject lines).
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists•linuxfoundation•org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>OK, then running under Valgrind whilst sending gbt RPCs would be the next step.
On Mon, Oct 19, 2015 at 9:17 PM, Multipool Admin <admin@multipool•us> wrote:
Quoted text (100 lines)
> My nodes are continuously running getblocktemplate and getinfo, and I also
> suspected the issue is in either gbt or the rpc server.
>
> The instance only takes a few hours to get up to that memory usage.
> On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <
> bitcoin-dev@lists•linuxfoundation•org> wrote:
>
>> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail•com>
>> wrote:
>>
>> This is *most likely* the mempool, but is just not reported correctly.
>>
>>
>> I did some testing with PR #6410's better mempool reporting. The improved
>> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
>> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
>> The excess memory usage that I'm seeing is on the order of 100x higher than
>> the mempool "bytes": value. As such, I think it's unlikely that this is the
>> mempool, or at least not normal/correct mempool behavior.
>>
>> Another user (admin@multipool•us) reported 35 GB of RSS usage. I'm
>> guessing his bitcoind has been running longer than any of mine. His server
>> definitely has more RAM. I don't know which email list he is subscribed to
>> (probably XT), so I'm sharing it with both lists to make sure you're all
>> aware of how big an issue this can be.
>>
>> In the meantime you can mitigate the mempool growth by setting
>> `-mintxfee`, see
>>
>> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>>
>>
>> I have mintxfee and minrelaytxfee set to about 0.00003, which is high
>> enough to exclude essentially all of the of the 14700-14800 byte flood
>> transactions. My nodes' mempools only contain about one or two blocks'
>> worth of transactions. So I don't think this is correct either.
>>
>>
>>
>> Some additional notes on this issue:
>>
>> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
>> Core bitcoind process (commit d78a880) overnight with no mining connected
>> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
>> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
>> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
>> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
>> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
>> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
>> getmempoolinfo is still showing "usage" around 25MB or less.
>>
>> I'll do some more testing with this and see if I can make it repeatable,
>> and record the results more carefully. Expect a follow-up from me in a day
>> or two.
>>
>> 2. valgrind did not show anything super promising. It did report this:
>>
>> ==6880== LEAK SUMMARY:
>> ==6880== definitely lost: 0 bytes in 0 blocks
>> ==6880== indirectly lost: 0 bytes in 0 blocks
>> ==6880== possibly lost: 288 bytes in 1 blocks
>> ==6880== still reachable: 10,552 bytes in 39 blocks
>> ==6880== suppressed: 0 bytes in 0 blocks
>> (Bitcoin Core commit d78a880)
>>
>> and this:
>> ==6778== LEAK SUMMARY:
>> ==6778== definitely lost: 0 bytes in 0 blocks
>> ==6778== indirectly lost: 0 bytes in 0 blocks
>> ==6778== possibly lost: 320 bytes in 1 blocks
>> ==6778== still reachable: 10,080 bytes in 32 blocks
>> ==6778== suppressed: 0 bytes in 0 blocks
>> (Bitcoin XT commit fe446d)
>>
>> I haven't found anything in there yet that I think would produce the
>> multi-GB memory usage after running for a few days, but I could be missing
>> it. Email me if you want the full log.
>>
>> I did not try running getblocktemplate while valgrind was running. I'll
>> have to try that. I also have not let valgrind run for more than an hour.
>>
>>
>>
>> P.S.: Sorry for all the cross-post confusion and consequent flamewar
>> fallout. While it's probably too late for this thread, I'll make sure to
>> post in a manner that keeps the threads clearly separate in the future
>> (e.g. different subject lines).
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists•linuxfoundation•org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "bitcoin-xt" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bitcoin-xt+unsubscribe@googlegroups•com.
> For more options, visit https://groups.google.com/d/optout.
>I did that Sunday twice. I'll report the results soon. Short version is that it looks like valgrind is just finding 200 kB to 600 kB of pblocktemplate, which is declared as a static pointer. Not exactly the multi-GB leak I'm looking for, but possibly related.
I've also got two bitcoind processes running on the same machine that I started at the same time, running on different ports, all with the same settings, but one of which is serving getblocktemplate every 5-6 seconds and the other is not, while logging RSS on both every 6 seconds. RSS for the non-serving node is now 734 MB, and for the serving node 1997 MB. Graphs coming soon.
On Oct 20, 2015, at 3:12 AM, Mike Hearn <hearn@vinumeris•com> wrote:
Quoted text (66 lines)
> OK, then running under Valgrind whilst sending gbt RPCs would be the next step.
>
> On Mon, Oct 19, 2015 at 9:17 PM, Multipool Admin <admin@multipool•us> wrote:
> My nodes are continuously running getblocktemplate and getinfo, and I also suspected the issue is in either gbt or the rpc server.
>
> The instance only takes a few hours to get up to that memory usage.
>
> On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <bitcoin-dev@lists•linuxfoundation•org> wrote:
> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail•com> wrote:
>> This is *most likely* the mempool, but is just not reported correctly.
>
> I did some testing with PR #6410's better mempool reporting. The improved reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is about 2.5x to 3x higher than the serialized transaction sizes ("bytes":). The excess memory usage that I'm seeing is on the order of 100x higher than the mempool "bytes": value. As such, I think it's unlikely that this is the mempool, or at least not normal/correct mempool behavior.
>
> Another user (admin@multipool•us) reported 35 GB of RSS usage. I'm guessing his bitcoind has been running longer than any of mine. His server definitely has more RAM. I don't know which email list he is subscribed to (probably XT), so I'm sharing it with both lists to make sure you're all aware of how big an issue this can be.
>
>> In the meantime you can mitigate the mempool growth by setting `-mintxfee`, see
>> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>
> I have mintxfee and minrelaytxfee set to about 0.00003, which is high enough to exclude essentially all of the of the 14700-14800 byte flood transactions. My nodes' mempools only contain about one or two blocks' worth of transactions. So I don't think this is correct either.
>
>
>
> Some additional notes on this issue:
>
> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a Core bitcoind process (commit d78a880) overnight with no mining connected to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes, and RSS climbed to 1180 MB. An hour after that with more getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.] getmempoolinfo is still showing "usage" around 25MB or less.
>
> I'll do some more testing with this and see if I can make it repeatable, and record the results more carefully. Expect a follow-up from me in a day or two.
>
> 2. valgrind did not show anything super promising. It did report this:
>
> ==6880== LEAK SUMMARY:
> ==6880== definitely lost: 0 bytes in 0 blocks
> ==6880== indirectly lost: 0 bytes in 0 blocks
> ==6880== possibly lost: 288 bytes in 1 blocks
> ==6880== still reachable: 10,552 bytes in 39 blocks
> ==6880== suppressed: 0 bytes in 0 blocks
> (Bitcoin Core commit d78a880)
>
> and this:
> ==6778== LEAK SUMMARY:
> ==6778== definitely lost: 0 bytes in 0 blocks
> ==6778== indirectly lost: 0 bytes in 0 blocks
> ==6778== possibly lost: 320 bytes in 1 blocks
> ==6778== still reachable: 10,080 bytes in 32 blocks
> ==6778== suppressed: 0 bytes in 0 blocks
> (Bitcoin XT commit fe446d)
>
> I haven't found anything in there yet that I think would produce the multi-GB memory usage after running for a few days, but I could be missing it. Email me if you want the full log.
>
> I did not try running getblocktemplate while valgrind was running. I'll have to try that. I also have not let valgrind run for more than an hour.
>
>
>
> P.S.: Sorry for all the cross-post confusion and consequent flamewar fallout. While it's probably too late for this thread, I'll make sure to post in a manner that keeps the threads clearly separate in the future (e.g. different subject lines).
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation•org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
> --
> You received this message because you are subscribed to the Google Groups "bitcoin-xt" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bitcoin-xt+unsubscribe@googlegroups•com.
> For more options, visit https://groups.google.com/d/optout.
>More notes:
1. I ran a side-by-side comparison with two bitcoind processes (Core, same recent git commit as before) on the same computer with the same settings running on different ports. With both processes, I logged RSS (via /proc/$pid/status) every 6 seconds. With one of those processes, I also ran bitcoin-cli getblocktemplate > /dev/null every 6 seconds. I let that run for about 30 hours. A graph and links to the CSVs of raw data are below. Results seem pretty clear: the getblocktemplate RPC is implicated in this issue.
http://toom.im/files/memlog8518.csv http://toom.im/files/memlog-nogbt-8503.csv http://toom.im/files/bitcoind_memory_usage_gbt.png
2. I ran valgrind twice, for about 6 hours each, on bitcoind while hitting it with getblocktemplate every 6 hours. Full valgrind output can be found at these two URLs:
http://toom.im/files/valgrind-gbt-1.log http://toom.im/files/valgrind-gbt-2.log
The summary:
==4064== LEAK SUMMARY: ==4064== definitely lost: 0 bytes in 0 blocks ==4064== indirectly lost: 0 bytes in 0 blocks ==4064== possibly lost: 288 bytes in 1 blocks ==4064== still reachable: 527,594 bytes in 4,367 blocks ==4064== suppressed: 0 bytes in 0 blocks The main components of that still reachable section seem to just be one output of CreateNewBlock that's cached in case another getblocktemplate request is received before any new transactions come in:
==4064== 98,304 bytes in 1 blocks are still reachable in loss record 39 of 40 ==4064== at 0x4C29180: operator new(unsigned long) (vg_replace_malloc.c:324) ==4064== by 0x28EAA1: __gnu_cxx::new_allocator<CTransaction>::allocate(unsigned long, void const*) (new_allocator.h:104) ==4064== by 0x27EE44: __gnu_cxx::__alloc_traits<std::allocator<CTransaction> >::allocate(std::allocator<CTransaction>&, unsigned long) (alloc_traits.h:182) ==4064== by 0x26DFB0: std::_Vector_base<CTransaction, std::allocator<CTransaction> >::_M_allocate(unsigned long) (stl_vector.h:170) ==4064== by 0x2D5BDE: std::vector<CTransaction, std::allocator<CTransaction> >::_M_insert_aux(__gnu_cxx::__normal_iterator<CTransaction*, std::vector<CTransaction, std::allocator<CTransaction> > >, CTransaction const&) (vector.tcc:353) ==4064== by 0x2D3FF8: std::vector<CTransaction, std::allocator<CTransaction> >::push_back(CTransaction const&) (stl_vector.h:925) ==4064== by 0x2D113E: CreateNewBlock(CScript const&) (miner.cpp:298) ==4064== by 0x442D78: getblocktemplate(UniValue const&, bool) (rpcmining.cpp:513) ==4064== by 0x390CEB: CRPCTable::execute(std::string const&, UniValue const&) const (rpcserver.cpp:526) ==4064== by 0x41C5AB: HTTPReq_JSONRPC(HTTPRequest*, std::string const&) (httprpc.cpp:125) ==4064== by 0x3559BD: boost::detail::function::void_function_invoker2<bool (*)(HTTPRequest*, std::string const&), void, HTTPRequest*, std::string const&>::invoke(boost::detail::function::function_buffer&, HTTPRequest*, std::string const&) (function_template.hpp:112) ==4064== by 0x422520: boost::function2<void, HTTPRequest*, std::string const&>::operator()(HTTPRequest*, std::string const&) const (function_template.hpp:767)
There are a few other similar loss records (mostly referring to pblock or pblocktemplate in CreateNewBlock(...), but I see nothing that can explain the multi-GB memory consumption.
3. One user on the bitcointalk p2pool thread (https://bitcointalk.org/index.php?topic=18313.msg12733791#msg12733791) claimed that he had this memory usage issue on Linux, but not on Mac OS X, under a GBT workload in both situations. If this is true, that would suggest this might be a fragmentation issue due to poor memory allocation. The other likely hypothesis is bloated caches. Looking into those two possibilities will be my next steps.
On Oct 20, 2015, at 5:39 AM, Jonathan Toomim <j@toom•im> wrote:
Quoted text (74 lines)
> I did that Sunday twice. I'll report the results soon. Short version is that it looks like valgrind is just finding 200 kB to 600 kB of pblocktemplate, which is declared as a static pointer. Not exactly the multi-GB leak I'm looking for, but possibly related.
>
> I've also got two bitcoind processes running on the same machine that I started at the same time, running on different ports, all with the same settings, but one of which is serving getblocktemplate every 5-6 seconds and the other is not, while logging RSS on both every 6 seconds. RSS for the non-serving node is now 734 MB, and for the serving node 1997 MB. Graphs coming soon.
>
>
> On Oct 20, 2015, at 3:12 AM, Mike Hearn <hearn@vinumeris•com> wrote:
>
>> OK, then running under Valgrind whilst sending gbt RPCs would be the next step.
>>
>> On Mon, Oct 19, 2015 at 9:17 PM, Multipool Admin <admin@multipool•us> wrote:
>> My nodes are continuously running getblocktemplate and getinfo, and I also suspected the issue is in either gbt or the rpc server.
>>
>> The instance only takes a few hours to get up to that memory usage.
>>
>> On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <bitcoin-dev@lists•linuxfoundation•org> wrote:
>> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail•com> wrote:
>>> This is *most likely* the mempool, but is just not reported correctly.
>>
>> I did some testing with PR #6410's better mempool reporting. The improved reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is about 2.5x to 3x higher than the serialized transaction sizes ("bytes":). The excess memory usage that I'm seeing is on the order of 100x higher than the mempool "bytes": value. As such, I think it's unlikely that this is the mempool, or at least not normal/correct mempool behavior.
>>
>> Another user (admin@multipool•us) reported 35 GB of RSS usage. I'm guessing his bitcoind has been running longer than any of mine. His server definitely has more RAM. I don't know which email list he is subscribed to (probably XT), so I'm sharing it with both lists to make sure you're all aware of how big an issue this can be.
>>
>>> In the meantime you can mitigate the mempool growth by setting `-mintxfee`, see
>>> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>>
>> I have mintxfee and minrelaytxfee set to about 0.00003, which is high enough to exclude essentially all of the of the 14700-14800 byte flood transactions. My nodes' mempools only contain about one or two blocks' worth of transactions. So I don't think this is correct either.
>>
>>
>>
>> Some additional notes on this issue:
>>
>> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a Core bitcoind process (commit d78a880) overnight with no mining connected to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes, and RSS climbed to 1180 MB. An hour after that with more getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.] getmempoolinfo is still showing "usage" around 25MB or less.
>>
>> I'll do some more testing with this and see if I can make it repeatable, and record the results more carefully. Expect a follow-up from me in a day or two.
>>
>> 2. valgrind did not show anything super promising. It did report this:
>>
>> ==6880== LEAK SUMMARY:
>> ==6880== definitely lost: 0 bytes in 0 blocks
>> ==6880== indirectly lost: 0 bytes in 0 blocks
>> ==6880== possibly lost: 288 bytes in 1 blocks
>> ==6880== still reachable: 10,552 bytes in 39 blocks
>> ==6880== suppressed: 0 bytes in 0 blocks
>> (Bitcoin Core commit d78a880)
>>
>> and this:
>> ==6778== LEAK SUMMARY:
>> ==6778== definitely lost: 0 bytes in 0 blocks
>> ==6778== indirectly lost: 0 bytes in 0 blocks
>> ==6778== possibly lost: 320 bytes in 1 blocks
>> ==6778== still reachable: 10,080 bytes in 32 blocks
>> ==6778== suppressed: 0 bytes in 0 blocks
>> (Bitcoin XT commit fe446d)
>>
>> I haven't found anything in there yet that I think would produce the multi-GB memory usage after running for a few days, but I could be missing it. Email me if you want the full log.
>>
>> I did not try running getblocktemplate while valgrind was running. I'll have to try that. I also have not let valgrind run for more than an hour.
>>
>>
>>
>> P.S.: Sorry for all the cross-post confusion and consequent flamewar fallout. While it's probably too late for this thread, I'll make sure to post in a manner that keeps the threads clearly separate in the future (e.g. different subject lines).
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists•linuxfoundation•org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "bitcoin-xt" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to bitcoin-xt+unsubscribe@googlegroups•com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>On Tuesday 20 Oct 2015 20:01:16 Jonathan Toomim wrote:
Quoted text (3 lines)
> claimed that he had this memory usage issue on Linux, but not on Mac OS X, > under a GBT workload in both situations. If this is true, that would > suggest this might be a fragmentation issue due to poor memory allocation.
Please make sure you measure your memory usage correctly on Linux, it is notoriously easy to get misleading info from tools like top.
I use this one on Linux.
$cat ~/bin/showmemusage
#!/bin/sh
if test -z "$1"; then
echo "need a pid"
exit
fi
mem=`echo 0 $(cat /proc/$1/smaps | grep Pss | awk '{print $2}' | \
sed 's#^#+#' ) | bc`
echo "$mem KB"On Wed, Oct 21, 2015 at 2:01 PM, Jonathan Toomim <j@toom•im> wrote:
Quoted text (6 lines)
> 3. One user on the bitcointalk p2pool thread ( > https://bitcointalk.org/index.php?topic=18313.msg12733791#msg12733791) > claimed that he had this memory usage issue on Linux, but not on Mac OS X, > under a GBT workload in both situations. > >
Thanks for the thorough investigation. FYI, I believe I've experienced this issue on both Linux & Mac OSX. I've moved my P2Pool/bitcoind node (rossbennetts.com:9332) from Ubuntu to OSX since my P2Pool instability began and they both seem to become flaky after a few hours of uptime (since ~v0.10.something). I've tried both core and xt. Two other bitcoin nodes, one connected to electrum-server and one standalone, seem fine.
//long term lurker, first time poster... Ross...
The method I was using was essentially
grep VmRSS /proc/$pid/status
Comparing these two methods, I get
Your method (PSS): 2408313 My method (RSS): VmRSS: 2410396 kB
On Oct 21, 2015, at 12:29 AM, Tom Zander <tomz@freedommail•ch> wrote:
Quoted text (16 lines)
> On Tuesday 20 Oct 2015 20:01:16 Jonathan Toomim wrote:
> Please make sure you measure your memory usage correctly on Linux, it is
> notoriously easy to get misleading info from tools like top.
>
> I use this one on Linux.
>
> $cat ~/bin/showmemusage
> #!/bin/sh
> if test -z "$1"; then
> echo "need a pid"
> exit
> fi
>
> mem=`echo 0 $(cat /proc/$1/smaps | grep Pss | awk '{print $2}' | \
> sed 's#^#+#' ) | bc`
> echo "$mem KB"This is a real issue. The bitcoind process is getting killed every few days when it reaches around 55gb of usage on my server. On Oct 21, 2015 12:29 AM, "Tom Zander via bitcoin-dev" < bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (29 lines)
> On Tuesday 20 Oct 2015 20:01:16 Jonathan Toomim wrote:
> > claimed that he had this memory usage issue on Linux, but not on Mac OS
> X,
> > under a GBT workload in both situations. If this is true, that would
> > suggest this might be a fragmentation issue due to poor memory
> allocation.
>
> Please make sure you measure your memory usage correctly on Linux, it is
> notoriously easy to get misleading info from tools like top.
>
> I use this one on Linux.
>
> $cat ~/bin/showmemusage
> #!/bin/sh
> if test -z "$1"; then
> echo "need a pid"
> exit
> fi
>
> mem=`echo 0 $(cat /proc/$1/smaps | grep Pss | awk '{print $2}' | \
> sed 's#^#+#' ) | bc`
> echo "$mem KB"
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation•org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>You may want to add a cron job to restart bitcoind every day or two as a damage control mechanism until we figure this out.
On Oct 22, 2015, at 9:06 AM, Multipool Admin <admin@multipool•us> wrote:
Quoted text (34 lines)
> This is a real issue. The bitcoind process is getting killed every few days when it reaches around 55gb of usage on my server.
>
> On Oct 21, 2015 12:29 AM, "Tom Zander via bitcoin-dev" <bitcoin-dev@lists•linuxfoundation•org> wrote:
> On Tuesday 20 Oct 2015 20:01:16 Jonathan Toomim wrote:
> > claimed that he had this memory usage issue on Linux, but not on Mac OS X,
> > under a GBT workload in both situations. If this is true, that would
> > suggest this might be a fragmentation issue due to poor memory allocation.
>
> Please make sure you measure your memory usage correctly on Linux, it is
> notoriously easy to get misleading info from tools like top.
>
> I use this one on Linux.
>
> $cat ~/bin/showmemusage
> #!/bin/sh
> if test -z "$1"; then
> echo "need a pid"
> exit
> fi
>
> mem=`echo 0 $(cat /proc/$1/smaps | grep Pss | awk '{print $2}' | \
> sed 's#^#+#' ) | bc`
> echo "$mem KB"
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation•org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
> --
> You received this message because you are subscribed to the Google Groups "bitcoin-xt" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bitcoin-xt+unsubscribe@googlegroups•com.
> For more options, visit https://groups.google.com/d/optout.I think this thread has gotten to the stage where it should be moved to an issue on Github and not continue to CC the bitcoin-dev list (or any other list tbh).
On Thu, Oct 22, 2015 at 5:27 PM, Jonathan Toomim via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (54 lines)
> You may want to add a cron job to restart bitcoind every day or two as a
> damage control mechanism until we figure this out.
>
> On Oct 22, 2015, at 9:06 AM, Multipool Admin <admin@multipool•us> wrote:
>
> This is a real issue. The bitcoind process is getting killed every few days
> when it reaches around 55gb of usage on my server.
>
> On Oct 21, 2015 12:29 AM, "Tom Zander via bitcoin-dev"
> <bitcoin-dev@lists•linuxfoundation•org> wrote:
>>
>> On Tuesday 20 Oct 2015 20:01:16 Jonathan Toomim wrote:
>> > claimed that he had this memory usage issue on Linux, but not on Mac OS
>> > X,
>> > under a GBT workload in both situations. If this is true, that would
>> > suggest this might be a fragmentation issue due to poor memory
>> > allocation.
>>
>> Please make sure you measure your memory usage correctly on Linux, it is
>> notoriously easy to get misleading info from tools like top.
>>
>> I use this one on Linux.
>>
>> $cat ~/bin/showmemusage
>> #!/bin/sh
>> if test -z "$1"; then
>> echo "need a pid"
>> exit
>> fi
>>
>> mem=`echo 0 $(cat /proc/$1/smaps | grep Pss | awk '{print $2}' | \
>> sed 's#^#+#' ) | bc`
>> echo "$mem KB"
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists•linuxfoundation•org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "bitcoin-xt" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bitcoin-xt+unsubscribe@googlegroups•com.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation•org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>Btc Drak via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> writes:
Quoted text (3 lines)
> I think this thread has gotten to the stage where it should be moved > to an issue on Github and not continue to CC the bitcoin-dev list (or > any other list tbh).
Agreed. I couldn't see an issue, so I've opened one. Let's track this there, please.
https://github.com/bitcoin/bitcoin/issues/6876
Cheers, Rusty.