Bitcoin Core trial balloon: splitting blockchain engine and wallet
13 messages, 21 to 24 Feb 2014. Started by Jeff Garzik.
Jeff Garzik (3), Wladimir J. van der Laan (3), Mike Hearn (2), Dustin D. Trammell (2), William Yager, Peter Todd, James Hartig
[Meta: "Bitcoin Core" is the newfangled branding of bitcoind / Bitcoin-Qt reference implementation, in case you wondering.]
Several sites, including BitPay, use bitcoind outside the standard role of wallet software. bitcoind can be used purely for payment network access and management. I call this the "border router" role. Upcoming version 0.9 will feature the ability to disable the bitcoind wallet at compile time or runtime. This permits a more optimized border router profile, reducing process size by 40-200MB according to some reports.
Recent IRC discussion have floated a rough proposal for a wallet next-step: Running the Bitcoin Core wallet as a separate process, a separate binary, from the blockchain engine. The wallet process would communicate with the blockchain engine using existing RPC and P2P channels, becoming a real SPV client. This accomplishes a longstanding security goal of sandboxing away wallet keys and sensitive data from the network-exposed P2P engine, in a separate process, among other benefits.
Simple forking was explored a bit. I did some hacking in that direction, as it seemed potentially lightweight and somewhat easy to me: https://github.com/jgarzik/bitcoin/tree/fork fork+pipe is fine for Linux and OSX/BSD. However, Windows requires an exec-like solution to create a new process. MSDN does give us a Unix-pipe-like solution: http://msdn.microsoft.com/en-us/library/edze9h7e%28v=vs.80%29.aspx Others pointed to boost interprocess communication APIs, which come with their own set of caveats. Such a solution would involve a brand new IPC protocol, and lots of brand new glue code.
Separate programs seems better. Windows forces us to achieve process separation via exec-like method. We already have IPC: RPC + P2P. Modern OS's make localhost sockets just about as fast as other IPCs methods. Linux, at least, employs zero-copy for localhost sockets in many situations, similar to the kernel's pipe tricks.
Pieter has been working on headers-first sync: https://github.com/bitcoin/bitcoin/pull/2964 Moving along this wallet/blockchain engine split requires upping the review&test bandwidth on Pieter's PRs, such as https://github.com/bitcoin/bitcoin/pull/3514
Unsure how much of the separate-binary discussion Gavin saw, so cc'd for emphasis.
Bear in mind a separate process doesn't buy you anything without a sandbox, and those are expensive (in terms of complexity). On 21 Feb 2014 11:40, "Jeff Garzik" <jgarzik@bitpay•com> wrote:
Quoted text (63 lines)
> [Meta: "Bitcoin Core" is the newfangled branding of bitcoind / > Bitcoin-Qt reference implementation, in case you wondering.] > > Several sites, including BitPay, use bitcoind outside the standard > role of wallet software. bitcoind can be used purely for payment > network access and management. I call this the "border router" role. > Upcoming version 0.9 will feature the ability to disable the bitcoind > wallet at compile time or runtime. This permits a more optimized > border router profile, reducing process size by 40-200MB according to > some reports. > > Recent IRC discussion have floated a rough proposal for a wallet > next-step: Running the Bitcoin Core wallet as a separate process, a > separate binary, from the blockchain engine. The wallet process would > communicate with the blockchain engine using existing RPC and P2P > channels, becoming a real SPV client. This accomplishes a > longstanding security goal of sandboxing away wallet keys and > sensitive data from the network-exposed P2P engine, in a separate > process, among other benefits. > > Simple forking was explored a bit. I did some hacking in that > direction, as it seemed potentially lightweight and somewhat easy to > me: https://github.com/jgarzik/bitcoin/tree/fork fork+pipe is fine > for Linux and OSX/BSD. However, Windows requires an exec-like > solution to create a new process. MSDN does give us a Unix-pipe-like > solution: > http://msdn.microsoft.com/en-us/library/edze9h7e%28v=vs.80%29.aspx > Others pointed to boost interprocess communication APIs, which come > with their own set of caveats. Such a solution would involve a brand > new IPC protocol, and lots of brand new glue code. > > Separate programs seems better. Windows forces us to achieve process > separation via exec-like method. We already have IPC: RPC + P2P. > Modern OS's make localhost sockets just about as fast as other IPCs > methods. Linux, at least, employs zero-copy for localhost sockets in > many situations, similar to the kernel's pipe tricks. > > Pieter has been working on headers-first sync: > https://github.com/bitcoin/bitcoin/pull/2964 Moving along this > wallet/blockchain engine split requires upping the review&test > bandwidth on Pieter's PRs, such as > https://github.com/bitcoin/bitcoin/pull/3514 > > Unsure how much of the separate-binary discussion Gavin saw, so cc'd > for emphasis. > > -- > Jeff Garzik > Bitcoin core developer and open source evangelist > BitPay, Inc. https://bitpay.com/ > > > ------------------------------------------------------------------------------ > Managing the Performance of Cloud-Based Applications > Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. > Read the Whitepaper. > > http://pubads.g.doubleclick.net/gampad/clk?id=121054471&iu=/4140/ostg.clktrk > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development >
On Fri, Feb 21, 2014 at 7:27 AM, Mike Hearn <mike@plan99•net> wrote:
Quoted text (3 lines)
> Bear in mind a separate process doesn't buy you anything without a > sandbox, and those are expensive (in terms of complexity). >
Sandboxing in user space is complex, agreed,
The most straightforward way would be to run the blockchain daemon as a system service (with its own uid/gid and set of Apparmor/SELinux restrictions) and the wallet daemon as the user.
This would also allow sharing one blockchain daemon between multiple users and wallet processes (not necessarily on the same machine), something I've wanted to be able to do for a long time.
Wladimir
Running the network part of the core as a system service might make sense for server implementations, but it’s a pain in the rear for most users.
That said, I think segregating the two processes is a great idea. Let’s just try to avoid some complicated scheme that involves necessarily running things under multiple users.
Will
On Feb 21, 2014, at 0:43, Wladimir <laanwj@gmail•com> wrote:
Quoted text (1 line)
> The most straightforward way would be to run the blockchain daemon as a system service (with its own uid/gid and set of Apparmor/SELinux restrictions) and the wallet daemon as the user.
RE "doesn't buy you anything" Today, when unlocked, plaintext private keys reside in the same address space as the blockchain engine (BCE). Process separation increases the difficulty of accessing key data from the BCE, even presuming a normal, no-chroot, same-uid, parent-child process relationship. The attack surface is clearly changed from "one buffer overflow can touch this data."
Regardless, the split makes sense given existing modularity and coding directions. I wouldn't micro-focus on the "sandbox" word.
On Fri, Feb 21, 2014 at 1:27 AM, Mike Hearn <mike@plan99•net> wrote:
Quoted text (67 lines)
> Bear in mind a separate process doesn't buy you anything without a sandbox, > and those are expensive (in terms of complexity). > > On 21 Feb 2014 11:40, "Jeff Garzik" <jgarzik@bitpay•com> wrote: >> >> [Meta: "Bitcoin Core" is the newfangled branding of bitcoind / >> Bitcoin-Qt reference implementation, in case you wondering.] >> >> Several sites, including BitPay, use bitcoind outside the standard >> role of wallet software. bitcoind can be used purely for payment >> network access and management. I call this the "border router" role. >> Upcoming version 0.9 will feature the ability to disable the bitcoind >> wallet at compile time or runtime. This permits a more optimized >> border router profile, reducing process size by 40-200MB according to >> some reports. >> >> Recent IRC discussion have floated a rough proposal for a wallet >> next-step: Running the Bitcoin Core wallet as a separate process, a >> separate binary, from the blockchain engine. The wallet process would >> communicate with the blockchain engine using existing RPC and P2P >> channels, becoming a real SPV client. This accomplishes a >> longstanding security goal of sandboxing away wallet keys and >> sensitive data from the network-exposed P2P engine, in a separate >> process, among other benefits. >> >> Simple forking was explored a bit. I did some hacking in that >> direction, as it seemed potentially lightweight and somewhat easy to >> me: https://github.com/jgarzik/bitcoin/tree/fork fork+pipe is fine >> for Linux and OSX/BSD. However, Windows requires an exec-like >> solution to create a new process. MSDN does give us a Unix-pipe-like >> solution: >> http://msdn.microsoft.com/en-us/library/edze9h7e%28v=vs.80%29.aspx >> Others pointed to boost interprocess communication APIs, which come >> with their own set of caveats. Such a solution would involve a brand >> new IPC protocol, and lots of brand new glue code. >> >> Separate programs seems better. Windows forces us to achieve process >> separation via exec-like method. We already have IPC: RPC + P2P. >> Modern OS's make localhost sockets just about as fast as other IPCs >> methods. Linux, at least, employs zero-copy for localhost sockets in >> many situations, similar to the kernel's pipe tricks. >> >> Pieter has been working on headers-first sync: >> https://github.com/bitcoin/bitcoin/pull/2964 Moving along this >> wallet/blockchain engine split requires upping the review&test >> bandwidth on Pieter's PRs, such as >> https://github.com/bitcoin/bitcoin/pull/3514 >> >> Unsure how much of the separate-binary discussion Gavin saw, so cc'd >> for emphasis. >> >> -- >> Jeff Garzik >> Bitcoin core developer and open source evangelist >> BitPay, Inc. https://bitpay.com/ >> >> >> ------------------------------------------------------------------------------ >> Managing the Performance of Cloud-Based Applications >> Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. >> Read the Whitepaper. >> >> http://pubads.g.doubleclick.net/gampad/clk?id=121054471&iu=/4140/ostg.clktrk >> _______________________________________________ >> Bitcoin-development mailing list >> Bitcoin-development@lists•sourceforge•net >> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
On Fri, Feb 21, 2014 at 7:50 AM, William Yager <will.yager@gmail•com> wrote:
Quoted text (3 lines)
> Running the network part of the core as a system service might make sense > for server implementations, but it’s a pain in the rear for most users. >
Come on, making it a possibility doesn't affect other kinds of use cases in any way. Are you just arguing for the sake of arguing?
Wladimir
I'm not sure it does really - typical C/C++ exploits let you run arbitrary code, at which point you can quite easily ptrace the other process and do whatever you want with it, or read /proc/pid/mem etc. But process separation is certainly a prerequisite for sandboxing so I'm not arguing against such a change, just pointing out that it requires some work to really get the benefits. Also an SPV Bitcoin Core would obviously be of tremendous utility all by itself ...
On Fri, Feb 21, 2014 at 12:20 PM, Jeff Garzik <jgarzik@bitpay•com> wrote:
Quoted text (89 lines)
> RE "doesn't buy you anything" Today, when unlocked, plaintext > private keys reside in the same address space as the blockchain engine > (BCE). Process separation increases the difficulty of accessing key > data from the BCE, even presuming a normal, no-chroot, same-uid, > parent-child process relationship. The attack surface is clearly > changed from "one buffer overflow can touch this data." > > Regardless, the split makes sense given existing modularity and coding > directions. I wouldn't micro-focus on the "sandbox" word. > > On Fri, Feb 21, 2014 at 1:27 AM, Mike Hearn <mike@plan99•net> wrote: > > Bear in mind a separate process doesn't buy you anything without a > sandbox, > > and those are expensive (in terms of complexity). > > > > On 21 Feb 2014 11:40, "Jeff Garzik" <jgarzik@bitpay•com> wrote: > >> > >> [Meta: "Bitcoin Core" is the newfangled branding of bitcoind / > >> Bitcoin-Qt reference implementation, in case you wondering.] > >> > >> Several sites, including BitPay, use bitcoind outside the standard > >> role of wallet software. bitcoind can be used purely for payment > >> network access and management. I call this the "border router" role. > >> Upcoming version 0.9 will feature the ability to disable the bitcoind > >> wallet at compile time or runtime. This permits a more optimized > >> border router profile, reducing process size by 40-200MB according to > >> some reports. > >> > >> Recent IRC discussion have floated a rough proposal for a wallet > >> next-step: Running the Bitcoin Core wallet as a separate process, a > >> separate binary, from the blockchain engine. The wallet process would > >> communicate with the blockchain engine using existing RPC and P2P > >> channels, becoming a real SPV client. This accomplishes a > >> longstanding security goal of sandboxing away wallet keys and > >> sensitive data from the network-exposed P2P engine, in a separate > >> process, among other benefits. > >> > >> Simple forking was explored a bit. I did some hacking in that > >> direction, as it seemed potentially lightweight and somewhat easy to > >> me: https://github.com/jgarzik/bitcoin/tree/fork fork+pipe is fine > >> for Linux and OSX/BSD. However, Windows requires an exec-like > >> solution to create a new process. MSDN does give us a Unix-pipe-like > >> solution: > >> http://msdn.microsoft.com/en-us/library/edze9h7e%28v=vs.80%29.aspx > >> Others pointed to boost interprocess communication APIs, which come > >> with their own set of caveats. Such a solution would involve a brand > >> new IPC protocol, and lots of brand new glue code. > >> > >> Separate programs seems better. Windows forces us to achieve process > >> separation via exec-like method. We already have IPC: RPC + P2P. > >> Modern OS's make localhost sockets just about as fast as other IPCs > >> methods. Linux, at least, employs zero-copy for localhost sockets in > >> many situations, similar to the kernel's pipe tricks. > >> > >> Pieter has been working on headers-first sync: > >> https://github.com/bitcoin/bitcoin/pull/2964 Moving along this > >> wallet/blockchain engine split requires upping the review&test > >> bandwidth on Pieter's PRs, such as > >> https://github.com/bitcoin/bitcoin/pull/3514 > >> > >> Unsure how much of the separate-binary discussion Gavin saw, so cc'd > >> for emphasis. > >> > >> -- > >> Jeff Garzik > >> Bitcoin core developer and open source evangelist > >> BitPay, Inc. https://bitpay.com/ > >> > >> > >> > ------------------------------------------------------------------------------ > >> Managing the Performance of Cloud-Based Applications > >> Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. > >> Read the Whitepaper. > >> > >> > http://pubads.g.doubleclick.net/gampad/clk?id=121054471&iu=/4140/ostg.clktrk > >> _______________________________________________ > >> Bitcoin-development mailing list > >> Bitcoin-development@lists•sourceforge•net > >> https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > > > -- > Jeff Garzik > Bitcoin core developer and open source evangelist > BitPay, Inc. https://bitpay.com/ >
On Fri, Feb 21, 2014 at 04:11:06PM +0530, Mike Hearn wrote:
Quoted text (19 lines)
> On Fri, Feb 21, 2014 at 12:20 PM, Jeff Garzik <jgarzik@bitpay•com> wrote: > > > RE "doesn't buy you anything" Today, when unlocked, plaintext > > private keys reside in the same address space as the blockchain engine > > (BCE). Process separation increases the difficulty of accessing key > > data from the BCE, even presuming a normal, no-chroot, same-uid, > > parent-child process relationship. The attack surface is clearly > > changed from "one buffer overflow can touch this data." > > > > Regardless, the split makes sense given existing modularity and coding > > directions. I wouldn't micro-focus on the "sandbox" word. > > I'm not sure it does really - typical C/C++ exploits let you run arbitrary > code, at which point you can quite easily ptrace the other process and do > whatever you want with it, or read /proc/pid/mem etc. But process > separation is certainly a prerequisite for sandboxing so I'm not arguing > against such a change, just pointing out that it requires some work to > really get the benefits. Also an SPV Bitcoin Core would obviously be of > tremendous utility all by itself ...
The seccomp mechanism would work well here - it's a syscall whitelister, which makes ptrace useless, among other things. Used by Chrome as of v23 to sandbox the renderers.
We'd probably need to use it with chroot and whitelist the open() call so that the existing code can create new blockfiles and do whatever leveldb does.
On Fri, 2014-02-21 at 01:09 -0500, Jeff Garzik wrote:
Quoted text (8 lines)
> Recent IRC discussion have floated a rough proposal for a wallet > next-step: Running the Bitcoin Core wallet as a separate process, a > separate binary, from the blockchain engine. The wallet process would > communicate with the blockchain engine using existing RPC and P2P > channels, becoming a real SPV client. This accomplishes a > longstanding security goal of sandboxing away wallet keys and > sensitive data from the network-exposed P2P engine, in a separate > process, among other benefits.
PLEASE.
For those of us that prefer the reference software and also manage multiple wallets, having to store a copy of the blockchain for each one eats up disk space quite quickly. If I could run a local blockchain server (or a local network one, even) and then have whichever wallet I start up use that instead of maintain its own copy of the blockchain, my world would be much, much happier.
Sandboxing keys and sensitive wallet data away from the attack surface introduced by the network interfaces into another separate process is also a good security move. Don't forget to sanitize your IPC inputs (:
Thanks,
On Fri, 2014-02-21 at 07:43 +0100, Wladimir wrote:
Quoted text (3 lines)
> The most straightforward way would be to run the blockchain daemon as > a system service (with its own uid/gid and set of Apparmor/SELinux > restrictions) and the wallet daemon as the user.
This assumes you as a user have the rights to do so. This would be preferred, but in some cases may not be possible. Perhaps it should be optional?
Quoted text (3 lines)
> This would also allow sharing one blockchain daemon between multiple > users and wallet processes (not necessarily on the same machine), > something I've wanted to be able to do for a long time.
Agreed (:
On Fri, Feb 21, 2014 at 8:04 PM, Dustin D. Trammell <dtrammell@dustintrammell•com> wrote:
Quoted text (6 lines)
> For those of us that prefer the reference software and also manage > multiple wallets, having to store a copy of the blockchain for each one > eats up disk space quite quickly. If I could run a local blockchain > server (or a local network one, even) and then have whichever wallet I > start up use that instead of maintain its own copy of the blockchain, my > world would be much, much happier.
You mean running multiple wallets simultaneously? Agreed.
Multiple wallets, used serially, works fine today. I manage multiple wallets using symlink replacement.
On Sat, Feb 22, 2014 at 2:09 AM, Dustin D. Trammell < dtrammell@dustintrammell•com> wrote:
Quoted text (9 lines)
> On Fri, 2014-02-21 at 07:43 +0100, Wladimir wrote: > > The most straightforward way would be to run the blockchain daemon as > > a system service (with its own uid/gid and set of Apparmor/SELinux > > restrictions) and the wallet daemon as the user. > > This assumes you as a user have the rights to do so. This would be > preferred, but in some cases may not be possible. Perhaps it should be > optional? >
No! I'm proposing that we force everyone to do it. Using all means necessary. There should be regular audits that everyone is running the software exactly in my configuration, and if not, a special task force will take care that spankings are carried out on the spot.
Repeated offenders will lose their BitLicense. </s>
Please stop kicking this dead horse. It was just a random idea. Maybe a way how Linux distributions could structure it, but it may or may not apply in your case. And that's fine, this is free software development, you can do whatever you want!
Let's try to bring this discussion back to its original intention: for anyone that wants to concretely help this along, please help reviewing and testing the pull requests that jgarzik mentions.
Wladimir BTW: All of those patches are helpful for monolithic-bitcoind as well as they (lay the groundwork for) speeding up block synchronization.
Setting aside all security benefits (which the user can obviously choose to implement or ignore), a major benefit here is being able to have multiple wallets use the same blockchain process. I have 3 different bitcoind processes running on the same server to utilize multiple wallets. Using them serially isn't an option in my case. Also, peers can run the cheaper process instead of having the wallet functionality which isn't even used.
On the security front, this doesn't seem to be any less secure and it gives the user the flexibility to make it as secure as they feel comfortable. If they want to run them both as the same user with no SELinux or file protections (this isn't stopping or encouraging that) they're already doing that now with bitcoind, albeit with possibly a larger attack surface.
Thanks,