Faster databases than LevelDB
6 messages, 17 Sep 2013. Started by Mike Hearn.
Mike Hearn (2), Jorge Timón (2), Greg Maxwell, Mark Friedenbach
LevelDB is fast - very fast if you give it enough CPU time and disk seeks. But it's not the last word in performance.
HyperLevelDB is a forked LevelDB with some changes, mostly, finer grained locking and changes to how compaction works:
http://hyperdex.org/performance/leveldb/
However, it comes with a caveat - one of the changes they made is to take away write throttling if compaction falls behind, the app itself is expected to do that.
Sophia is a competitor to LevelDB. The website claims that in benchmarks it completely smokes LevelDB. I have not explored how it does this or tried to replicate their benchmarks myself:
http://sphia.org/index.html http://sphia.org/benchmarks.html
It's written in C and BSD licensed.
As an example of the kind of speedup they claim to be capable of, they say LevelDB could do 167,476 random reads per second on their SSD based machine. Sophia could do 438,084 reads/sec. Random reads are of course the most interesting for us because that's what UTXO lookups involve.
They also compare against HyperLevelDB, where the differences are much less pronounced and actually HyperLevelDB appears to be able to do random writes faster than Sophia.
Only slightly related to this... What's the reason why BerkleyDB is maintained for the wallet? I think it would be a good thing to get rid of the libdb4.8++-dev dependency that makes bitcoind harder to compile on debian and ubuntu. Unless, of course, there's a reason I am missing...
On 9/17/13, Mike Hearn <mike@plan99•net> wrote:
Quoted text (30 lines)
> LevelDB is fast - very fast if you give it enough CPU time and disk seeks. > But it's not the last word in performance. > > HyperLevelDB is a forked LevelDB with some changes, mostly, finer grained > locking and changes to how compaction works: > > http://hyperdex.org/performance/leveldb/ > > However, it comes with a caveat - one of the changes they made is to take > away write throttling if compaction falls behind, the app itself is > expected to do that. > > Sophia is a competitor to LevelDB. The website claims that in benchmarks it > completely smokes LevelDB. I have not explored how it does this or tried to > replicate their benchmarks myself: > > http://sphia.org/index.html > http://sphia.org/benchmarks.html > > It's written in C and BSD licensed. > > As an example of the kind of speedup they claim to be capable of, they say > LevelDB could do 167,476 random reads per second on their SSD based > machine. Sophia could do 438,084 reads/sec. Random reads are of course the > most interesting for us because that's what UTXO lookups involve. > > They also compare against HyperLevelDB, where the differences are much less > pronounced and actually HyperLevelDB appears to be able to do random writes > faster than Sophia. >
Nobody has written code to use a better format, migrate old wallets, etc.
On Tue, Sep 17, 2013 at 1:41 PM, Jorge Timón <jtimon@monetize•io> wrote:
Quoted text (52 lines)
> Only slightly related to this... > What's the reason why BerkleyDB is maintained for the wallet? > I think it would be a good thing to get rid of the libdb4.8++-dev > dependency that makes bitcoind harder to compile on debian and ubuntu. > Unless, of course, there's a reason I am missing... > > > On 9/17/13, Mike Hearn <mike@plan99•net> wrote: > > LevelDB is fast - very fast if you give it enough CPU time and disk > seeks. > > But it's not the last word in performance. > > > > HyperLevelDB is a forked LevelDB with some changes, mostly, finer grained > > locking and changes to how compaction works: > > > > http://hyperdex.org/performance/leveldb/ > > > > However, it comes with a caveat - one of the changes they made is to take > > away write throttling if compaction falls behind, the app itself is > > expected to do that. > > > > Sophia is a competitor to LevelDB. The website claims that in benchmarks > it > > completely smokes LevelDB. I have not explored how it does this or tried > to > > replicate their benchmarks myself: > > > > http://sphia.org/index.html > > http://sphia.org/benchmarks.html > > > > It's written in C and BSD licensed. > > > > As an example of the kind of speedup they claim to be capable of, they > say > > LevelDB could do 167,476 random reads per second on their SSD based > > machine. Sophia could do 438,084 reads/sec. Random reads are of course > the > > most interesting for us because that's what UTXO lookups involve. > > > > They also compare against HyperLevelDB, where the differences are much > less > > pronounced and actually HyperLevelDB appears to be able to do random > writes > > faster than Sophia. > > > > > -- > Jorge Timón > > http://freico.in/ >
On Tue, Sep 17, 2013 at 4:00 AM, Mike Hearn <mike@plan99•net> wrote:
Quoted text (2 lines)
> LevelDB is fast - very fast if you give it enough CPU time and disk seeks. > But it's not the last word in performance.
I'd looked at the hyperleveldb, but their performance graphs made it seem like it would be slower for the actual database sizes we're using today.
Is there a competitor that specializes in being more robust to corruption? :(
On 9/17/13, Mike Hearn <mike@plan99•net> wrote:
Quoted text (1 line)
> Nobody has written code to use a better format, migrate old wallets, etc.
ACK, thanks.
Also somewhat related, I have been looking for some time now to abstract out the UTXO and block databases so that a variety of key/value stores could be used as a backend, configured by a command line parameter. In particular, it would be interesting for some server applications to support HyperDex, which is basically a distributed, fault-tolerant version of LevelDB:
By the same mechanism you could just as easily support a Sophia backend.
Mark
On 9/17/13 4:00 AM, Mike Hearn wrote:
Quoted text (36 lines)
> LevelDB is fast - very fast if you give it enough CPU time and > disk seeks. But it's not the last word in performance. > > HyperLevelDB is a forked LevelDB with some changes, mostly, finer > grained locking and changes to how compaction works: > > http://hyperdex.org/performance/leveldb/ > > However, it comes with a caveat - one of the changes they made is > to take away write throttling if compaction falls behind, the app > itself is expected to do that. > > Sophia is a competitor to LevelDB. The website claims that in > benchmarks it completely smokes LevelDB. I have not explored how it > does this or tried to replicate their benchmarks myself: > > http://sphia.org/index.html http://sphia.org/benchmarks.html > > It's written in C and BSD licensed. > > As an example of the kind of speedup they claim to be capable of, > they say LevelDB could do 167,476 random reads per second on their > SSD based machine. Sophia could do 438,084 reads/sec. Random reads > are of course the most interesting for us because that's what UTXO > lookups involve. > > They also compare against HyperLevelDB, where the differences are > much less pronounced and actually HyperLevelDB appears to be able > to do random writes faster than Sophia. > > > > > ------------------------------------------------------------------------------ > >
LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99!
Quoted text (13 lines)
> 1,500+ hours of tutorials including VisualStudio 2012, Windows 8, > SharePoint 2013, SQL 2012, MVC 4, more. BEST VALUE: New > Multi-Library Power Pack includes Mobile, Cloud, Java, and UX > Design. Lowest price ever! Ends 9/20/13. > http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk > > > > > _______________________________________________ Bitcoin-development > mailing list Bitcoin-development@lists•sourceforge•net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development >