Libconsensus phase 2

> After talking to some people about libconsensus in the last Hong Kong
> conference I realized that my initial plan of exposing one more thing
> at a time would actually probably slow things down.
>
> There's still a promised pdf with pictures that will be released, and
> actually drafting the UML pictures helped realize that the whole
> explanation could be much simpler if #7091 was merged first as the
> last step in phase 1 (that phase has so many contributors that I will
> probably never get finished documenting it). Matt Corallo's idea of
> exposing VerifyScript() through a C API certainly helped a lot in
> cementing the more-minimal-than-earlier dependencies (thanks to Cory
> Fields among many other people before him) that are not part of the
> incomplete but existing libbitcoinconsensus library.
>
> Given this success in protecting encapsulation by exposing things in a
> new library, my instinct was to expose more things: VerifyHeader(),
> VerifyTx() and VerifyBlock() [in that order].
> But all those three new functions depend on storage in one way or
> another. That was part of my reasoning to expose VerifyHeader() first,
> because I believe there will be less discussion on a common interface
> for the stored longest chain than for the utxo view (which may depend
> on other transactions spent within the same block).
> In any case, I realized we should finish putting all the consensus
> critical code in the libconsensus lib and then worry about its "final"
> API.
>
> Therefore I changed the goal of the phase 2 in my libconsensus
> encapsulation planning from "expose VerifyHeader() in the existing
> libconsensus library" to "build all the consensus critical code within
> the existing libconsensus library, even if we don't expose anything
> else". I believe this is much feasible for a single Bitcoin Core
> release cycle and also more of a priority. Other implementations
> experimenting with libconsensus like
> https://github.com/libbitcoin/libbitcoin-consensus will have the
> chance to compare their reimplementations with the future complete
> libbitcoinconsensus without having to worry about the C API, which
> ideally they will help to define.
>
> I repeat, the goal of phase 2 in my upcoming libconsensus
> encapsulation plan is to fully decouple libconsensus from Bitcoin
> Core.
> In phase 3, we can refine the storage interfaces and focus on a
> quasi-final C API.
> In phase 4, we can refine and take out code that doesn't belong in
> libconsensus like CTxOut::GetDustThreshold() in
> primitives/transaction.h and move all those consensus files to the
> consensus folder before creating a separate sub-repository like for
> libsecp256k1. Note that most of the file-moving work can be in
> parallel to phases 2 and 3 and, in fact, by any new developer that is
> willing to exchange rebase-patience for meaningless github stats (I'll
> do it if nobody else wants, but I'm more than happy to delegate there:
> I have more than enough github meaningless stats already).
>
> As said, the document with pictures and the update to #6714 are still
> promised, but until they're ready, merging/reviewing #7091, #7287,
> #7310 and #7311 could do a great deal to make later steps in
> libconsensus phase 2 more readable.
> Most reviewers probably don't need to see any "big picture" to tell
> whether certain functions on Bitcoin Core are consensus-critical or
> not, or whether consensus critical code needs to depend on util.o or
> not.
> But I wouldn't be writing to the mailing list without a plan with
> further words nor pictures if I didn't had what I believe is a
> complete implementation of what I just defined as "libconsensus phase
> 2".
>
> Phase 3 should finish long pending discussions like "should
> libconsensus be C++14 or plain C" which should NOT delay phase 2.
> Phase 4 should be mostly trivial: rename files to the target dir and
> move the remaining unused code out of libconsensus.
> Phase 5 should make Bitcoin Core eat its own dog food and use
> libbitcoinconsensus oonly by its generic C API (I'm sorry if this
> looks too far away for me to even think about detailing it).
>
> The work in progress branch (but hopefully being finished, nit and
> merged within the 0.12.99 cycle) can be found in:
> https://github.com/jtimon/bitcoin/commits/libconsensus-f2
>
> Before sipa asks, signing code may make it into a new library but
> SHOULDN'T BE PART OF LIBBITCOINCONSENSUS. Ideally, all exposed
> functions will return true or false and an error string. It is based
> on last-0.12.99 3cd836c1 but by popular demand I can open it as a
> "DEPENDENT-tagged" PR linking to smaller steps and keeping track of
> steps done. Analogous to the about to be replaced (for a simpler and
> more maintainable example of testchain) #6382. If people like
> Wladimir, Cory and Pieter cannot see that I've been able to reduce my
> overall cry-for-review noise thanks to github adoption of emacs'
> org-mode's [ ] vs [X] I can alwways leave those "big picture" branches
> as "private" branches out of the pull request count.
>
> I expect to publish a phase 3 branch very shortly. But as said I
> expect a lot of discussion on the API part, so I don't expect big
> movements in phase 3 until phase 2 is done (as said phase 4 is
> orthogonal to anything, this time git will say "verified MOVEONLY" for
> us).
>
> To finish this long mail, if you are new to free software and would
> like to get familiarized with Bitcoin Core development in particular,
> moving one file is a simple task that you can always besure you can do
> right.
> The way I plan to hand this to you, you won't need to convince anyone
> to publicly confirm that your "MOVEONLY" commit being legit, because
> all your remaining work will be to build on one platform (ideally you
> should do a gitian build, but embarrassingly enough for someone
> touching consensus code I just trust travis ) and trust travis (as
> said, that's what I do from my laptop, but I plan to buy my own
> building machine [and maybe outsource it for free in some protocol
> that hasn't been invented, sorry again for the distraction]) and fix
> the includes that have stopped working.
>
> I intend to create an issue to move all the files in this list one by one:
>
> https://github.com/bitcoin/bitcoin/pull/7091/files#diff-480477e89f9b6ddafb30c4383dcdd705R250
>
> But don't hesitate to contact me if are eager for moving some files,
> because I believe we can save a few lines of total diff if we chose
> the order of the movements properly.
>
> Sorry, I forgot many people read this list again.
> Happy to answer any question.
>
> Specially about https://github.com/jtimon/bitcoin/commits/libconsensus-f2
>

> Jorge, first, thanks again for your work on this.
>
> Without creating and using a public blockchain interface in phase 2, how
> will you isolate the database dependency from consensus critical code?
> Is it that the interface will exist but you will recommend against its use?

> This work presumes that the users of the library reject the argument
> that the database implementation is consensus critical code. Faithful
> reproduction of stored data is a prerequisite for a validity. But a
> common store implementation is only slightly more reasonable for this
> library than a common RAM implementation.

> On Tue, Jan 12, 2016 at 8:17 PM, Eric Voskuil <eric@voskuil•org> wrote:
>> Jorge, first, thanks again for your work on this.
>>
>> Without creating and using a public blockchain interface in phase 2, how
>> will you isolate the database dependency from consensus critical code?
>> Is it that the interface will exist but you will recommend against its use?
>
> The interface will exist but it will be a C++ interface that fits
> better with Bitcoin Core internals.
> See an initial draft of what could be the storage interface:
> https://github.com/jtimon/bitcoin/blob/1717db89c6db17ea65ddbd5eb19034315db0b059/src/consensus/storage_interfaces_cpp.h
>
> Phase 3 will consist on discussing and refining that interface to also
> define the C interfaces using structs of function pointers instead of
> classes (see https://github.com/jtimon/bitcoin/blob/2bcc07c014e5dd000030e666be047dfa11f54c10/src/consensus/interfaces.h
> for an early draft) that is needed for the "final" C API.
> But since I think there will be more discussion and work defining
> those interfaces, I would rather start with ANY interface that allows
> us to decouple the consensus code from chain.o and coins.o, which we
> don't want to be built as part of the consensus building package
> (which is used for building both libbitcoinconsensus and Bitcoin
> Core).

> Future potential users are more than welcomed to draft their own C
> APIs and that experience should be useful for phase 3.
> I was expecting you, for example, to include the whole consensus code
> (even if it lacks a C API) in
> https://github.com/libbitcoin/libbitcoin-consensus for better testing
> of the equivalent code in libbitcoin. You are kind of taking the C API
> part out already, so this time you will just have less things to
> delete/ignore.

>> This work presumes that the users of the library reject the argument
>> that the database implementation is consensus critical code. Faithful
>> reproduction of stored data is a prerequisite for a validity. But a
>> common store implementation is only slightly more reasonable for this
>> library than a common RAM implementation.
>
> This is a concern that has been risen repeatedly.
> I am aware that faithful reproduction of the stored data is a
> prerequisite for consensus validity. On the other hand, my presumption
> is that a libbitcoinconsensus that forces its users to a given unifrom
> storage will likely had much less users and any alternative
> implementation that wants to implement its own custom storage would
> have to necessarily reimplement the consensus validation code.
> Doing it this way is more flexible. We can relatively easily implement
> another library (if I remember correctly, last time we talked about it
> we reffered to it as "libconsensus plus", but there's probably better
> names) also takes care of storage for the users that don't want to
> take the risks of reimplementing the storage (probably just using
> Bitcoin Core's structures).
>
> Unlike me, Luke Dashjr, for example, advocated for the
> storage-dependent version, but I believe that implementing both
> versions was an acceptable solution to him.
> It is certainly an acceptable solution for me. I don't want to force
> anyone that doesn't want or need to take the risks reimplementing the
> consensus storage part to do so. But at the same time I really believe
> that it would be a mistake to not allow it optionally.
>
> Does that make sense?