Bootstrapping full nodes post-pruning

Apologies if this has been discussed elsewhere. I don't recall us ever reaching a solid conclusion on it.

A node that has pruned its block chain cannot serve the chain to new nodes. So there are three options for bootstrapping a newly installed node:

1) Have some kind of special archival nodes that never prune (advertised via the services field?). Encourage people to run them, somehow.

2) Ship a post-pruning block chain and tx index with the client downloads, so the client starts up already bootstrapped.

3) Some combination of both. It's safe to assume some people will keep unpruned chains around no matter what. But for many users (2) is easiest and archival nodes would be put under less load if they were used only by users who wish to fully bootstrap from only the code.

I remember some people, Greg in particular, who were not a fan of approach (2) at all, though it has the benefit of speeding startup for new users as there's no indexing overhead.

On Sun, Jun 10, 2012 at 7:06 PM, Mike Hearn <mike@plan99•net> wrote:

> I remember some people, Greg in particular, who were not a fan of
> approach (2) at all, though it has the benefit of speeding startup for
> new users as there's no indexing overhead.

I'm not a fan of anything which introduces unauditable single source material. "Trust us" is a bad place to be because it would greatly increase the attractiveness of compromising developers.

If we wanted to go the route of shipping pruned chains I'd prefer to have a deterministic process to produce archival chains and then start introducing commitments to them in the blockchain or something like that. Then a client doing a reverse header sync[1] would bump into a commitment for an archival chain that they have and would simply stop syncing and use the archival chain for points before that.

This would leave it so that the distribution of the software could still be audited.

More generally we should start doing something with the service announcements so that full nodes that don't have enough bandwidth to support a lot of syncing from new nodes can do so without turning off listening.

[1] https://en.bitcoin.it/wiki/User:Gmaxwell/Reverse_header-fetching_sync

> If we wanted to go the route of shipping pruned chains I'd prefer to
> have a deterministic process to produce archival chains

Yeah, that sounds reasonable. I mean, I can't see why pruning would not be deterministic. So if you download a binary that contains a pre-indexed and pruned chain up to block 180,000 or whatever, you should be able to blow away the data files and run with "-syncto=180000 -prune", then check the hashes of the newly created files vs what you downloaded.

Unless BDB has some weird behaviour in it, that shouldn't require any additional effort, and anyone could set up a cron job to verify the downloads match what is expected.

Even if a more complex scheme is used whereby commitments are in the block chain, somebody still has to verify the binaries match the source. If that isn't true, the software could do anything and you'd never know.

On Mon, Jun 11, 2012 at 4:36 PM, Mike Hearn <mike@plan99•net> wrote:

> Unless BDB has some weird behaviour in it, that shouldn't require any

HAHAHA. Have you consider doing comedy full time?

Actual BDB files are absolutely not deterministic. Nor is the raw blockchain itself currently, because blocks aren't always added in the same order (plus they get orphans in them)

But the serious inter-version compatibility problems as well as poor space efficiency make BDB a poor candidate for read only pruned indexes.

> Even if a more complex scheme is used whereby commitments are in the
> block chain, somebody still has to verify the binaries match the
> source. If that isn't true, the software could do anything and you'd
> never know.

The binaries distributed by bitcoin.org are all already compiled deterministically and validated by multiple independent parties. In the future there will be a downloader tool (e.g. for updates) which will automatically check for N approvals before accepting an update, even for technically unsophisticated users.

This will produce a full chain of custody which tracks the actual binaries people fetch to specific source code which can be audited, so substitution attacks will at least in theory always be detectable. Of course, you're left with Ken Thompson's compiler attack but even that can be substantially closed.

> Actual BDB files are absolutely not deterministic. Nor is the raw
> blockchain itself currently, because blocks aren't always added in the
> same order (plus they get orphans in them)

That's true. Though if you prune up to the last checkpoint, orphans before that point can be safely thrown away.

I wonder if swapping out bdb for LevelDB might make sense at some point. I'm not sure how deterministic that is either though :)