Signet
11 messages, 8 to 14 Mar 2019. Started by Karl-Johan Alm.
Karl-Johan Alm (6), Matt Corallo, Lautaro Dragan, David A. Harding, Anthony Towns, Varunram Ganesh
Hello,
As some of you already know, I've been working on a network called "signet", which is bascially a complement to the already existing testnet, except it is completely centralized, and blocks are signed by a specific key rather than using proof of work.
Benefits of this:
1. It is more predictable than testnet. Miners appear and disappear regularly, causing irregular block generation.
2. Since it is centrally controlled, it is easy to perform global testing, such as reorgs (e.g. the network performs a 4 block reorg by request, or as scheduled).
3. It is more stable than testnet, which occasionally sees several thousand block reorgs.
4. It is trivial to spin up (and shut down) new signets to make public tests where anyone can participate.
Anyone can create a signet at any time, simply by creating a key pair and creating a challenge (scriptPubKey). The network can then be used globally by anyone, assuming the creator sends some coins to the other participants.
Having a persistent signet would be beneficial in particular to services which need a stable place to test features over an extended period of time. My own company implements protocols on top of Bitcoin with sidechains. We need multi-node test frameworks to behave in a predictable manner (unlike testnet) and with the same standardness relay policy as mainnet.
Signets consist of 2 parameters: the challenge script (scriptPubKey) and the solution length. (The latter is needed to retain fixed length block headers, despite having an additional payload.)
I propose that a default persistent "signet1" is created, which can be replaced in future versions e.g. if the coins are unwisely used as real money, similarly to what happened to previous testnets. This signet is picked by default if a user includes -signet without providing any of the parameters mentioned above. The key holder would be someone sufficiently trusted in the community, who would be willing to run the system (block generation code, faucet, etc). It could be made a little more sturdy by using 1-of-N multisig as the challenge, in case 1 <= x < N of the signers disappear. If people oppose this, it can be skipped, but will mean people can't just jump onto signet without first tracking down parameters from somewhere.
Implementation-wise, the code adds an std::map with block hash to block signature. This is serialized/deserialized as appropriate (Segwit witness style), which means block headers in p2p messages are (80 + solution_length) bytes. Block header non-contextual check goes from checking if block header hash < target to checking if the payload is a valid signature for the block header hash instead.
Single commit with code (will split into commits and make PR later, but just to give an idea what it looks like): https://github.com/kallewoof/bitcoin/pull/4
I don't think this PR is overly intrusive, and I'm hoping to be able to get signet code into Bitcoin Core eventually, and am equally hopeful that devs of other (wallet etc) implementations will consider supporting it.
Feedback requested on this.
Attribution: parts of the signet code (in particular signblock and getnewblockhex) were adapted from the ElementsProject/elements repository. When PR is split into atomic commits, I will put appropriate attribution there.
To make testing easier, it may make sense to keep the existing block header format (and PoW) and instead apply the signature rules to some field in the coinbase transaction. This means SPV clients (assuming they only connect to honest/trusted nodes) work as-is.
A previous idea regarding reorgs (that I believe Greg came up with) is to allow multiple keys to sign blocks, with one signing no reorgs and one signing a reorg every few blocks, allowing users to choose the behavior they want.
Quoted text (38 lines)
> On Mar 8, 2019, at 00:54, Karl-Johan Alm via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> wrote: > > Hello, > > As some of you already know, I've been working on a network called "signet", which is bascially a complement to the already existing testnet, except it is completely centralized, and blocks are signed by a specific key rather than using proof of work. > > Benefits of this: > > 1. It is more predictable than testnet. Miners appear and disappear regularly, causing irregular block generation. > > 2. Since it is centrally controlled, it is easy to perform global testing, such as reorgs (e.g. the network performs a 4 block reorg by request, or as scheduled). > > 3. It is more stable than testnet, which occasionally sees several thousand block reorgs. > > 4. It is trivial to spin up (and shut down) new signets to make public tests where anyone can participate. > > Anyone can create a signet at any time, simply by creating a key pair and creating a challenge (scriptPubKey). The network can then be used globally by anyone, assuming the creator sends some coins to the other participants. > > Having a persistent signet would be beneficial in particular to services which need a stable place to test features over an extended period of time. My own company implements protocols on top of Bitcoin with sidechains. We need multi-node test frameworks to behave in a predictable manner (unlike testnet) and with the same standardness relay policy as mainnet. > > Signets consist of 2 parameters: the challenge script (scriptPubKey) and the solution length. (The latter is needed to retain fixed length block headers, despite having an additional payload.) > > I propose that a default persistent "signet1" is created, which can be replaced in future versions e.g. if the coins are unwisely used as real money, similarly to what happened to previous testnets. This signet is picked by default if a user includes -signet without providing any of the parameters mentioned above. The key holder would be someone sufficiently trusted in the community, who would be willing to run the system (block generation code, faucet, etc). It could be made a little more sturdy by using 1-of-N multisig as the challenge, in case 1 <= x < N of the signers disappear. If people oppose this, it can be skipped, but will mean people can't just jump onto signet without first tracking down parameters from somewhere. > > Implementation-wise, the code adds an std::map with block hash to block signature. This is serialized/deserialized as appropriate (Segwit witness style), which means block headers in p2p messages are (80 + solution_length) bytes. Block header non-contextual check goes from checking if block header hash < target to checking if the payload is a valid signature for the block header hash instead. > > Single commit with code (will split into commits and make PR later, but just to give an idea what it looks like): https://github.com/kallewoof/bitcoin/pull/4 > > I don't think this PR is overly intrusive, and I'm hoping to be able to get signet code into Bitcoin Core eventually, and am equally hopeful that devs of other (wallet etc) implementations will consider supporting it. > > Feedback requested on this. > > Attribution: parts of the signet code (in particular signblock and getnewblockhex) were adapted from the ElementsProject/elements repository. When PR is split into atomic commits, I will put appropriate attribution there. > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Hi Karl-Johan, my two cents:
At Po.et we use regtest to simulate reorgs in integration tests in Travis / CircleCI. It has proved quite useful.
In general regtest for automated testing has given us all we needed so far, but I admit we have a rather simple use of Bitcoin right now (colored coins).
For local development, we sometimes use a script that "mines" blocks in regtest periodically. It was my goal to also use this method in QA, although we wound up using testnet and didn't encounter any problems so far.
Out of curiosity: what limitations did you find in using, for example, a private network of bitcoin core nodes running regtest? (this gives you the same power as centralization without any changes or extra functionality required)
El vie., 8 de mar. de 2019 a la(s) 16:02, Karl-Johan Alm via bitcoin-dev < bitcoin-dev@lists•linuxfoundation•org> escribió:
Quoted text (75 lines)
> Hello, > > As some of you already know, I've been working on a network called > "signet", which is bascially a complement to the already existing testnet, > except it is completely centralized, and blocks are signed by a specific > key rather than using proof of work. > > Benefits of this: > > 1. It is more predictable than testnet. Miners appear and disappear > regularly, causing irregular block generation. > > 2. Since it is centrally controlled, it is easy to perform global testing, > such as reorgs (e.g. the network performs a 4 block reorg by request, or as > scheduled). > > 3. It is more stable than testnet, which occasionally sees several > thousand block reorgs. > > 4. It is trivial to spin up (and shut down) new signets to make public > tests where anyone can participate. > > Anyone can create a signet at any time, simply by creating a key pair and > creating a challenge (scriptPubKey). The network can then be used globally > by anyone, assuming the creator sends some coins to the other participants. > > Having a persistent signet would be beneficial in particular to services > which need a stable place to test features over an extended period of time. > My own company implements protocols on top of Bitcoin with sidechains. We > need multi-node test frameworks to behave in a predictable manner (unlike > testnet) and with the same standardness relay policy as mainnet. > > Signets consist of 2 parameters: the challenge script (scriptPubKey) and > the solution length. (The latter is needed to retain fixed length block > headers, despite having an additional payload.) > > I propose that a default persistent "signet1" is created, which can be > replaced in future versions e.g. if the coins are unwisely used as real > money, similarly to what happened to previous testnets. This signet is > picked by default if a user includes -signet without providing any of the > parameters mentioned above. The key holder would be someone sufficiently > trusted in the community, who would be willing to run the system (block > generation code, faucet, etc). It could be made a little more sturdy by > using 1-of-N multisig as the challenge, in case 1 <= x < N of the signers > disappear. If people oppose this, it can be skipped, but will mean people > can't just jump onto signet without first tracking down parameters from > somewhere. > > Implementation-wise, the code adds an std::map with block hash to block > signature. This is serialized/deserialized as appropriate (Segwit witness > style), which means block headers in p2p messages are (80 + > solution_length) bytes. Block header non-contextual check goes from > checking if block header hash < target to checking if the payload is a > valid signature for the block header hash instead. > > Single commit with code (will split into commits and make PR later, but > just to give an idea what it looks like): > https://github.com/kallewoof/bitcoin/pull/4 > > I don't think this PR is overly intrusive, and I'm hoping to be able to > get signet code into Bitcoin Core eventually, and am equally hopeful that > devs of other (wallet etc) implementations will consider supporting it. > > Feedback requested on this. > > Attribution: parts of the signet code (in particular signblock and > getnewblockhex) were adapted from the ElementsProject/elements repository. > When PR is split into atomic commits, I will put appropriate attribution > there. > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >
Hi Matt,
On Sat, Mar 9, 2019 at 5:20 AM Matt Corallo <lf-lists@mattcorallo•com> wrote:
Quoted text (2 lines)
> > To make testing easier, it may make sense to keep the existing block header format (and PoW) and instead apply the signature rules to some field in the coinbase transaction. This means SPV clients (assuming they only connect to honest/trusted nodes) work as-is.
Keeping the PoW rule and moving the signature would mean DoS attacks would be trivial as anyone could mine blocks without a signature in them, unless you ramped up the difficulty, which would mean it's just another testnet. It's a test network, admittedly, but I think it would kind of defeat the purpose.
Quoted text (1 line)
> A previous idea regarding reorgs (that I believe Greg came up with) is to allow multiple keys to sign blocks, with one signing no reorgs and one signing a reorg every few blocks, allowing users to choose the behavior they want.
Not sure how this would work in practice. The idea with signet is to have an actual network that is occasionally reorged, i.e. it's a global network (for those participating) that everyone agrees on. Not sure how you would have choices there.
Hi Lautaro,
Using regtest is not ideal for public networks, as anyone anywhere can just rewrite the blockchain at their whim by mining a ton of blocks.
On Sun, Mar 10, 2019 at 4:52 AM Lautaro Dragan <lautarodragan@gmail•com> wrote:
Quoted text (49 lines)
> > Hi Karl-Johan, my two cents: > > At Po.et we use regtest to simulate reorgs in integration tests in Travis / CircleCI. It has proved quite useful. > > In general regtest for automated testing has given us all we needed so far, but I admit we have a rather simple use of Bitcoin right now (colored coins). > > For local development, we sometimes use a script that "mines" blocks in regtest periodically. It was my goal to also use this method in QA, although we wound up using testnet and didn't encounter any problems so far. > > Out of curiosity: what limitations did you find in using, for example, a private network of bitcoin core nodes running regtest? (this gives you the same power as centralization without any changes or extra functionality required) > > El vie., 8 de mar. de 2019 a la(s) 16:02, Karl-Johan Alm via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> escribió: >> >> Hello, >> >> As some of you already know, I've been working on a network called "signet", which is bascially a complement to the already existing testnet, except it is completely centralized, and blocks are signed by a specific key rather than using proof of work. >> >> Benefits of this: >> >> 1. It is more predictable than testnet. Miners appear and disappear regularly, causing irregular block generation. >> >> 2. Since it is centrally controlled, it is easy to perform global testing, such as reorgs (e.g. the network performs a 4 block reorg by request, or as scheduled). >> >> 3. It is more stable than testnet, which occasionally sees several thousand block reorgs. >> >> 4. It is trivial to spin up (and shut down) new signets to make public tests where anyone can participate. >> >> Anyone can create a signet at any time, simply by creating a key pair and creating a challenge (scriptPubKey). The network can then be used globally by anyone, assuming the creator sends some coins to the other participants. >> >> Having a persistent signet would be beneficial in particular to services which need a stable place to test features over an extended period of time. My own company implements protocols on top of Bitcoin with sidechains. We need multi-node test frameworks to behave in a predictable manner (unlike testnet) and with the same standardness relay policy as mainnet. >> >> Signets consist of 2 parameters: the challenge script (scriptPubKey) and the solution length. (The latter is needed to retain fixed length block headers, despite having an additional payload.) >> >> I propose that a default persistent "signet1" is created, which can be replaced in future versions e.g. if the coins are unwisely used as real money, similarly to what happened to previous testnets. This signet is picked by default if a user includes -signet without providing any of the parameters mentioned above. The key holder would be someone sufficiently trusted in the community, who would be willing to run the system (block generation code, faucet, etc). It could be made a little more sturdy by using 1-of-N multisig as the challenge, in case 1 <= x < N of the signers disappear. If people oppose this, it can be skipped, but will mean people can't just jump onto signet without first tracking down parameters from somewhere. >> >> Implementation-wise, the code adds an std::map with block hash to block signature. This is serialized/deserialized as appropriate (Segwit witness style), which means block headers in p2p messages are (80 + solution_length) bytes. Block header non-contextual check goes from checking if block header hash < target to checking if the payload is a valid signature for the block header hash instead. >> >> Single commit with code (will split into commits and make PR later, but just to give an idea what it looks like): https://github.com/kallewoof/bitcoin/pull/4 >> >> I don't think this PR is overly intrusive, and I'm hoping to be able to get signet code into Bitcoin Core eventually, and am equally hopeful that devs of other (wallet etc) implementations will consider supporting it. >> >> Feedback requested on this. >> >> Attribution: parts of the signet code (in particular signblock and getnewblockhex) were adapted from the ElementsProject/elements repository. When PR is split into atomic commits, I will put appropriate attribution there. >> >> _______________________________________________ >> bitcoin-dev mailing list >> bitcoin-dev@lists•linuxfoundation•org >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
On Sun, Mar 10, 2019 at 09:43:43AM +0900, Karl-Johan Alm via bitcoin-dev wrote:
Quoted text (3 lines)
> Keeping the PoW rule and moving the signature would mean DoS attacks > would be trivial as anyone could mine blocks without a signature in > them
Sure, but anyone could also just connect their lite client to a trusted node (or nodes) on signet. The nodes would protect the clients from missing/invalid-signature DoS and the clients wouldn't have to implement any more network-level changes than they need to now for testnet.
For people who don't want to run their own trusted signet nodes, there could be a list of signet nodes run by well-known Bitcoiners (and this could even be made available via a simple static dns seeder lite clients could use).
Quoted text (8 lines)
> On Sat, Mar 9, 2019 at 5:20 AM Matt Corallo <lf-lists@mattcorallo•com> > wrote: > > A previous idea regarding reorgs (that I believe Greg came up with) > > is to allow multiple keys to sign blocks, with one signing no reorgs > > and one signing a reorg every few blocks, allowing users to choose > > the behavior they want. > > Not sure how this would work in practice.
This post from Maxwell could be the idea Corallo is describing:
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-August/016348.html
I read it as:
- Trusted signer Alice only signs extensions of her previous blocks
- Trusted signer Bob periodically extends one of Alice's blocks
(either the tip or an earlier block) with a chain that grows faster
than Alice's chain, becoming the most-PoW chain. At some point he
stops and Alice's chain overtakes Bob's fork as the most-PoW chain
- User0 who wants to ignore reorg problems starts his node with
-signet -signers="alice", causing his node to only accept blocks
from Alice.
- User1 who wants to consider reorg problems starts his node with
-signet -signers="alice,bob", causing his node to accept blocks from
both Alice and Bob, thus experiencing periodic reorgs.
- There can also be other signing keys for any sort of attack
that can be practically executed, allowing clients to test their
response to the attack when they want to but also ignore any
disruption it would otherwise cause the rest of the time.
- As an alternative to particular signing keys, there could just be
flags put in the header versionbits, header nonce, or generation
transaction indicating how the block should be classified (e.g.
no_reorg, reorg_max6, reorg_max144, merkle_vulnerability, special0,
special1, etc...)
(If something like this is implemented, I propose reserving one of the signing keys/classification flags for use by any of Bitcoin's more devious devs in unannounced attacks. Having to occasionally dig through weird log messages and odd blocks with other Bitcoin dorks on IRC in order to figure out why things went horribly sideways in our signet clients sounds to me like an enjoyable experience. :-)
-Dave
Hello all,
I started writing code that puts the signature in the coinbase transaction similar to the witness commitment, and encountered a potential issue. See inline comments below.
On Mon, Mar 11, 2019 at 2:02 AM David A. Harding <dave@dtrt•org> wrote:
Quoted text (15 lines)
> > On Sun, Mar 10, 2019 at 09:43:43AM +0900, Karl-Johan Alm via bitcoin-dev wrote: > > Keeping the PoW rule and moving the signature would mean DoS attacks > > would be trivial as anyone could mine blocks without a signature in > > them > > Sure, but anyone could also just connect their lite client to a trusted > node (or nodes) on signet. The nodes would protect the clients from > missing/invalid-signature DoS and the clients wouldn't have to implement > any more network-level changes than they need to now for testnet. > > For people who don't want to run their own trusted signet nodes, there > could be a list of signet nodes run by well-known Bitcoiners (and this > could even be made available via a simple static dns seeder lite clients > could use).
This sounds sensible. One issue here is that the "proper" signer will be orders of magnitude slower than the fake miner when constructing blocks. Because the signature is now stuffed into the coinbase transaction, it becomes a part of the block merkle root, so the true miner now has to (1) create a block, (2) sign it, (3) check if hash < target, (4) nudge nonce if not, and then repeat from step (2) until it finds a valid block. I.e. it has to sign the entire thing for every nonce.
Quoted text (3 lines)
> This post from Maxwell could be the idea Corallo is describing: > > https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-August/016348.html
That's pretty cool. The plan I had was to set up some simple interface where anyone could "order" reorgs whenever they wanted to. It would reorg/double spend people on request (e.g. "send 1 signetcoin to signet1qfoobar and then double spend it in a reorg 3 blocks deep") and so on.
With that kind of tool, I don't know if you need the alternate signing approach you described, but I could be mistaken.
On Fri, Mar 08, 2019 at 03:20:49PM -0500, Matt Corallo via bitcoin-dev wrote:
Quoted text (3 lines)
> To make testing easier, it may make sense to keep the existing block header > format (and PoW) and instead apply the signature rules to some field in the > coinbase transaction.
Maybe make the signature be an optional addition to the header, so that you can have a "light node" that doesn't download/verify sigs and a full node that does? (So signatures just sign the traditional 80-byte header, and aren't included in the block's tx merkle root, and the prevHash reflects the hash of the previous block's 80-byte header, without the signature)
I think you could do that by adding a p2p service bit to say "send me signatures if you have them / I can send you signatures", which changes the p2p encoding of the header from (ver, prev, mrkl, time, bits, nonce) to (ver, prev, mrkl, time, 0, nonce, bits, sig), and change header processing to ignore headers from nodes that don't have the service bit set?
If you did this, it might be a good idea to enforce including the previous block's header signature in the current block's coinbase.
Cheers, aj
Hello Kalle,
I like your idea of a signet as it would greatly help test reorgs and stuff without having to experiment with regtest. But I'm a bit concerned about running a common signet (Signet1) controlled by a trusted entity. I guess if someone wants to test signet on a global scale, they could spin up a couple nodes in a couple places (and since it is anyway trusted, they can choose to run it on centralised services like AWS). Another concern is that the maintainer might have unscheduled work, emergencies, etc and that could affect how people test stuff on. This would also mean that we need people to run signet1 nodes in parallel with current testnet nodes (one could argue that Signet is trusted anyway and this doesn't matter, still)
I'm sure you would have considered these while designing, so would be great to hear your thoughts.
Regards, Varunram
Hi Anthony,
On Wed, Mar 13, 2019 at 12:23 PM Anthony Towns <aj@erisian•com•au> wrote:
Quoted text (7 lines)
> > Maybe make the signature be an optional addition to the header, so > that you can have a "light node" that doesn't download/verify sigs > and a full node that does? (So signatures just sign the traditional > 80-byte header, and aren't included in the block's tx merkle root, and > the prevHash reflects the hash of the previous block's 80-byte header, > without the signature)
This seems to be what everyone around me thinks is the best approach. I.e. signet is a "p2p level agreement" that an additional signature is required for a block to be considered fully valid.
It has the added complexity that a signature-aware signet node talking to a non-signature-aware signet node should reject/discard headers sent from the peer, or you will run into situations where a node doesn't know if the headers are valid or not and has to hold onto them indefinitely, which is a situation that currently does not occur in "regular" nets.
If you detach the signature from the header, you also end up with cases where a malicious user could send garbage data as the signature for a valid header, forcing peers to mark that header as invalid, even though it isn't. That is regardless of whether a fix is in place for the above, too.
Quoted text (2 lines)
> If you did this, it might be a good idea to enforce including the previous > block's header signature in the current block's coinbase.
Yeah that is one of the ideas we had early on, and I think it's a good one to do. It doesn't mean someone cannot spam a bunch of invalid headers at block height current_tip+1, but at least you can get all but the latest signature now. So as long as you are able to fetch the latest signature, you should theoretically be able to verify the entire chain just from the blocks + that one sig.
-Kalle.
Hi Varunram,
On Wed, Mar 13, 2019 at 3:41 PM Varunram Ganesh <varunramganesh@gmail•com> wrote:
Quoted text (4 lines)
> > I like your idea of a signet as it would greatly help test reorgs and stuff without having to experiment with regtest. But I'm a bit concerned about running a common signet (Signet1) controlled by a trusted entity. I guess if someone wants to test signet on a global scale, they could spin up a couple nodes in a couple places (and since it is anyway trusted, they can choose to run it on centralised services like AWS). Another concern is that the maintainer might have unscheduled work, emergencies, etc and that could affect how people test stuff on. This would also mean that we need people to run signet1 nodes in parallel with current testnet nodes (one could argue that Signet is trusted anyway and this doesn't matter, still) > > I'm sure you would have considered these while designing, so would be great to hear your thoughts.
For starters, I assume that the signer would run an automated script that generated blocks on regular intervals without requiring manual interaction. So even if the signer went on a vacation, the network would keep on ticking. I also assume the signer would be running a faucet service so users could get coins as needed. Ultimately though, if a signer ended up vanishing or being unreliable, people would just set up a new signet with a different signer and use that instead, so ultimately it's not a big deal.