Build your own nHashType

Seeking feedback on a proposal that will allow a transaction signer to explicitly specify what is to be serialized for the signature hash. The basic idea is to make the nHashType general enough that we won't need a new sighash flag every time a new use case comes up.

If implemented into bitcoin (via a soft fork), this would make malleability almost a non-issue (the TXID referenced by inputs just need to be updated previous TX changes) and would enable hardware wallets to securely sign without needing to download/process each transaction it spends from.

Please let me know your thoughts.

https://github.com/scmorse/bitcoin-misc/blob/master/sighash_proposal.md

Hi Stephen,

It's an interesting idea. I'm not sure that all the combinations make sense. Excluding the connected output script or value but still signing the prev tx hash appears pointless: the script cannot change anyway, and you still need to know what it is to actually calculate the inputs to it, so what is the point of this?

I also worry that quite a few of these combinations could be unexpectedly dangerous. If you don't sign the prevout hash or value and combine it with a regular pay-to-address output then you've effectively written a blank cheque that can be used by anyone, to claim any money ever sent to that address ... no? And then any p2p node or miner could do so, making the transaction pretty useless.

That isn't inherently a problem as long as people understand which combinations have what effects or cannot be used for various reasons. But it would need good documentation and careful thought to explore each possibility people might use.

I'll leave the soft fork business to one side for now. I think any change in CHECKSIG or new version of it would likely be ready around the same time as the hard fork we need for changing the block size limit anyway, and it's much cleaner to do it that way.

The most important change that we need in sighash calculation, IMO, is ensuring that you don't have to hash data over and over again without a good reason. The current sighash definition is unfortunate because it's possible to make small transactions that involve hashing huge amounts of data. It's not clear to me that your proposal fixes that: ideally there would be one exactly one sighash for one transaction no matter how many checksigs are involved in verifying it.

thanks, -mike

Hi Mike,

Hi Stephen,

>
> It's an interesting idea. I'm not sure that all the combinations make
> sense. Excluding the connected output script or value but still signing the
> prev tx hash appears pointless: the script cannot change anyway, and you
> still need to know what it is to actually calculate the inputs to it, so
> what is the point of this?
>

That's a good point, maybe SIGHASH_WITHOUT_PREV_SCRIPTPUBKEY and SIGHASH_WITHOUT_PREV_VALUE should be assumed false, since you need the data anyway. That gets the total number of flags down to 17. If we eliminate SIGHASH_WITHOUT_TX_VERSION (I can't think of any good reason for this one), then we're down to a 2-byte nHashType. SIGHASH_SIGN_STACK_ELEMENT could also be removed, I'm not convinced of the usefulness of that one either.

>
> I also worry that quite a few of these combinations could be unexpectedly
> dangerous. If you don't sign the prevout hash or value and combine it with
> a regular pay-to-address output then you've effectively written a blank
> cheque that can be used by anyone, to claim any money ever sent to that
> address ... no? And then any p2p
>

node or miner could do so, making the transaction pretty useless.

>
> That isn't inherently a problem as long as people understand which
> combinations have what effects or cannot be used for various reasons. But
> it would need good documentation and careful thought to explore each
> possibility people might use.
>

I don't think it's quite a blank check, but it would enable replay attacks in the form of sending the money to the same place it was sent before if an address ever receives coins again. Almost like auto-forwarding addresses. If, in addition, you signed with just that input and no outputs as well, then you're basically forfeiting your rights to any coins sent to that address.

It allows for some dangerous combinations, but we already have some dangerous nHashTypes. e.g. SIGHASH_NONE | SIGHASH_ANYONECANPAY. Good documentation and careful developers shouldn't have any issues if they use a standard set of sighash flag combinations for their standard use cases. But developers that need special combinations can now use them, so long as they are careful and think things through.

>
> I'll leave the soft fork business to one side for now. I think any change
> in CHECKSIG or new version of it would likely be ready around the same time
> as the hard fork we need for changing the block size limit anyway, and it's
> much cleaner to do it that way.
>
> The most important change that we need in sighash calculation, IMO, is
> ensuring that you don't have to hash data over and over again without a
> good reason. The current sighash definition is unfortunate because it's
> possible to make small transactions that involve hashing huge amounts of
> data. It's not clear to me that your proposal fixes that: ideally there
> would be one exactly one sighash for one transaction no matter how many
> checksigs are involved in verifying it.
>
>

It's hard, though, because there is different data needs to be signed for each input. Although, I suppose if you signed your input with SIGHASH_WITHOUT_PREV_SCRIPTPUBKEY, SIGHASH_WITHOUT_PREV_VALUE, and the equivalent of SIGHASH_ALL, then the hash that needs to be signed would be the same for all of your inputs. Strangely enough, I think we might have just found use cases for the flags that we had nearly dismissed.

Another possibility would be to put the previous scriptPubKey and previous output value at the END of the serialized transaction, so that you could make use of some sort of a signature hash midstate. But that feels a little messy. It sort of makes sense to have a base serialization for a transaction and then append it with whatever input/output specific information you have, but still, messy.

Is hashing transaction data once for each input really a huge bottleneck, though? Do mobile devices have an issue with this?

Best, Stephen

On Thu, Apr 9, 2015 at 7:10 AM, Stephen Morse <stephencalebmorse@gmail•com> wrote:

> Is hashing transaction data once for each input really a huge bottleneck,
> though? Do mobile devices have an issue with this?
>

Think about what slow transaction verification speed means. Slower propagation across the network. More work per node. Greater opportunity for algorithmic attacks, races and other shenanigans by attackers.

>
> I don't think it's quite a blank check, but it would enable replay attacks
> in the form of sending the money to the same place it was sent before if an
> address ever receives coins again.
>

Right, good point. I wonder if this sort of auto forwarding could even be a useful feature. I can't think of one right now.

> It's hard, though, because there is different data needs to be signed for
> each input.
>

Yes but is that fundamental or is there a way to avoid it? That's what I'm getting at.

> Another possibility would be to put the previous scriptPubKey and previous
> output value at the END of the serialized transaction, so that you could
> make use of some sort of a signature hash midstate.
>

Interesting idea! I don't agree it's messy. If anything it should be simpler than what we have today - the need to edit a transaction *in the middle* means that sighash computation involves constantly reserializing a transaction before it even gets to be hashed.

> Is hashing transaction data once for each input really a huge bottleneck,
> though? Do mobile devices have an issue with this?
>

Consider what happens with very large transactions, like a big assurance contract that might have thousands of inputs and be multiple megabytes in size. Obviously such large transactions cannot happen today, but there is user demand for giant contracts (or at least, users tell me there is, whether they'd actually do it for real is a bit unclear).

On Thu, Apr 09, 2015 at 07:22:52AM -0700, Jeff Garzik wrote:

> On Thu, Apr 9, 2015 at 7:10 AM, Stephen Morse <stephencalebmorse@gmail•com>
> wrote:
>
> > Is hashing transaction data once for each input really a huge bottleneck,
> > though? Do mobile devices have an issue with this?
> >
>
>
> Think about what slow transaction verification speed means.  Slower
> propagation across the network.  More work per node.  Greater opportunity
> for algorithmic attacks, races and other shenanigans by attackers.

Keep in mind though we can always make part of the soft-fork be to make the hash operations in the new CHECKSIG mechanism consume sigops.

For the OP: Have you looked at how CODESEPARATOR allows the signature to sign code to run as part of verifying the signature? E.g. my signature can say "valid if you run these additional opcodes and they return true" where those additional opcodes take the transaction, hash it in the defined way, and verify that the ECC signature correctly signs that hash and the hash of the additional opcodes. For instance in this case making a signature that's only valid if the tx fee is less than the defined amount would be a matter of GET_FEE <max fee + 1> LESSTHAN VERIFY

This can be a much more general mechanism with easy to test modular opcodes; for the consensus-critical codebase this can result in a much easier and simpler to test CHECKSIG facility than a dozen new flags.

Regarding the re-hashing the transaction data once per input being a bottleneck, I was mistakenly only thinking about this from the point of view of the signer. Full nodes have to check all transactions' inputs, which is much more costly, as the link Gavin posted shows.

On Thu, Apr 9, 2015 at 10:45 AM, Mike Hearn <mike@plan99•net> wrote:

>
> Right, good point. I wonder if this sort of auto forwarding could even be
> a useful feature. I can't think of one right now.
>

I can think of a few convoluted use cases, but not any good ones. People have definitely looked for this feature before, though, just look at this Bitcoin SE post <http://bitcoin.stackexchange.com/questions/1495/is-there-a-way-to-automatically-send-bitcoins-from-one-wallet-to-another>. I think there are better ways to handle key management than auto-forwarding, though. Anyone looking for this feature probably just wasn't aware that there are better solutions.

On Thu, Apr 9, 2015 at 10:45 AM, Mike Hearn <mike@plan99•net> wrote:

>
> Yes but is that fundamental or is there a way to avoid it? That's what I'm
> getting at.
>

In the bitcointalk article referenced, Sergio actually gave us the answer:

> Hash(Tx,previn-index) = Hash ( Hash(outputs) || Hash

(Inputs-with-script-cleared) || <previn-index> )

>   (for SIGHASH_ALL)
>   This way the values "Hash(outputs)" and

"Hash(Inputs-with-script-cleared)" can be cached and reused.

Basically, just re-order the way stuff is serialized. Put the stuff that is nearly always signed at the beginning, and vice versa. I'll see if I can update the proposal to make this optimization possible. What I suspect, though, is that with all the new controls, blocks with ordinary transactions will verify faster, but an attacker could still create a very CPU intensive block by signing inputs with a wide variety of nHashTypes and then signing the last one with the equivalent of SIGHASH_ALL. I don't think that's a big limitation, though, the attack is already somewhat possible, and would be very hard to do, and doesn't really gain the attacker anything (other than infamy).

On Thu, Apr 9, 2015 at 1:28 PM, Peter Todd <pete@petertodd•org> wrote:

> For the OP: Have you looked at how CODESEPARATOR allows the signature to
> sign code to run as part of verifying the signature? E.g. my signature
> can say "valid if you run these additional opcodes and they return true"
> where those additional opcodes take the transaction, hash it in the
> defined way, and verify that the ECC signature correctly signs that
> hash and the hash of the additional opcodes. For instance in this case
> making a signature that's only valid if the tx fee is less than the
> defined amount would be a matter of GET_FEE <max fee + 1> LESSTHAN VERIFY
>

I've never been able to really see a good use case for OP_CODESEPARATOR, and I'm not sure I completely have my head wrapped around what you're proposing. From this <http://bitcoin.stackexchange.com/questions/34013/what-is-op-codeseparator-used-for> and this <https://bitcointalk.org/index.php?topic=52949.msg631255#msg631255>, though, it seems like OP_CODESEPARATOR cannot really be made useful unless you already have a way to sign without hashing the TXIDs referenced by your input, in which case you need to modify the nHashType.

Best, Stephen

On Wed, Apr 8, 2015 at 7:50 PM, Stephen Morse <stephencalebmorse@gmail•com> wrote:

> Seeking feedback on a proposal that will allow a transaction signer to
> explicitly specify what is to be serialized for the signature hash. The
> basic idea is to make the nHashType general enough that we won't need a new
> sighash flag every time a new use case comes up.
>
> If implemented into bitcoin (via a soft fork), this would make malleability
> almost a non-issue (the TXID referenced by inputs just need to be updated
> previous TX changes) and would enable hardware wallets to securely sign
> without needing to download/process each transaction it spends from.

I'm not sure if I'm super fond of that particular non-programmatic but many options approach; It sort of has the problem that there are relatively few useful options that don't rapidly extend into a choose your own adventure with too many options to count; so you take a complexity penalty perhaps without a matching functionality payoff.

but thats not why I'm commenting...

I wonder if anyone noticed that any sighash masking that eliminates the txin txid enables covenants?

Covenants are payments which constrain their future payments (like deed covenants), I've written about them in the past https://bitcointalk.org/index.php?topic=278122.0 ... they can sometimes be pretty useful but are also potentially a irritating hit to fungibility, at least if used stupidly.

the approach here is that you make the scriptpubkey contain "[push: 0x30, 0x06, 0x02, 0x01, 0x04, 0x02, 0x01, 0x04, flags] [push pubkey resulting from pubkey recovery] OP_CHECKSIG" and set the flags to match only the things you want to enforce in the spending transaction hash them up and recover the EC public point. You can think of that construct as giving a you a OP_MASKED_TRANSACTION_HASH_EQUALS ... the recovered pubkey is just a kind of message hash, though a weird and expensive to compute one.

I don't currently see how to get a perpetual covenant out of it-- e.g. a coin that anyone can spend, but only to its same scriptpubkey, (the obvious way requires the ability to be able to checksig stuff on the stack) though I wouldn't be shocked if it were possible with a sufficiently complex sighash flag and nothing else.

On Thu, Apr 09, 2015 at 10:56:20PM -0400, Stephen Morse wrote:

> On Thu, Apr 9, 2015 at 1:28 PM, Peter Todd <pete@petertodd•org> wrote:
>
> > For the OP: Have you looked at how CODESEPARATOR allows the signature to
> > sign code to run as part of verifying the signature? E.g. my signature
> > can say "valid if you run these additional opcodes and they return true"
> > where those additional opcodes take the transaction, hash it in the
> > defined way, and verify that the ECC signature correctly signs that
> > hash and the hash of the additional opcodes. For instance in this case
> > making a signature that's only valid if the tx fee is less than the
> > defined amount would be a matter of GET_FEE <max fee + 1> LESSTHAN VERIFY
> >
>
> I've never been able to really see a good use case for OP_CODESEPARATOR,
> and I'm not sure I completely have my head wrapped around what you're
> proposing. From this
> <http://bitcoin.stackexchange.com/questions/34013/what-is-op-codeseparator-used-for>
>  and this
> <https://bitcointalk.org/index.php?topic=52949.msg631255#msg631255>,
> though, it seems like OP_CODESEPARATOR cannot really be made useful unless
> you already have a way to sign without hashing the TXIDs referenced by your
> input, in which case you need to modify the nHashType.

I wrote up how to do this on #bitcoin-wizards, Dec 9th 2014:

17:13 < petertodd> hearn: even now you can use OP_CODESEPARATOR to implement efficient payword schemes 17:14 < petertodd> hearn: early on you could have used it to do some really useful after-the-fact signing delegation by wrapping a IF ENDIF around the CODESEPARATOR introduced into the middle of the scriptSig/scriptPubKey pair - shame we got rid of that without thinking the design through 17:15 < petertodd> hearn: e.g. "create a signature that delegates signing authority to another pubkey" 17:15 < petertodd> probably all 100% accidental... but a nice accident 17:16 < hearn> it's probably for the best. i can imagine such things being a surprise for implementations not expecting them. a script 2.0 effort that incorporates lots of neat features but still lets script 1.0 work would be nice to have, one day 17:17 < petertodd> satoshi belived in 1 implementation, and by putting CODESEPARATOR into the scriptSig/scriptPubKey concatenation you had to opt-in to making that feature possible to use in any particular scriptPubKey 17:17 < petertodd> w/o the mis-matched ENDIF you can't pull off that trick because you can't turn CODESEPARATOR off 17:19 < petertodd> to be explicit: scriptPubKey: ENDIF <pubkey> CHECKSIG, then the normal case is scriptSig: <signature> 1 IF 17:19 < petertodd> they concatenate to <signature> 1 IF ENDIF <pubkey> CHECKSIG, CODESEPARATOR is evaluated, and the signature is evaluated on the script ENDIF <pubkey> CHECKSIG 17:20 < petertodd> to delegate signing authority after the fact sign a signature on the script <pubkey2> 0 IF ENDIF <pubkey> CHECKSIG 17:21 < petertodd> (remember that CODESEPARATORS are removed by SignatureHash()) 17:22 < petertodd> oops, I mean: <pubkey2> CHECKSIGVERIFY 0 IF ENDIF <pubkey> CHECKSIG 17:22 < petertodd> anyway, to finally spend it, create another signature with pubkey2 signing the script <pubkey2> CHECKSIGVERIFY 0 IF ENDIF <pubkey> CHECKSIG again, and finally spend it with the scriptSig: <pubkey-sig> <pubkey2-sig> CODESEPARATOR <pubkey2> 0 IF 17:24 < petertodd> after concatenation the script: <pubkey-sig> <pubkey2-sig> CODESEPARATOR <pubkey2> 0 IF CODESEPARATOR ENDIF <pubkey> CHECKSIG is evaluated, the inner signature satisfies, and the outer signature is satisfied only if the scriptPubKey was essentially changed after the fact to also require the inner, second, pubkey2 to be satisfied 17:26 < petertodd> a nice use-case would, forinstance, have been to have a signing robot be able to create signatures offline for a given txout with SIGHASH_SINGLE such that you had a spending limit enforced, and exactly who was then allowed to spend the funds - say a department of a company - could be picked after the fact without re-spending the txout 17:33 < petertodd> gmaxwell: re: script validation state, a good model would be to have the tx input to EvalScript() essentially be a CMerkleTx, and the prevout scriptPubKey be the prevout CTxOut (*maybe* the prevout tx itself... bit dubious there...)