Two questions about segwit implementation
10 messages, 25 to 27 May 2019. Started by Aymeric Vitte.
Aymeric Vitte (4), Johnson Lau (3), Pieter Wuille, Thomas Kerin, Kostas Karasavvas
I realized recently that my segwit implementation was not correct, basically some time ago, wrongly reading the specs (and misleaded by what follows), I thought that scriptsig would go into witness data as it was, but that's not the case, op_pushdata is replaced by varlen
Now reading correctly the specs, they seem to be not totally correct, then the first question is: why OP_0 is 00 in witness data and not 0100? Does this apply to other op_codes? This does not look logical at all
The second question is: why for non segwit inputs there is a 00 length in segwit data, what is the rational for that? It should just be nothing since you don't need this to reconciliate things
Quoted text (7 lines)
> On 26 May 2019, at 7:56 AM, Aymeric Vitte via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> wrote: > > I realized recently that my segwit implementation was not correct, > basically some time ago, wrongly reading the specs (and misleaded by > what follows), I thought that scriptsig would go into witness data as it > was, but that's not the case, op_pushdata is replaced by varlen >
Witness is not script. There is no op_pushdata or any other opcodes.
Witness is a stack. For each input, the witness starts with a CCompactSize for the number of stack elements for this input. Each stack element in turns starts with a CCompactSize for the size of this element, followed by the actual data
Quoted text (4 lines)
> Now reading correctly the specs, they seem to be not totally correct, > then the first question is: why OP_0 is 00 in witness data and not 0100? > Does this apply to other op_codes? This does not look logical at all >
A “00” element means the size of this element is zero. Since it’s zero size, no data is followed. This will create an empty element on the stack. It’s effectively same as OP_0 (Again, witness is not script)
A “0100” element means the element size is one, and the data for this element is “00”. So it will leave an 1-byte element on the stack.
Quoted text (3 lines)
> The second question is: why for non segwit inputs there is a 00 length > in segwit data, what is the rational for that? It should just be nothing > since you don't need this to reconciliate things
The “00” here means "this input has no witness stack element”. You need this even for non segwit inputs, because there is no way to tell whether an input is segwit-enabled or not, until you look up the UTXO, which might not be always available. Transaction serialization couldn’t rely on contextual information.
However, if all inputs have no stack element, the spec requires you to always use the non-segwit serialization.
Quoted text (5 lines)
> > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
OK, thanks, understood for OP_0 but still for the 00 number of witness data for non segwit inputs the one that is doing the transaction knows which inputs are segwit or not, then parsing the transaction you can associate the correct input to the correct witness data, without the need of 00, so I must be missing the use case
Le 26/05/2019 à 16:33, Johnson Lau a écrit :
Quoted text (33 lines)
>> On 26 May 2019, at 7:56 AM, Aymeric Vitte via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> wrote: >> >> I realized recently that my segwit implementation was not correct, >> basically some time ago, wrongly reading the specs (and misleaded by >> what follows), I thought that scriptsig would go into witness data as it >> was, but that's not the case, op_pushdata is replaced by varlen >> > Witness is not script. There is no op_pushdata or any other opcodes. > > Witness is a stack. For each input, the witness starts with a CCompactSize for the number of stack elements for this input. Each stack element in turns starts with a CCompactSize for the size of this element, followed by the actual data > > >> Now reading correctly the specs, they seem to be not totally correct, >> then the first question is: why OP_0 is 00 in witness data and not 0100? >> Does this apply to other op_codes? This does not look logical at all >> > A “00” element means the size of this element is zero. Since it’s zero size, no data is followed. This will create an empty element on the stack. It’s effectively same as OP_0 (Again, witness is not script) > > A “0100” element means the element size is one, and the data for this element is “00”. So it will leave an 1-byte element on the stack. > > >> The second question is: why for non segwit inputs there is a 00 length >> in segwit data, what is the rational for that? It should just be nothing >> since you don't need this to reconciliate things > The “00” here means "this input has no witness stack element”. You need this even for non segwit inputs, because there is no way to tell whether an input is segwit-enabled or not, until you look up the UTXO, which might not be always available. Transaction serialization couldn’t rely on contextual information. > > However, if all inputs have no stack element, the spec requires you to always use the non-segwit serialization. > >> _______________________________________________ >> bitcoin-dev mailing list >> bitcoin-dev@lists•linuxfoundation•org >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >
This is not how it works. While the transaction creator may know which inputs are segwit, the validators have no way to tell until they look up the UTXO set.
In a transaction, all information about an input the validators have is the 36-byte outpoint (txid + index). Just by looking at the outpoint, there is no way to tell whether it is segwit-enabled or not. So there needs to be a way to tell the validator that “the witness for this input is empty”, and it is the “00”.
Quoted text (7 lines)
> On 27 May 2019, at 12:18 AM, Aymeric Vitte <vitteaymeric@gmail•com> wrote: > > ……. for the 00 number of witness > data for non segwit inputs the one that is doing the transaction knows > which inputs are segwit or not, then parsing the transaction you can > associate the correct input to the correct witness data, without the > need of 00, so I must be missing the use case
I did not phrase correctly in fact, what I meant is: if the validator sees empty or witness script in scriptSig, then this is a segwit input, and doing this one by one the validator can associate the correct segwit data to the correct segwit input, so 00 does not look to be needed
Le 26/05/2019 à 18:28, Johnson Lau a écrit :
Quoted text (12 lines)
> This is not how it works. While the transaction creator may know which inputs are segwit, the validators have no way to tell until they look up the UTXO set. > > In a transaction, all information about an input the validators have is the 36-byte outpoint (txid + index). Just by looking at the outpoint, there is no way to tell whether it is segwit-enabled or not. So there needs to be a way to tell the validator that “the witness for this input is empty”, and it is the “00”. > >> On 27 May 2019, at 12:18 AM, Aymeric Vitte <vitteaymeric@gmail•com> wrote: >> >> ……. for the 00 number of witness >> data for non segwit inputs the one that is doing the transaction knows >> which inputs are segwit or not, then parsing the transaction you can >> associate the correct input to the correct witness data, without the >> need of 00, so I must be missing the use case >
Empty scriptSig doesn’t imply segwit input: if the previous scriptPubKey is OP_1 (which does not allow witness), it could still be spent with an empty scriptSig
Similarly, a scriptSig looking like a spend of P2SH-segwit doesn’t imply segwit input: if the previous scriptPubKey is empty, it could be spent with a push of any non-zero value.
Quoted text (20 lines)
> On 27 May 2019, at 1:09 AM, Aymeric Vitte <vitteaymeric@gmail•com> wrote: > > I did not phrase correctly in fact, what I meant is: if the validator > sees empty or witness script in scriptSig, then this is a segwit input, > and doing this one by one the validator can associate the correct segwit > data to the correct segwit input, so 00 does not look to be needed > > Le 26/05/2019 à 18:28, Johnson Lau a écrit : >> This is not how it works. While the transaction creator may know which inputs are segwit, the validators have no way to tell until they look up the UTXO set. >> >> In a transaction, all information about an input the validators have is the 36-byte outpoint (txid + index). Just by looking at the outpoint, there is no way to tell whether it is segwit-enabled or not. So there needs to be a way to tell the validator that “the witness for this input is empty”, and it is the “00”. >> >>> On 27 May 2019, at 12:18 AM, Aymeric Vitte <vitteaymeric@gmail•com> wrote: >>> >>> ……. for the 00 number of witness >>> data for non segwit inputs the one that is doing the transaction knows >>> which inputs are segwit or not, then parsing the transaction you can >>> associate the correct input to the correct witness data, without the >>> need of 00, so I must be missing the use case >>
On Sun, May 26, 2019, 07:07 Aymeric Vitte via bitcoin-dev < bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (13 lines)
> I realized recently that my segwit implementation was not correct, > basically some time ago, wrongly reading the specs (and misleaded by > what follows), I thought that scriptsig would go into witness data as it > was, but that's not the case, op_pushdata is replaced by varlen > > Now reading correctly the specs, they seem to be not totally correct, > then the first question is: why OP_0 is 00 in witness data and not 0100? > Does this apply to other op_codes? This does not look logical at all > > The second question is: why for non segwit inputs there is a 00 length > in segwit data, what is the rational for that? It should just be nothing > since you don't need this to reconciliate things >
This is a question that belongs on https://bitcoin.stackexchange.com, not this list.
Cheers,
The scriptSig when evaluated populates the stack so opcodes can operate on them. A witness is essentially a list of data elements, quite similar to the script stack (the witness is passed in as the script stack in fact)
OP_0 when evaluated pushes a _zero length_ value onto the stack, hence the 00 (the varlen) in the witness serialization. OP_1 (51 in decimal) pushes 0x01 to the stack, so when serialized would be 0101.
It may help to consider the entire witness structure as vector<vector<data element>> and it's length must equal the number of inputs - so a non-segwit input must have a zero sized witness.
On 5/26/19 12:56 AM, Aymeric Vitte via bitcoin-dev wrote:
Quoted text (17 lines)
> I realized recently that my segwit implementation was not correct, > basically some time ago, wrongly reading the specs (and misleaded by > what follows), I thought that scriptsig would go into witness data as it > was, but that's not the case, op_pushdata is replaced by varlen > > Now reading correctly the specs, they seem to be not totally correct, > then the first question is: why OP_0 is 00 in witness data and not 0100? > Does this apply to other op_codes? This does not look logical at all > > The second question is: why for non segwit inputs there is a 00 length > in segwit data, what is the rational for that? It should just be nothing > since you don't need this to reconciliate things > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Well, OK, then back to non standard stuff and bitcoin considers that an OP_1 or empty scriptpubkey is something that can exist, sipa does not like my questions on this list but this is a bit frightening in fact to see that after 10 years an OP_1 scriptpubkey or empty one can be a use case, thanks Thomas also, where all of this is clearly documented so people don't bother the list not to produce approximative stuff remains mysterious
Le 26/05/2019 à 19:24, Johnson Lau a écrit :
Quoted text (24 lines)
> Empty scriptSig doesn’t imply segwit input: if the previous scriptPubKey is OP_1 (which does not allow witness), it could still be spent with an empty scriptSig > > Similarly, a scriptSig looking like a spend of P2SH-segwit doesn’t imply segwit input: if the previous scriptPubKey is empty, it could be spent with a push of any non-zero value. > >> On 27 May 2019, at 1:09 AM, Aymeric Vitte <vitteaymeric@gmail•com> wrote: >> >> I did not phrase correctly in fact, what I meant is: if the validator >> sees empty or witness script in scriptSig, then this is a segwit input, >> and doing this one by one the validator can associate the correct segwit >> data to the correct segwit input, so 00 does not look to be needed >> >> Le 26/05/2019 à 18:28, Johnson Lau a écrit : >>> This is not how it works. While the transaction creator may know which inputs are segwit, the validators have no way to tell until they look up the UTXO set. >>> >>> In a transaction, all information about an input the validators have is the 36-byte outpoint (txid + index). Just by looking at the outpoint, there is no way to tell whether it is segwit-enabled or not. So there needs to be a way to tell the validator that “the witness for this input is empty”, and it is the “00”. >>> >>>> On 27 May 2019, at 12:18 AM, Aymeric Vitte <vitteaymeric@gmail•com> wrote: >>>> >>>> ……. for the 00 number of witness >>>> data for non segwit inputs the one that is doing the transaction knows >>>> which inputs are segwit or not, then parsing the transaction you can >>>> associate the correct input to the correct witness data, without the >>>> need of 00, so I must be missing the use case >
On Sun, May 26, 2019 at 5:07 PM Aymeric Vitte via bitcoin-dev < bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (10 lines)
> I realized recently that my segwit implementation was not correct, > basically some time ago, wrongly reading the specs (and misleaded by > what follows), I thought that scriptsig would go into witness data as it > was, but that's not the case, op_pushdata is replaced by varlen > > Now reading correctly the specs, they seem to be not totally correct, > then the first question is: why OP_0 is 00 in witness data and not 0100? > Does this apply to other op_codes? This does not look logical at all > >
op_pushdata allows for unsigned integers and thus it uses the extra byte to specify that.
A varint (varlen) is unsigned. 0 and anything <= 252 is one byte length, so 00.
Quoted text (5 lines)
> The second question is: why for non segwit inputs there is a 00 length > in segwit data, what is the rational for that? It should just be nothing > since you don't need this to reconciliate things > >
Because you don't have the number of witnesses as you have for inputs/outputs. The witness data length is assumed to be the same as the inputs length. And then for non-segwit inputs you need the 00 to specify that it is empty.
Hope that helps.