Requesting BIP assignment; Flexible Transactions.

As the title suggests, I would like to formally request the assignment of a BIP number for my FT spec.

Thank you!

Source;

https://github.com/zander/bips/blob/FlexTrans/bip-9999.mediawiki

<pre>
  BIP: ??
  Title: Flexible Transactions
  Author: Tom Zander <tomz@freedommail•ch>
  Status: Draft
  Type: Standards Track
  Created: 2016-07-27
</pre>

==Abstract==

This BIP describes the next step in making Bitcoin's most basic element, the transaction, more flexible and easier to extend. At the same time this fixes all known cases of malleability and resolves significant amounts of technical debt.

==Summary==

Flexible Transactions uses the fact that the first 4 bytes in a transaction determine the version and that the majority of the clients use a non-consensus rule (a policy) to not accept transaction version numbers other than those specifically defined by Bitcoin. This BIP chooses a new version number, 4, and defines that the data following the bytes for the version is in a format called Compact Message Format (CMF). CMF is a flexible, token based format where each token is a combination of a name, a format and a value. Because the name is added we can skip unused tokens and we can freely add new tokens in a simple manner in future. Soft fork upgrades will become much easier and cleaner this way.

This protocol upgrade cleans up past soft fork changes like BIP68 which reuse existing fields and do them in a much better to maintain and easier to parse system. It creates the building blocks to allow new features to be added much cleaner in the future.

It also shows to be possible to remove signatures from transactions with minimal upgrades of software and still maintain a coherent transaction history. Tests show that this can reduce space usage to about 75%.

==Motivation==

Token based file-formats are not new, systems like XML and HTMl use a similar system to allow future growth and they have been quite successful for decades in part because of this property.

Bitcoin needs a similar way of making the transaction future-proof because re-purposing not used fields for new features is not good for creating maintainable code.

Next to that this protocol upgrade will re-order the data-fields which allows us to cleanly fix the malleability issue which means that future technologies like Lightning Network will depend on this BIP being deployed.

At the same time, due to this re-ordering of data fields, it becomes very easy to remove signatures from a transaction without breaking its tx-id, which is great for future pruning features.

=== Tokens ===

In the compact message format we define tokens and in this specification we define how these tokens are named, where they can be placed and which are optional. To refer to XML, this specification would be the schema of a transaction.

CMF tokens are triplets of name, format (like PositiveInteger) and value. Names in this scope are defined much like an enumeration where the actual integer value (id, below) is equally important to the written name. If any token found that is not covered in the next table will make the transaction that contains it invalid.

{| class="wikitable"
|-
! Name !! id !! Format !! Default Value !! Description
|-
|TxEnd         ||  0 ||BoolTrue ||  Required    ||A marker that is the last
byte in the txid calculation
|-
|TxInPrevHash  ||  1 ||ByteArray||  Required    ||TxId we are spending
|-
|TxPrevIndex   ||  2 ||Integer  ||      0       ||Index in prev tx we are
spending (applied to previous TxInPrevHash)
|-
|TxInScript    ||  3 ||ByteArray||  Required    ||The 'input' part of the
script
|-
|TxOutValue    ||  4 ||Integer  ||  Required    ||Amount of satoshi to
transfer
|-
|TxOutScript   ||  5 ||ByteArray||  Required    ||The 'output' part of the
script
|-
|LockByBlock   ||  6 ||Integer  ||  Optional    ||BIP68 replacement
|-
|LockByTime    ||  7 ||Integer  ||  Optional    ||BIP68 replacement
|-
|ScriptVersion ||  8 ||Integer  ||      2       ||Defines script version for
outputs following
|-
|NOP_1x        || 1x || . ||  Optional    ||Values that will be ignored by
anyone parsing the transaction
|}

=== Scripting changes ===

In the current version of Bitcoin-script, version 1, there are various opcodes that are used to validate the cryptographic proofs that users have to provide in order to spend outputs.

The OP_CHECKSIG is the most well known and, as its name implies, it validates a signature. In the new version of 'script' (version 2) the data that is signed is changed to be equivalent to the transaction-id. This is a massive simplification and also the only change between version 1 and version 2 of script.

=== Serialization order===

The tokens defined above have to be serialized in a certain order for the transaction to be well-formatted. Not serializing transactions in the order specified would allow multiple interpretations of the data which can't be allowed. There is still some flexibility and for that reason it is important for implementors to remember that the actual serialized data is used for the calculation of the transaction-id. Reading and writing it may give you a different output and when the txid changes, the signatures will break.

At a macro-level the transaction has these segments. The order of the segments can not be changed, but you can skip segments.

{| class="wikitable"
!Segment !! Description
|-
|   Inputs   || Details about inputs.
|-
|  Outputs   || Details and scripts for outputs
|-
| Additional || For future expansion
|-
| Signatures || The scripts for the inputs
|-
|   TxEnd    || End of the transaction
|}

The TxId is calculated by taking the serialized transaction without the Signatures and the TxEnd and hashing that.

{| class="wikitable"
!Segment !! Tags !! Description
|-
|Inputs||TxInPrevHash and TxInPrevIndex||Index can be skipped, but in any
input the PrevHash always has to come first
|-
|Outputs||TxOutScript, TxOutValue||Order is not relevant
|-
|Additional||LockByBlock  LockByTime NOP_1x
|-
|Signatures||TxInScript||Exactly the same amount as there are inputs
|-
|TxEnd||TxEnd
|}

TxEnd is there to allow a parser to know when one transaction in a stream has ended, allowing the next to be parsed.

Notice that the token ScriptVersion is currently not allowed because we don't have any valid value to give it. But if we introduce a new script version it would be placed in the outputs segment.

=== Script v2 ===

The default value of ScriptVersion is number 2, as opposed to the version 1 of script that the is in use today. The version 2 is mostly identical to version one, including upgrades made to it over the years and in the future. The only exception is that the OP_CHECKSIG is made dramatically simpler. The input-type for OP_CHECKSIG is now no longer configurable, it is always '1' and the content that will be signed is the txid.

TODO: does check-multisig need its own mention?

=== Block-malleability ===

The effect of leaving the signatures out of the calculation of the transaction-id implies that the signatures are also not used for the calculation of the merkle tree. This means that changes in signatures would not be detectable. Except naturally by the fact that missing or broken signatures breaks full validation. But it is important to detect modifications to such signatures outside of validating all transactions.

For this reason the merkle tree is extended to include (append) the hash of the v4 transactions (and those alone) where the hash is taken over a data-blob that is build up from:

1. the tx-id
2. the CMF-tokens 'TxInScript'

=== Future extensibility ===

The NOP_1x wildcard used in the table explaining tokens is actually a list of 10 values that currently are specified as NOP (no-operation) tags.

Any implementation that supports the v4 transaction format should ignore this field in a transaction. Interpreting and using the transaction as if that field was not present at all.

Future software may use these fields to decorate a transaction with additional data or features. Transaction generating software should not trivially use these tokens for their own usage without cooperation and communication with the rest of the Bitcoin ecosystem as miners certainly have the option to reject transactions that use unknown-to-them tokens.

==Reference Implementation==

Bitcoin Classic includes this in its beta releases and a reference implementation can be found at;

https://github.com/bitcoinclassic/bitcoinclassic/pull/186

==Deployment==

To be determined

==References==

[https://github.com/bitcoinclassic/documentation/blob/master/spec/compactmessageformat.md] CMF

On Tuesday, September 20, 2016 5:15:45 PM Tom via bitcoin-dev wrote:

> As the title suggests, I would like to formally request the assignment of a
> BIP number for my FT spec.

Please open a pull request on the bitcoin/bips repo after this has been discussed a bit on the ML.

Note that at least a basic backwards compatibility section is required for assignment, especially since this appears to be a hard-fork proposal (if it is meant as a soft-fork, please explain how that would work).

> ==Motivation==
>
> Token based file-formats are not new, systems like XML and HTMl use a
> similar system to allow future growth and they have been quite successful
> for decades in part because of this property.

There is already a binary-safe format called EBML. Why not use that as the basis for this BIP?

> Next to that this protocol upgrade will re-order the data-fields which
> allows us to cleanly fix the malleability issue which means that future
> technologies like Lightning Network will depend on this BIP being deployed.

Note this won't fix malleability entirely, only third-party malleability (much like segwit). The sender can always modify the transaction with more inputs/outputs.

> |TxEnd         ||  0 ||BoolTrue ||  Required    ||A marker that is the last
> byte in the txid calculation

The last byte in the txid calculation, or the last byte of the entire transaction?

It seems from the later comments, that it is the end of the transaction as a whole. Yet a separator between the txid and non-txid data would probably be valuable, rather than hard-coding txid to skip signature types (which may be unknown to old nodes, when extended).

> The OP_CHECKSIG is the most well known and, as its name implies, it
> validates a signature.
> In the new version of 'script' (version 2) the data that is signed is
> changed to be equivalent to the transaction-id. This is a massive
> simplification and also the only change between version 1 and version 2 of
> script.

This seems to be a major regression. What is the replacement for SIGHASH_SINGLE and SIGHASH_ANYONECANPAY?

When revising OP_CHECKSIG, it would also be nice to add the ability to use *only* a hash of the prevout's scriptPubKey in the input, so that *when* the prevtx is malleated, the spending one remains valid. (This use case is currently not supported.)

> === Serialization order===
>
> The tokens defined above have to be serialized in a certain order for the
> transaction to be well-formatted.  Not serializing transactions in the
> order specified would allow multiple interpretations of the data which
> can't be allowed.

What happens if nodes encounter a different order in a block?

> Notice that the token ScriptVersion is currently not allowed because we
> don't have any valid value to give it. But if we introduce a new script
> version it would be placed in the outputs segment.

What happens if I put ScriptVersion=1 here?

> === Script v2 ===
>
> TODO: does check-multisig need its own mention?

Does multisig still require a dummy item on the stack?

> === Block-malleability ===
>
> For this reason the merkle tree is extended to include (append) the hash of
> the v4 transactions (and those alone) where the hash is taken over a
> data-blob that is build up from:

How should nodes know where in the merkle-tree the txids end, and the v4hashes begin?

Luke

On Tue, Sep 20, 2016 at 07:15:45PM +0200, Tom via bitcoin-dev wrote:

> === Serialization order===
>
> The tokens defined above have to be serialized in a certain order for the
> transaction to be well-formatted.  Not serializing transactions in the
> order specified would allow multiple interpretations of the data which
> can't be allowed.

If the order of the tokens is fixed, the tokens themselves are redundant information when tokens are required; when tokens may be omitted, a simple "Some/None" flag to mark whether or not the optional data has been omitted is appropriate.

Also, if you're going to break compatibility with all existing software, it makes sense to use a format that extends the merkle tree down into the transaction inputs and outputs.

On Tuesday 20 Sep 2016 21:31:47 Luke Dashjr wrote:

> On Tuesday, September 20, 2016 5:15:45 PM Tom via bitcoin-dev wrote:
> > As the title suggests, I would like to formally request the assignment of
> > a
> > BIP number for my FT spec.
>
> Please open a pull request on the bitcoin/bips repo after this has been
> discussed a bit on the ML.

> It seems from the later comments, that it is the end of the transaction as a
> whole. Yet a separator between the txid and non-txid data would probably be
> valuable, rather than hard-coding txid to skip signature types (which may
> be unknown to old nodes, when extended).
>
> > The OP_CHECKSIG is the most well known and, as its name implies, it
> > validates a signature.
> > In the new version of 'script' (version 2) the data that is signed is
> > changed to be equivalent to the transaction-id. This is a massive
> > simplification and also the only change between version 1 and version 2 of
> > script.
>
> This seems to be a major regression. What is the replacement for
> SIGHASH_SINGLE and SIGHASH_ANYONECANPAY?

How is this a regression? Can you explain what functionality is lost please?

> When revising OP_CHECKSIG, it would also be nice to add the ability to use
> *only* a hash of the prevout's scriptPubKey in the input, so that *when* the
> prevtx is malleated, the spending one remains valid. (This use case is
> currently not supported.)

Maybe for the next version of script :)

> > Notice that the token ScriptVersion is currently not allowed
> What happens if I put ScriptVersion=1 here?

The transaction is invalid...

> > === Block-malleability ===
> >
> > For this reason the merkle tree is extended to include (append) the hash
> > of
> > the v4 transactions (and those alone) where the hash is taken over a
>
> > data-blob that is build up from:
>
> How should nodes know where in the merkle-tree the txids end, and the
> v4hashes begin?

Because the txid based ones are not going away. So the number of transactions in the block can be used to determine when the pure tx-id segment stops and when the v4 hashes begin. Then its up to the client to rebuild the tree from that list based on the larger input set to get the same root-node.

I clarified many little things on my clone of the bips, check there if you want to see the details.

Thanks for your email Peter!

On Tuesday 20 Sep 2016 17:56:44 Peter Todd wrote:

> On Tue, Sep 20, 2016 at 07:15:45PM +0200, Tom via bitcoin-dev wrote:
> > === Serialization order===
> >
> > The tokens defined above have to be serialized in a certain order for the
> > transaction to be well-formatted.  Not serializing transactions in the
> > order specified would allow multiple interpretations of the data which
> > can't be allowed.
>
> If the order of the tokens is fixed, the tokens themselves are redundant
> information when tokens are required; when tokens may be omitted, a simple
> "Some/None" flag to mark whether or not the optional data has been omitted
> is appropriate.

This is addressed in the spec; https://github.com/bitcoinclassic/documentation/blob/master/spec/transactionv4.md

«The way towards that flexibility is to use a generic concept made popular various decades ago with the XML format. The idea is that we give each field a name and this means that new fields can be added or optional fields can be omitted from individual transactions»

> Also, if you're going to break compatibility with all existing software, it
> makes sense to use a format that extends the merkle tree down into the
> transaction inputs and outputs.

Please argue your case.

Just glancing over your BIP, I wonder if we should use Protobuf. It uses this "flexible" format already and is quite compact/binary. We use Protobuf already for the payment protocol, and there is very good tool support.

On Wednesday 21 Sep 2016 14:00:23 Andreas Schildbach via bitcoin-dev wrote:

> Just glancing over your BIP, I wonder if we should use Protobuf. It uses
> this "flexible" format already and is quite compact/binary. We use
> Protobuf already for the payment protocol, and there is very good tool
> support.

There is a lot of overlap between different binary formats. Looking through the on-the-wire protocol you'll see that my spec is very similar. Practically all the advantages of protobuf are present in CMF. I can write you a java parser if you want, it should be easy to port from Qt/C++ code :) https://github.com/bitcoinclassic/transactions

CMF: https://github.com/bitcoinclassic/documentation/blob/master/spec/compactmessageformat.md

There is no tool support needed, just one or two classes. Which personally I think is an advantage.

Some advantages of CMF over protobuf from the top of my head;

* It reuses the var-int parsing that Bitcoin uses (which is itself slightly different from others).

* zero-copy support (not relevant for this bip, though).

* Additional values addition (i.e. adding new data) is .. tricky in protobuf. https://developers.google.com/protocol-buffers/docs/proto#updating

* In my experience parsing a message manually (like a SOX parser) is much better in reporting errors and detecting wrong usages than auto-generated code (but personally I'm not much a fan of auto-generated APIs) at all...

* Generated parsing/writing code will not be as fast as we can make it.

* CMF is more compact (uses less bytes) for its messages.

Protobuf is something I've used before and I think we can do better. I think that CMF together with some support classes can do this better.

On Tue, Sep 20, 2016 at 5:15 PM, Tom via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> wrote:

> BIP number for my FT spec.

This document does not appear to be concretely specified enough to review or implement from it.

For example, it does not specify the serialization of "integer" (is it a 32 bit word in network byte order or?) nor does it specify how the presence of the optional fields are signaled nor the cardinality of the inputs or outputs. For clearly variable length elements ('bytearray') no mention is made of their length encoding. etc.

Without information like this, I don't see how someone could realistically begin reviewing this proposal.

The motivation seems unclear to me as well: The scheme is described as 'flexible' but it appears to remove flexibility from the existing system. The "schema" appears to be hardcoded and never communicated. If the goal is to simply have a more compact on the wire representation, this could be done without changing the serialization used for hashing or the serialization used for costing.

On Wednesday 21 Sep 2016 18:01:30 Gregory Maxwell via bitcoin-dev wrote:

> On Tue, Sep 20, 2016 at 5:15 PM, Tom via bitcoin-dev
>
> <bitcoin-dev@lists•linuxfoundation•org> wrote:
> > BIP number for my FT spec.
>
> This document does not appear to be concretely specified enough to
> review or implement from it.
>
> For example, it does not specify the serialization of "integer"

It refers to the external specification which is linked at the bottom. In that spec you'll see that "Integer" is the standard var-int that Bitcoin has used for years.

> nor does it specify how the
> presence of the optional fields are signaled

How does one signals an optional field except of in the spec? Thats the job of a specification.

> nor the cardinality of
> the inputs or outputs.

Did you miss this in the 3rd table ? I suggest clicking on the github bips repo link as tables are not easy to read in mediawiki plain format that the email contained.

> For clearly variable length elements
> ('bytearray') no mention is made of their length encoding. etc.

Also in the external CMF spec.

> Without information like this, I don't see how someone could
> realistically begin reviewing this proposal.

I agree, that would be bad. Luckily that you just missed the link :) Here it is; https://github.com/bitcoinclassic/documentation/blob/master/spec/compactmessageformat.md

> The motivation seems unclear to me as well: The scheme is described as
> 'flexible' but it appears to remove flexibility from the existing
> system. The "schema" appears to be hardcoded and never communicated.

Being hardcoded and never communicated is what the current format does to. How does that "remove flexibility".

Also read my reply to Peter Todd on why this is flexible.

> If the goal is to simply have {snip}

It is not.

Thanks for asking, I understand that the CMF spec is useful to see as well. Hopefully you can now review it properly since I linked to it above.

Cheers!

On Thu, Sep 22, 2016 at 10:56:31AM +0200, Tom via bitcoin-dev wrote:

> On Wednesday 21 Sep 2016 18:01:30 Gregory Maxwell via bitcoin-dev wrote:
> > On Tue, Sep 20, 2016 at 5:15 PM, Tom via bitcoin-dev
> >
> > <bitcoin-dev@lists•linuxfoundation•org> wrote:
> > > BIP number for my FT spec.
> >
> > This document does not appear to be concretely specified enough to
> > review or implement from it.
> >
> > For example, it does not specify the serialization of "integer"
>
> It refers to the external specification which is linked at the bottom.
> In that spec you'll see that "Integer" is the standard var-int that Bitcoin
> has used for years.

I think BIPs should be self-contained, or rely on previous BIPs, whenever possible. Referencing an external formatting document should be avoided and requiring readers to reverse engineer a reference implementation doesn't seem too user friendly either. Publishing a BIP with CMF would certainly help, and completing this spec with the details that are missing, or only "defined" in the implementation, would be better.

> > nor does it specify how the
> > presence of the optional fields are signaled
>
> How does one signals an optional field except of in the spec? Thats the job of
> a specification.

So the presence is signaled by encountering the tag, which contains both token type and name-reference. The encoder and decoder operations could be described better.

> > nor the cardinality of
> > the inputs or outputs.
>
> Did you miss this in the 3rd table ?  I suggest clicking on the github bips
> repo link as tables are not easy to read in mediawiki plain format that the
> email contained.

Minor nit: that table is not well-formed. As was pointed out in the normalized transaction ID BIP, your proposal only addresses third-party malleability, since signers can simply change the transaction and re-sign it. This is evident from the fact that inputs and outputs do not have a canonical order and it would appear that tokens can be re-ordered in segments. Dependencies of tokens inside a segment are also rather alarming (TxInPrevHash <-> TxInPrevIndex, TxOutScript <-> TxOutValue).

Finally, allowing miners to reject transactions with unknown fields makes the OP_NOPs unusable since they'd result in forks: non-upgraded nodes would reject blocks from upgraded nodes.

Regards, Christian

On Thursday 22 Sep 2016 13:10:49 Christian Decker via bitcoin-dev wrote:

> On Thu, Sep 22, 2016 at 10:56:31AM +0200, Tom via bitcoin-dev wrote:
> > On Wednesday 21 Sep 2016 18:01:30 Gregory Maxwell via bitcoin-dev wrote:
> > > On Tue, Sep 20, 2016 at 5:15 PM, Tom via bitcoin-dev
> > >
> > > <bitcoin-dev@lists•linuxfoundation•org> wrote:
> > > > BIP number for my FT spec.
> > >
> > > This document does not appear to be concretely specified enough to
> > > review or implement from it.
> > >
> > > For example, it does not specify the serialization of "integer"
> >
> > It refers to the external specification which is linked at the bottom.
> > In that spec you'll see that "Integer" is the standard var-int that
> > Bitcoin
> > has used for years.
>
> I think BIPs should be self-contained, or rely on previous BIPs,
> whenever possible. Referencing an external formatting document should
> be avoided

If luke-jr thinks I should submit CMF as a BIP, I can certainly do that. Luke, what do you think?

I don't have a preference either way.

> > > nor does it specify how the
> > > presence of the optional fields are signaled
> >
> > How does one signals an optional field except of in the spec? Thats the
> > job of a specification.
>
> So the presence is signaled by encountering the tag, which contains
> both token type and name-reference. The encoder and decoder operations
> could be described better.

I'm sorry, I'm not following you here. Is there a question?

> > > nor the cardinality of
> > > the inputs or outputs.
> >
> > Did you miss this in the 3rd table ?  I suggest clicking on the github
> > bips
> > repo link as tables are not easy to read in mediawiki plain format that
> > the
> > email contained.
>
> Minor nit: that table is not well-formed.

I am not very well versed in mediawiki tables, and I found github has some incompatibilities too. The markdown one looks better; https://github.com/bitcoinclassic/documentation/blob/master/spec/transactionv4.md

> As was pointed out in the
> normalized transaction ID BIP, your proposal only addresses
> third-party malleability, since signers can simply change the
> transaction and re-sign it.

I have to disagree. That is not malleability. Creating a new document and re- signing it is not changing anything. Its re-creating. Something that the owner of the coin has every right to do.

> This is evident from the fact that inputs
> and outputs do not have a canonical order and it would appear that
> tokens can be re-ordered in segments.

Sorry, what is evident? You seem to imply that it is uncommon that you can create two transactions of similar intent but using different bytes. You would be wrong with this implication as this is very common. You can just alter the order of the inputs, for instance.

I am unable to see what the point is you are trying to make. Is there a question or a suggestion for improvement here?

> Dependencies of tokens inside a
> segment are also rather alarming (TxInPrevHash <-> TxInPrevIndex,
> TxOutScript <-> TxOutValue).

Maybe you missed this line;
  «TxInPrevHash and TxInPrevIndex
   Index can be skipped, but in any input the PrevHash always has
   to come first»

If you still see something alarming, let me know. You can look at the code in Bitcoin Classic and notice that it really isn't anything complicated or worrying.

> Finally, allowing miners to reject transactions with unknown fields
> makes the OP_NOPs unusable

Hmm, it looks like you are mixing terminology and abstraction-levels. OP_NOP is a field from script and there is no discussion about any rejection based on script in this BIP at all.

Rejection of transactions is done on there being unrecognised tokens in the transaction formatting itself.

Thank you for your email to my BIP, I hope you got the answers you were looking for :)

On Wed, Sep 21, 2016 at 11:32:33AM +0200, Tom wrote:

> Thanks for your email Peter!
>
> On Tuesday 20 Sep 2016 17:56:44 Peter Todd wrote:
> > On Tue, Sep 20, 2016 at 07:15:45PM +0200, Tom via bitcoin-dev wrote:
> > > === Serialization order===
> > >
> > > The tokens defined above have to be serialized in a certain order for the
> > > transaction to be well-formatted.  Not serializing transactions in the
> > > order specified would allow multiple interpretations of the data which
> > > can't be allowed.
> >
> > If the order of the tokens is fixed, the tokens themselves are redundant
> > information when tokens are required; when tokens may be omitted, a simple
> > "Some/None" flag to mark whether or not the optional data has been omitted
> > is appropriate.
>
> This is addressed in the spec;
> https://github.com/bitcoinclassic/documentation/blob/master/spec/transactionv4.md
>
> «The way towards that flexibility is to use a generic concept made popular
> various decades ago with the XML format. The idea is that we give each
> field a name and this means that new fields can be added or optional fields
> can be omitted from individual transactions»

That argument is not applicable to required fields: the code to get the fields from the extensible format is every bit as complex as the very simple code required to deserialize/serialize objects in the current system.

In any case your BIP needs to give some explicit examples of hypothetical upgrades in the future, how they'd take advantage of this, and what the code to do so would look like.

> > Also, if you're going to break compatibility with all existing software, it
> > makes sense to use a format that extends the merkle tree down into the
> > transaction inputs and outputs.
>
> Please argue your case.

See my arguments re: segwit a few months ago, e.g. the hardware wallet txin proof use-case.

On Thursday, 22 September 2016 14:26:18 CEST Peter Todd wrote:

> > «The way towards that flexibility is to use a generic concept made
> > popular various decades ago with the XML format. The idea is that we
> > give each field a name and this means that new fields can be added or
> > optional fields can be omitted from individual transactions»
>
> That argument is not applicable to required fields:

The argument that optional fields can be omitted is not applicable to required fields, you are correct. That should be rather obvious because required fields are not optional fields.

> the code to get the
> fields from the extensible format is every bit as complex as the very
> simple code required to deserialize/serialize objects in the current
> system.

Probably a tiny bit more complex as the current format assumes a lot more.

You may have misread my email because there was no argument made towards complexity. The argument was towards flexibility.

> In any case your BIP needs to give some explicit examples of hypothetical
> upgrades in the future, how they'd take advantage of this, and what the
> code to do so would look like.

Why?

> > > Also, if you're going to break compatibility with all existing
> > > software, it makes sense to use a format that extends the merkle
> > > tree down into the transaction inputs and outputs.
> >
> > Please argue your case.
>
> See my arguments re: segwit a few months ago, e.g. the hardware wallet
> txin proof use-case.

Please consider that I'm not going to search for something based on a vague reference like that, if you want to convince me you could you at least provide a URL? You want me to see the value of your idea, I think you should at least provide the argument. Isn't that fair?

Thanks for your email Peter, would love you to put a bit more time into understanding flexible transactions and we can have a proper discussion about it.

On Thu, Sep 22, 2016 at 02:09:38PM +0200, Tom via bitcoin-dev wrote:

> On Thursday 22 Sep 2016 13:10:49 Christian Decker via bitcoin-dev wrote:
> >
> > I think BIPs should be self-contained, or rely on previous BIPs,
> > whenever possible. Referencing an external formatting document should
> > be avoided
>
> If luke-jr thinks I should submit CMF as a BIP, I can certainly do that.
> Luke, what do you think?
>
> I don't have a preference either way.
>
> >
> > So the presence is signaled by encountering the tag, which contains
> > both token type and name-reference. The encoder and decoder operations
> > could be described better.
>
> I'm sorry, I'm not following you here. Is there a question?

Nope, just clarifying how presence or absence is indicated :-)

> >
> > Minor nit: that table is not well-formed.
>
> I am not very well versed in mediawiki tables, and I found github has some
> incompatibilities too.
> The markdown one looks better;
> https://github.com/bitcoinclassic/documentation/blob/master/spec/transactionv4.md

It's just some rows have 3 columns, others have 2. It's a minor nit really.

> > As was pointed out in the
> > normalized transaction ID BIP, your proposal only addresses
> > third-party malleability, since signers can simply change the
> > transaction and re-sign it.
>
> I have to disagree. That is not malleability. Creating a new document and re-
> signing it is not changing anything. Its re-creating. Something that the owner
> of the coin has every right to do.

Same thing I was arguing back then, however Luke pointed out that malleability just refers to the possibility of modifying a transaction after the fact. Always referring to "third-party malleability" avoids this ambiguity.

> > This is evident from the fact that inputs
> > and outputs do not have a canonical order and it would appear that
> > tokens can be re-ordered in segments.
>
> Sorry, what is evident? You seem to imply that it is uncommon that you can
> create two transactions of similar intent but using different bytes.
> You would be wrong with this implication as this is very common. You can just
> alter the order of the inputs, for instance.
>
> I am unable to see what the point is you are trying to make. Is there a
> question or a suggestion for improvement here?
>
> > Dependencies of tokens inside a
> > segment are also rather alarming (TxInPrevHash <-> TxInPrevIndex,
> > TxOutScript <-> TxOutValue).
>
> Maybe you missed this line;
>   «TxInPrevHash and TxInPrevIndex
>    Index can be skipped, but in any input the PrevHash always has
>    to come first»

Nope, that is exactly the kind of dependency I was talking about. Instead of nesting a construct like the current transactions do, you rely on the order of tokens to imply that they belong together.

> If you still see something alarming, let me know.
> You can look at the code in Bitcoin Classic and notice that it really isn't
> anything complicated or worrying.
>
>
> > Finally, allowing miners to reject transactions with unknown fields
> > makes the OP_NOPs unusable
>
> Hmm, it looks like you are mixing terminology and abstraction-levels.  OP_NOP
> is a field from script and there is no discussion about any rejection based on
> script in this BIP at all.
>
> Rejection of transactions is done on there being unrecognised tokens in the
> transaction formatting itself.

Ah, thanks for clearing that up. However, the problem persists, if we add new fields that a non-upgraded node doesn't know about and it rejects transactions containing it, we'll have a hard-fork. It should probably not reject transactions with unknown fields if the transaction is included in a block.

> Thank you for your email to my BIP, I hope you got the answers you were
> looking for :)

Cheers, Christian

On Friday, 23 September 2016 13:42:36 CEST Christian Decker via bitcoin-dev wrote:

> > I have to disagree. That is not malleability. Creating a new document
> > and re- signing it is not changing anything. Its re-creating.
> > Something that the owner of the coin has every right to do.
> Same thing I was arguing back then, however Luke pointed out that
> malleability just refers to the possibility of modifying a transaction
> after the fact.

I am not a fan of redefining dictionary words. I'll stick to the universally excepted one, thanks.

> Nope, that is exactly the kind of dependency I was talking
> about. Instead of nesting a construct like the current transactions
> do, you rely on the order of tokens to imply that they belong
> together.

> if we
> add new fields that a non-upgraded node doesn't know about and it
> rejects transactions containing it, we'll have a hard-fork. It should
> probably not reject transactions with unknown fields if the
> transaction is included in a block.

This is addressed here; https://github.com/bitcoin/bips/blob/master/bip-0134.mediawiki#future-extensibility