More precise type information in API reference

Dear Bitcoin devs,

I am the author of OCaml-bitcoin [1], a library offering an OCaml interface to the official Bitcoin client API. For those who may be unfamiliar with it, OCaml is one of those functional programming languages with a very rich and expressive type system [2]. Given its emphasis on safety, its industrial users are disproportionally found in the aerospace and financial sectors.

Now, OCaml programmers care a lot about types, because experience has taught them that deep down most programming errors are just type errors. From this stems my request: please consider defining more precisely the type information associated with each API call in the JSON-RPC reference [3].

To give you a better idea of what I'm talking about, please take a look at the API offered by OCaml-bitcoin [4], and the associated type definitions [5] (note that these have not been updated for Bitcoin Core 0.10 yet). I've created the type definitions from information gathered from the Bitcoin wiki and from looking at the Bitcoin Core source-code. I wouldn't be surprised if it contains errors, because neither the source-code nor the wiki is very precise about the actual types being used. As an example, consider type hexspk_t ("hex representation of script public key"). Is this really the same type used in both signrawtransaction and createmultisig?

Improving this situation would pose a minimal burden on bitcoin devs: all that would be required is defining the precise set of types used in the RPC API, and annotating the RPC calls either in the source-code itself or in the API reference documentation. It would make writing bindings such as mine far easier and less error prone, and it would have the added advantage of better documenting the Bitcoin Core source-code itself.

Also, note that it is not necessary to extend this request to the deep data structures returned by some API calls. Consider for instance the gettransaction function of the OCaml-bitcoin API: it returns the raw JSON object without any attempt to process it. This is because that's a fairly niche facility, and the bindings would balloon in size if I were to process every single large return object. Instead, the bindings take the more pragmatic stance of only processing the parameters and return results where a strong type discipline is imperative.

When I raised this issue on IRC a number of questions were posed. What follows is my attempt to answer them:

   Q: What does it matter, if JSON only has a tiny set of types?

   A: JSON being the serialisation format is irrelevant.  The client
bindings
      know that even if a public ECDSA key is serialised as a string, it
does
      not stop being a public ECDSA key, and should only be used where a
public
      ECDSA key is expected.

   Q: What does it matter if the types are not even distinguished in the
C++
      source of Bitcoin Core?

   A: That is unfortunate, because it opens the door to bugs caused by
type
      errors.  Moreover, even if the C++ source is "stringly-typed" and
does
      not enforce a strong type discipline, that does not mean that the
types
      are not there.  Even if a public and private key are both
represented
      as strings, can you use one where the other is expected?  If not,
then
      they actually have different types!

   Q: Isn't this a maintenance nightmare, given the changes to Bitcoin
core?

   A: Actually, the most burdensome part is what motivated this message:
      keeping track of the types used.  If the Bitcoin API reference were
      more precise, keeping the bindings up-to-date would be trivial and
      even mechanical, because the API is now fairly stable.

Thank you very much for your attention, and for all the work you guys put into Bitcoin development. It is much appreciated and not acknowledged often enough!

Best regards, Dario Teixeira

[1] https://github.com/darioteixeira/ocaml-bitcoin
[2] http://ocaml.org/learn/description.html
[3] https://bitcoin.org/en/developer-reference#bitcoin-core-apis
[4] http://ocaml-bitcoin.forge.ocamlcore.org/apidoc/Bitcoin.ENGINE.html
[5] http://ocaml-bitcoin.forge.ocamlcore.org/apidoc/Bitcoin.html

Hi again,

My excuses: the formatting of the message seems to have been eaten by a Gremlin. Here's a second attempt:

=================================================================

Dear Bitcoin devs,

I am the author of OCaml-bitcoin [1], a library offering an OCaml interface to the official Bitcoin client API. For those who may be unfamiliar with it, OCaml is one of those functional programming languages with a very rich and expressive type system [2]. Given its emphasis on safety, its industrial users are disproportionally found in the aerospace and financial sectors.

Now, OCaml programmers care a lot about types, because experience has taught them that deep down most programming errors are just type errors. From this stems my request: please consider defining more precisely the type information associated with each API call in the JSON-RPC reference [3].

To give you a better idea of what I'm talking about, please take a look at the API offered by OCaml-bitcoin [4], and the associated type definitions [5] (note that these have not been updated for Bitcoin Core 0.10 yet). I've created the type definitions from information gathered from the Bitcoin wiki and from looking at the Bitcoin Core source-code. I wouldn't be surprised if it contains errors, because neither the source-code nor the wiki is very precise about the actual types being used. As an example, consider type hexspk_t ("hex representation of script public key"). Is this really the same type used in both signrawtransaction and createmultisig?

Improving this situation would pose a minimal burden on bitcoin devs: all that would be required is defining the precise set of types used in the RPC API, and annotating the RPC calls either in the source-code itself or in the API reference documentation. It would make writing bindings such as mine far easier and less error prone, and it would have the added advantage of better documenting the Bitcoin Core source-code itself.

Also, note that it is not necessary to extend this request to the deep data structures returned by some API calls. Consider for instance the gettransaction function of the OCaml-bitcoin API: it returns the raw JSON object without any attempt to process it. This is because that's a fairly niche facility, and the bindings would balloon in size if I were to process every single large return object. Instead, the bindings take the more pragmatic stance of only processing the parameters and return results where a strong type discipline is imperative.

When I raised this issue on IRC a number of questions were posed. What follows is my attempt to answer them:

   Q: What does it matter, if JSON only has a tiny set of types?

   A: JSON being the serialisation format is irrelevant.  The client
      bindings know that even if a public ECDSA key is serialised as
      a string, it does not stop being a public ECDSA key, and should
      only be used where a public ECDSA key is expected.

   Q: What does it matter if the types are not even distinguished in
      the C++ source of Bitcoin Core?

   A: That is unfortunate, because it opens the door to bugs caused by
      type errors.  Moreover, even if the C++ source is "stringly-typed"
      and does not enforce a strong type discipline, that does not mean
      that the types are not there.  Even if a public and private key
      are both represented as strings, can you use one where the other
      is expected?  If not, then they actually have different types!

   Q: Isn't this a maintenance nightmare, given the changes to Bitcoin
      core?

   A: Actually, the most burdensome part is what motivated this message:
      keeping track of the types used.  If the Bitcoin API reference were
      more precise, keeping the bindings up-to-date would be trivial and
      even mechanical, because the API is now fairly stable.

Thank you very much for your attention, and for all the work you guys put into Bitcoin development. It is much appreciated and not acknowledged often enough!

Best regards, Dario Teixeira

[1] https://github.com/darioteixeira/ocaml-bitcoin
[2] http://ocaml.org/learn/description.html
[3] https://bitcoin.org/en/developer-reference#bitcoin-core-apis
[4] http://ocaml-bitcoin.forge.ocamlcore.org/apidoc/Bitcoin.ENGINE.html
[5] http://ocaml-bitcoin.forge.ocamlcore.org/apidoc/Bitcoin.html

On Tue, Feb 17, 2015 at 01:33:28PM +0000, Dario Teixeira wrote:

> From this stems my request: please consider defining
> more precisely the type information associated with each API call
> in the JSON-RPC reference [3].

Hi Dario,

I'm the primary author of the Bitcoin.org JSON-RPC reference, and I'd be happy to help.

Do you think it would be possible for you to submit a minimal pull request against the docs adding the type information you need to just one RPC call? From there we can see how much work it would take to generalize that across all 100+ printed pages worth of RPC docs.

I've tried to make this as easy as possible: if you hover your mouse over the title of an RPC[1], an "Edit" link will appear that will take you to a page on GitHub to edit the file describing that RPC call. Or you can checkout[2] the website repository and edit the file locally; the individual RPCs are in _includes/ref/bitcoin-core/rpcs/rpcs/

    [1] For example, https://bitcoin.org/en/developer-reference#addmultisigaddress
    [2] https://github.com/bitcoin/bitcoin.org

If you have any questions about the editing process, or anything else, please feel free to email me at this address or PM harding on Freenode.

Thanks!,

-Dave

Hi Dave,

> I'm the primary author of the Bitcoin.org JSON-RPC reference, and I'd
> be happy to help.

Thanks -- it is much appreciated!

> Do you think it would be possible for you to submit a minimal pull
> request against the docs adding the type information you need to just
> one RPC call?  From there we can see how much work it would take to
> generalize that across all 100+ printed pages worth of RPC docs.

Sure, I would be glad to help. In fact, most of grunt work has already been done for the OCaml-bitcoin API [1,2], which could be used as a guide even if you don't use OCaml (beware that it may contain errors, though).

Besides tweaking each RPC call, we would need to add a new section to the docs, to be placed before the RPC calls. This List of Types section would list all the custom types defined in the API, providing a brief description for each and indicating the JSON type used for serialisation (almost invariantly "string", I reckon).

But before I submit a pull request, allow me to exemplify with AddMultiSigAddress:

Parameter #1: (no changes)

Parameter #2: I would keep "array" as the type of "Keys or Addresses". If we follow the convention of using a pipe character to represent unions, then the type for "Key or Address" is not "string", but "public_key | p2pkh_address". This would also require adding two entries to the List of Types section, describing public_key and p2pkh_address, and letting the users know that they are serialised as JSON strings. (Note that the precise type of "Keys or Addresses" is actually "Array [public_key | p2pkh_address]". However, since the type of each element is also listed, this info is redundant and we can safely state it's just an array.)

Parameter #3: The type should be "account", obviously.

Result: a p2sh_address. If there is only one type of such P2SH addresses and they can be used interchangeably, then this would suffice. If however, there are actually several incompatible kinds of P2SH address, then we'd need to be more precise about which one we mean.

Thanks again for your time! Kind regards, Dario Teixeira

[1] http://ocaml-bitcoin.forge.ocamlcore.org/apidoc/Bitcoin.ENGINE.html
[2] http://ocaml-bitcoin.forge.ocamlcore.org/apidoc/Bitcoin.html