Adding callback hooks to the satoshi client

Hey, guys.

I've been writing a number of apps that require realtime event notifications, where the JSON-RPC API clearly doesn't suffice.

There are two approaches I've been taking to this end:

1) Writing my own library for dealing with raw bitcoin structures and connecting to bitcoin nodes via the bitcoin protocol. 2) Making custom builds of the satoshi client putting callback hooks in key points.

Neither of these two approaches is ideal. (1) involves a lot of code duplication, (2) involves patching the satoshi client source each time I grab a later version, with the everpresent risk of something breaking and the need to continue maintaining these patches. Moreover, unfortunately many of these key points happen to be in files like main.cpp which see frequent changes.

I would like to propose adding these callback hooks to the main branch. I am willing to help locate these key points, reorganize the code to place these methods in separate source files, define a callback mechanism, and contribute source code.

-Eric Lombrozo

I spent some time changing the internal bitcoin code to use callback hooks, but its far from complete (or even really usable from anything other than the code in the satoshi client itself, it doesnt even have any deregister methods!). As it sits now, it is likely to get more eyeballs and merged for 0.7. If you need additional specific callbacks, adding them would be cool, though I wouldn't recommend relying on the blockstore API to remain even remotely stable for the foreseeable future.

https://github.com/bitcoin/bitcoin/pull/771

Matt

On Wed, 2012-03-21 at 22:13 -0700, Eric Lombrozo wrote:

> Hey, guys.
>
> I've been writing a number of apps that require realtime event
> notifications, where the JSON-RPC API clearly doesn't suffice.
>
> There are two approaches I've been taking to this end:
>
> 1) Writing my own library for dealing with raw bitcoin structures and
> connecting to bitcoin nodes via the bitcoin protocol.
> 2) Making custom builds of the satoshi client putting callback hooks
> in key points.
>
> Neither of these two approaches is ideal. (1) involves a lot of code
> duplication, (2) involves patching the satoshi client source
> each time I grab a later version, with the everpresent risk of
> something breaking and the need to continue maintaining these patches.
> Moreover, unfortunately many of these key points happen to be in files
> like main.cpp which see frequent changes.
>
> I would like to propose adding these callback hooks to the main
> branch. I am willing to help locate these key points, reorganize the
> code
> to place these methods in separate source files, define a callback
> mechanism, and contribute source code.
>
> -Eric Lombrozo

You might also want to check out libcoin. Its a fork of the satoshi client: https://github.com/ceptacle/libcoin

Matt

Eric Lombrozo <elombrozo@gmail•com> wrote:

Hey, guys.

I've been writing a number of apps that require realtime event notifications, where the JSON-RPC API clearly doesn't suffice.

There are two approaches I've been taking to this end:

1) Writing my own library for dealing with raw bitcoin structures and connecting to bitcoin nodes via the bitcoin protocol. 2) Making custom builds of the satoshi client putting callback hooks in key points.

Neither of these two approaches is ideal. (1) involves a lot of code duplication, (2) involves patching the satoshi client source each time I grab a later version, with the everpresent risk of something breaking and the need to continue maintaining these patches. Moreover, unfortunately many of these key points happen to be in files like main.cpp which see frequent changes.

I would like to propose adding these callback hooks to the main branch. I am willing to help locate these key points, reorganize the code to place these methods in separate source files, define a callback mechanism, and contribute source code.

-Eric Lombrozo

_____________________________________________

This SF email is sponsosred by: Try Windows Azure free for 90 days Click Here http://p.sf.net/sfu/sfd2d-msazure

Hi Eric,

What hooks are you looking for in particular ?

libcoin supports registration of listeners for new blocks and new transactions. These are e.g. used for connecting the Wallet to the Node.

Cheers,

M

On 22/03/2012, at 06:39, bitcoin-list@bluematt•me wrote:

> You might also want to check out libcoin. Its a fork of the satoshi client: https://github.com/ceptacle/libcoin
>
> Matt
>
> Eric Lombrozo <elombrozo@gmail•com> wrote:
> Hey, guys.
>
> I've been writing a number of apps that require realtime event
> notifications, where the JSON-RPC API clearly doesn't suffice.
>
> There are two approaches I've been taking to this end:
>
> 1) Writing my own library for dealing with raw bitcoin structures and
> connecting to bitcoin nodes via the bitcoin protocol.
> 2) Making custom builds of the satoshi client putting callback hooks
> in key points.
>
> Neither of these two approaches is ideal. (1) involves a lot of code
> duplication, (2) involves patching the satoshi client source
> each time I grab a later version, with the everpresent risk of
> something breaking and the need to continue maintaining these patches.
> Moreover, unfortunately many of these key points happen to be in files
> like main.cpp which see frequent changes.
>
> I would like to propose adding these
> callback hooks to the main
>
> branch. I am willing to help locate these key points, reorganize the
> code
> to place these methods in separate source files, define a callback
> mechanism, and contribute source code.
>
> -Eric Lombrozo
>
>
> This SF email is sponsosred by:
> Try Windows Azure free for 90 days Click Here
> http://p.sf.net/sfu/sfd2d-msazure
>
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
> ------------------------------------------------------------------------------
> This SF email is sponsosred by:
> Try Windows Azure free for 90 days Click Here
> http://p.sf.net/sfu/sfd2d-msazure_______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Michael Gronager, PhD Director, Ceptacle Jens Juels Gade 33 2100 Copenhagen E Mobile: +45 31 45 14 01 E-mail: gronager@ceptacle•com Web: http://www.ceptacle.com/

On Thursday, March 22, 2012 1:13:13 AM Eric Lombrozo wrote:

> I would like to propose adding these callback hooks to the main
> branch. I am willing to help locate these key points, reorganize the
> code to place these methods in separate source files, define a callback
> mechanism, and contribute source code.

I'm not sure it's a good idea to let other code into the main wallet-handling process/memory space. I'd suggest looking into designing a workable Wallet Protocol for general use between the wallet and GUIs/applications.

https://en.bitcoin.it/wiki/Wallet_protocol has a draft I made a long time ago, but really seems a bit overcomplicated.

The callback architecture could be such that other code would never need to enter into the wallet-handling process/memory space. For instance, client applications could subscribe a particular URL to get sent an HTTP POST.

For the apps I've been working on, there really isn't any need to access the wallet space. I was talking more about events like "A new transaction was just seen" or "A new block was just seen", like what libcoin seems to support (sorry, Michael, I haven't really had a chance to look at it in depth but I will). Then there are other types of events for other bitcoin messages could also be useful: new addr, new node connected, node disconnected, bitcoin alert, etc...

Then there are events for dealing with potential attacks: DoS attempt, double-spend attempts (two transactions seen with valid signatures claiming the same output), node sending malformed messages, etc...

And then there are alerts pertaining to the status of the bitcoind process itself: bitcoind started, bitcoind ready to accept connections, bitcoind stopping, etc...

None of these events require the callback subscriber to have any access to the bitcoind process/memory space and all the I/O could be done via IPC or over network sockets.

I'd also like to be able to get detailed information of message format errors for debugging other nodes that connect to bitcoind. For instance, if a transaction is rejected because the signature is invalid, I want to know this. If it's because the amount is out of range or because the output couldn't be connected, I want to know this, too. I especially want to know if it was because the transaction is claiming an output that has already been claimed by another transaction.

For now, I've had to resort to sticking tracers and stubs into bitcoind. It would be really nice to not only be able to write an error log but to also let the connecting node know what went wrong. Obviously these types of messages should *not* be part of the bitcoin protocol itself since it invites all kinds of attacks. But it would be wonderful to have a side channel for this type of data, and it could also be done using callbacks.

The callback mechanism could be configurable in a similar fashion to the RPC in the bitcoin.conf file.

-Eric Lombrozo