Currency/exchange rate information API
14 messages, 4 to 13 Mar 2017. Started by Luke Dashjr.
Tim Ruffing (5), Luke Dashjr (4), Karl-Johan Alm, Jonas Schnelli, Andreas Schildbach, Marcel Jamin, Andrew LeCody
Investigating what it would take to add fiat currency information to Bitcoin Knots, I noticed Electrum currently has many implementations, one for each exchange rate provider, due to lack of a common format for such data.
Therefore, I put together an initial draft of a BIP that could standardise this so wallets (or other software) and exchange rate providers can simply interoperate without a lot of overhead reimplementing the same thing many ways.
One thing I am unsure about, is that currently this draft requires using XBT (as BTC) for Bitcoin amounts. It would seem nicer to use satoshis, but those don't really have a pseudo-ISO currency code to fit in nicely...
Current draft here:
https://github.com/luke-jr/bips/blob/bip-xchgrate/bip-xchgrate.mediawiki
Thoughts? Anything critical missing? Ways to make the interface better?
Luke
I am surprised nothing like this exists already, so am all for the idea.
Maybe I am misunderstanding, but I'm not sure you want to have thousand separators and other locale stuff in there. All currencies including USD are often shown in Swedish using space or dot as thousand separator and comma as decimal separator, e.g. "1.234,56 USD" or "1 234,56 USD". I.e. this is something that the locale of the user's environment defines, not something the server should have opinions about. It is also not ideal to propose a given format based on the user's locale, as some users have preferences for this (I personally use US locale for numbers, and a US person who is visiting Sweden wouldn't want this to suddenly change).
-Kalle.
On Sat, Mar 4, 2017 at 12:27 AM, Luke Dashjr via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (23 lines)
> Investigating what it would take to add fiat currency information to Bitcoin > Knots, I noticed Electrum currently has many implementations, one for each > exchange rate provider, due to lack of a common format for such data. > > Therefore, I put together an initial draft of a BIP that could standardise > this so wallets (or other software) and exchange rate providers can simply > interoperate without a lot of overhead reimplementing the same thing many > ways. > > One thing I am unsure about, is that currently this draft requires using XBT > (as BTC) for Bitcoin amounts. It would seem nicer to use satoshis, but those > don't really have a pseudo-ISO currency code to fit in nicely... > > Current draft here: > https://github.com/luke-jr/bips/blob/bip-xchgrate/bip-xchgrate.mediawiki > > Thoughts? Anything critical missing? Ways to make the interface better? > > Luke > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
I'm not sure if a BIP is the right thing in that case, given that the provided functionality is not special to Bitcoin and can be used in other contexts as well.
But ignoring this, the server should be authenticated at a minimum. Otherwise manipulating exchange rates seems to be a nice way for the attacker on the wire to make money...
Apart from that, my feeling is that it could be simplified. Is longpolling useful? And is the historical rate thing really necessary for typical applications?
If yes, the client should be allowed to decide on which time scale the data should be. (tick, min, hour, day, ...) That goes together with clearly defining the type field (something like low, high, open, close, but without flexibility). Think of a candle-stick chart basically.
Also, pushing may be more appropriate for "current" rates than polling. Then no polling interval is necessary. On the other hand, this adds complexity in other places, e.g., state.
Tim
On Sat, 2017-03-04 at 08:27 +0000, Luke Dashjr via bitcoin-dev wrote:
Quoted text (32 lines)
> Investigating what it would take to add fiat currency information to > Bitcoin > Knots, I noticed Electrum currently has many implementations, one for > each > exchange rate provider, due to lack of a common format for such data. > > Therefore, I put together an initial draft of a BIP that could > standardise > this so wallets (or other software) and exchange rate providers can > simply > interoperate without a lot of overhead reimplementing the same thing > many > ways. > > One thing I am unsure about, is that currently this draft requires > using XBT > (as BTC) for Bitcoin amounts. It would seem nicer to use satoshis, > but those > don't really have a pseudo-ISO currency code to fit in nicely... > > Current draft here: > https://github.com/luke-jr/bips/blob/bip-xchgrate/bip-xchgrate.me > diawiki > > Thoughts? Anything critical missing? Ways to make the interface > better? > > Luke > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
On Monday, March 06, 2017 5:37:24 AM Tim Ruffing via bitcoin-dev wrote:
Quoted text (3 lines)
> But ignoring this, the server should be authenticated at a > minimum. Otherwise manipulating exchange rates seems to be a nice > way for the attacker on the wire to make money...
HTTPS would be used for that. It's not something that needs to be at a higher layer.
Quoted text (2 lines)
> Apart from that, my feeling is that it could be simplified. Is > longpolling useful?
I would think so, at least for Bitcoin since rates can change significantly in a short period of time (or can they anymore? I haven't really watched lately.)
Quoted text (1 line)
> And is the historical rate thing really necessary for typical applications?
When displaying historical transactions, it doesn't really make sense to use the current market rate, but rather the market rate at the time the payment was made. While wallets might simply cache it with the transaction, it would be perhaps nicer if it could be automatically restored for seed-only recoveries. In any case, if a service/wallet doesn't want to provide/use historical information, it can simply not implement that part.
Quoted text (4 lines)
> If yes, the client should be allowed to decide on which time scale the > data should be. (tick, min, hour, day, ...) That goes together with > clearly defining the type field (something like low, high, open, close, > but without flexibility). Think of a candle-stick chart basically.
How is the current draft insufficient for this?
Quoted text (3 lines)
> Also, pushing may be more appropriate for "current" rates than polling. > Then no polling interval is necessary. On the other hand, this adds > complexity in other places, e.g., state.
Pushing is what longpolling does.
Luke
I like the BIP. It can reduce workload during implementation on both sides of the API and it allows to show the user more data without implementing tons of proprietary APIs. It’s not directly Bitcoin specific (example: BIP32 is also not Bitcoin specific), but I think a BIP is the right way for this.
Quoted text (6 lines)
> >> Apart from that, my feeling is that it could be simplified. Is >> longpolling useful? > > I would think so, at least for Bitcoin since rates can change significantly in > a short period of time (or can they anymore? I haven't really watched lately.)
Long polling is a simple push concept that works on most type of network configurations (NAT, proxy, etc.). The only concern I see here is that an public API will quickly fill up the maximum allowed httpd connections. But it’s solvable.
Quoted text (9 lines)
> >> And is the historical rate thing really necessary for typical applications? > > When displaying historical transactions, it doesn't really make sense to use > the current market rate, but rather the market rate at the time the payment > was made. While wallets might simply cache it with the transaction, it would > be perhaps nicer if it could be automatically restored for seed-only > recoveries. In any case, if a service/wallet doesn't want to provide/use > historical information, it can simply not implement that part.
I’m also not sure how useful historical datapoint are. I don’t think the use case where someone wants to restore from a seed and get all exchange rates during the time of the payment is something users are looking for. However, It’s optional.
Quoted text (13 lines)
> >> If yes, the client should be allowed to decide on which time scale the >> data should be. (tick, min, hour, day, ...) That goes together with >> clearly defining the type field (something like low, high, open, close, >> but without flexibility). Think of a candle-stick chart basically. > > How is the current draft insufficient for this? > >> Also, pushing may be more appropriate for "current" rates than polling. >> Then no polling interval is necessary. On the other hand, this adds >> complexity in other places, e.g., state. > > Pushing is what longpolling does.
Agree with Luke. A „real“ push (though I’d say long-polling is the real push, AFAIK it’s also the technique behind Apple’s iOS push channel) would require a complex server setup that complicates many things like load-balancer, mem-caching, etc.
</jonas>
On Mon, 2017-03-06 at 07:09 +0000, Luke Dashjr wrote:
Quoted text (9 lines)
> On Monday, March 06, 2017 5:37:24 AM Tim Ruffing via bitcoin-dev > wrote: > > But ignoring this, the server should be authenticated at a > > minimum. Otherwise manipulating exchange rates seems to be a nice > > way for the attacker on the wire to make money... > > HTTPS would be used for that. It's not something that needs to be at > a higher > layer.
Sure, HTTPS is the way to go. But I think that should be required or at least noted in the BIP, because people could miss easily, e.g., "I don't need TLS, all the data is public anyway."
Quoted text (11 lines)
> When displaying historical transactions, it doesn't really make sense > to use > the current market rate, but rather the market rate at the time the > payment > was made. While wallets might simply cache it with the transaction, > it would > be perhaps nicer if it could be automatically restored for seed-only > recoveries. In any case, if a service/wallet doesn't want to > provide/use > historical information, it can simply not implement that part. >
Having the rate at the time of payment is indeed very useful, yes. However that requires just a single value per payment, and there is no query that tells the server "give me the value closest to timestamp t" or similar. Of course the client can download and keep a large part of history and extract the information on its own but I can imagine that not every clients wants to do that, and also the client does not know in advance the bounds (from, to) that it must query.
Quoted text (9 lines)
> > If yes, the client should be allowed to decide on which time scale > the > > data should be. (tick, min, hour, day, ...) That goes together with > > clearly defining the type field (something like low, high, open, > close, > > but without flexibility). Think of a candle-stick chart basically. > > How is the current draft insufficient for this? >
In the current draft the client or the server cannot specify granularity. If the clients only wants one value per day but for an entire year, then it has to perform many requests or download and process a very large response. Also, I think it's okay that the type field allows for arbitrary user- defined values, but it should also have some precisely defined values (e.g. the mentioned low/high/open/close/typical). For example, it's not clear currently what "low" means for a timestamp (as opposed to a time span). Is it the low of the entire day or the low since the previous record or something different?
One has to be careful not to add too much complexity though. As soon as one moves away from timestamps to something like hours or days, all kind of issues with timezone, daylight saving time etc. appear. Maybe a simple way to let the client ask "give me one value for every interval of 3600 seconds" or similar.
Quoted text (2 lines)
> Pushing is what longpolling does. >
That makes a lot of sense, yes.
Tim
On Mon, 2017-03-06 at 16:54 -0500, Tim Ruffing via bitcoin-dev wrote:
Quoted text (6 lines)
> > > Pushing is what longpolling does. > > > > That makes a lot of sense, yes. >
Forgot one thing: For longpolling, maybe we would like to have the possibility to request some periodic message from the server. Otherwise clients cannot distinguish between the situations 1. "value is still in the requested bounds (minrate, maxrate)" and 2. "connection has dropped". So the user may take a wrong decision because he assumed that the value is still in bounds holds but actually the server has died.
Tim
On Monday, March 06, 2017 9:54:16 PM Tim Ruffing wrote:
Quoted text (8 lines)
> Having the rate at the time of payment is indeed very useful, yes. > However that requires just a single value per payment, and there is no > query that tells the server "give me the value closest to timestamp t" > or similar. > Of course the client can download and keep a large part of history and > extract the information on its own but I can imagine that not every > clients wants to do that, and also the client does not know in advance > the bounds (from, to) that it must query.
It would be a privacy leak to request only the specific timestamps, but I suppose many wallets lack even basic privacy to begin with.
To address the bounds issue, I have specified that when from/to don't have an exact record, that the previous/next (respectively) is provided.
Hopefully this addresses both concerns?
Quoted text (4 lines)
> In the current draft the client or the server cannot specify > granularity. If the clients only wants one value per day but for an > entire year, then it has to perform many requests or download and > process a very large response.
That's what the "timedelta" field solves, no? If you want one value per day, you'd put 86400.
Quoted text (6 lines)
> Also, I think it's okay that the type field allows for arbitrary user- > defined values, but it should also have some precisely defined values > (e.g. the mentioned low/high/open/close/typical). > For example, it's not clear currently what "low" means for a timestamp > (as opposed to a time span). Is it the low of the entire day or the low > since the previous record or something different?
Is it not sufficient for the server to specify this in the description of the given currency-pair feed?
Luke
On Monday, March 06, 2017 10:02:53 PM Tim Ruffing via bitcoin-dev wrote:
Quoted text (6 lines)
> For longpolling, maybe we would like to have the possibility to request > some periodic message from the server. Otherwise clients cannot > distinguish between the situations 1. "value is still in the requested > bounds (minrate, maxrate)" and 2. "connection has dropped". So the user > may take a wrong decision because he assumed that the value is still > in bounds holds but actually the server has died.
That's the job of TCP. Do you think we need to explicitly specify a keepalive configuration? Seems like that would vary based on client or network.
On Mon, 2017-03-06 at 22:14 +0000, Luke Dashjr wrote:
Quoted text (9 lines)
> It would be a privacy leak to request only the specific timestamps, > but I > suppose many wallets lack even basic privacy to begin with. > > To address the bounds issue, I have specified that when from/to don't > have an > exact record, that the previous/next (respectively) is provided. > > Hopefully this addresses both concerns?
Yes, that solves it. You're right with the privacy concern however.
Quoted text (8 lines)
> > > In the current draft the client or the server cannot specify > > granularity. If the clients only wants one value per day but for an > > entire year, then it has to perform many requests or download and > > process a very large response. > > That's what the "timedelta" field solves, no? > If you want one value per day, you'd put 86400.
Oh sure, I had overlooked that.
Quoted text (15 lines)
> > > Also, I think it's okay that the type field allows for arbitrary > user- > > defined values, but it should also have some precisely defined > values > > (e.g. the mentioned low/high/open/close/typical). > > For example, it's not clear currently what "low" means for a > timestamp > > (as opposed to a time span). Is it the low of the entire day or the > low > > since the previous record or something different? > > Is it not sufficient for the server to specify this in the > description of the > given currency-pair feed?
That works but a standardized way of indicating that piece of information to the client is useful. Then the client can display a "connection status" to the user, e.g., an "possible out-of-date" warning like the warning sign in the Qt GUI when Bitcoin Core is catching up the network.
Tim
On Mon, 2017-03-06 at 17:30 -0500, Tim Ruffing via bitcoin-dev wrote:
Quoted text (6 lines)
> > That works but a standardized way of indicating that piece of > information to the client is useful. Then the client can display a > "connection status" to the user, e.g., an "possible out-of-date" > warning like the warning sign in the Qt GUI when Bitcoin Core is > catching up the network.
Wait, forget this reply, I mixed up the two issues of keepalive and definition of low, high etc... -.-
1. Keepalive for longpolling: As I said, this can be useful for an out-of-date warning. I don't know if this is better solved with TCP keepalive or on the higher layer.
2. Definition of low, high: My feeling is that there is nothing wrong with providing exact definitions in the BIP, i.e.., giving up the flexibility does not too hurt much. However all of this is a minor issue after all.
Tim
On 03/06/2017 06:37 AM, Tim Ruffing via bitcoin-dev wrote:
Quoted text (2 lines)
> And is the historical rate thing really necessary > for typical applications?
Yes, it is. Basically all incoming transactions are historical, as your wallet is likely not online when it happens.
Quoted text (5 lines)
> Why are multiple results separated by a line-feed rather than using a JSON Array?
> * Clients ought to cache historical data, and using a line-feed format allows them to simply append to a cache file.
> * Parsing JSON typically requires the entire data parsed together as a single memory object. Using simple lines to separate results, however, allows parsing a single result at a time.
> What if long descriptions require line and paragraph breaks?
> * Clients should word-wrap long lines, and JSON escapes newlines as "\n" which can be used doubly ("\n\n") for paragraph breaks.I'd file this under premature optimization at the cost of interoperability. If you use JSON, then please use it properly.
I'd also say it's the job of the parser to offer a way of doing that, e.g. in .NET you can easily achieve that with Newtonsoft's JSON parser: http://stackoverflow.com/questions/20374083/deserialize-json-array-stream-one-item-at-a-time.
On 4 March 2017 at 09:27, Luke Dashjr via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (24 lines)
> > Investigating what it would take to add fiat currency information to Bitcoin > Knots, I noticed Electrum currently has many implementations, one for each > exchange rate provider, due to lack of a common format for such data. > > Therefore, I put together an initial draft of a BIP that could standardise > this so wallets (or other software) and exchange rate providers can simply > interoperate without a lot of overhead reimplementing the same thing many > ways. > > One thing I am unsure about, is that currently this draft requires using XBT > (as BTC) for Bitcoin amounts. It would seem nicer to use satoshis, but those > don't really have a pseudo-ISO currency code to fit in nicely... > > Current draft here: > https://github.com/luke-jr/bips/blob/bip-xchgrate/bip-xchgrate.mediawiki > > Thoughts? Anything critical missing? Ways to make the interface better? > > Luke > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
This formatting of JSON isn't unheard of though, it's typically called JSON Streaming[1]. As long as exchanges implementing the API actually follow the BIP and keep one JSON object per line, it shouldn't be a problem to decode.
1. https://en.wikipedia.org/wiki/JSON_Streaming
On Tue, Mar 7, 2017 at 8:07 AM Marcel Jamin via bitcoin-dev < bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (58 lines)
> > Why are multiple results separated by a line-feed rather than using a
> JSON Array?
> > * Clients ought to cache historical data, and using a line-feed format
> allows them to simply append to a cache file.
> > * Parsing JSON typically requires the entire data parsed together as a
> single memory object. Using simple lines to separate results, however,
> allows parsing a single result at a time.
> > What if long descriptions require line and paragraph breaks?
> > * Clients should word-wrap long lines, and JSON escapes newlines as "\n"
> which can be used doubly ("\n\n") for paragraph breaks.
>
> I'd file this under premature optimization at the cost of
> interoperability. If you use JSON, then please use it properly.
>
> I'd also say it's the job of the parser to offer a way of doing that,
> e.g. in .NET you can easily achieve that with Newtonsoft's JSON
> parser:
> http://stackoverflow.com/questions/20374083/deserialize-json-array-stream-one-item-at-a-time
> .
>
> On 4 March 2017 at 09:27, Luke Dashjr via bitcoin-dev
> <bitcoin-dev@lists•linuxfoundation•org> wrote:
> >
> > Investigating what it would take to add fiat currency information to
> Bitcoin
> > Knots, I noticed Electrum currently has many implementations, one for
> each
> > exchange rate provider, due to lack of a common format for such data.
> >
> > Therefore, I put together an initial draft of a BIP that could
> standardise
> > this so wallets (or other software) and exchange rate providers can
> simply
> > interoperate without a lot of overhead reimplementing the same thing many
> > ways.
> >
> > One thing I am unsure about, is that currently this draft requires using
> XBT
> > (as BTC) for Bitcoin amounts. It would seem nicer to use satoshis, but
> those
> > don't really have a pseudo-ISO currency code to fit in nicely...
> >
> > Current draft here:
> >
> https://github.com/luke-jr/bips/blob/bip-xchgrate/bip-xchgrate.mediawiki
> >
> > Thoughts? Anything critical missing? Ways to make the interface better?
> >
> > Luke
> > _______________________________________________
> > bitcoin-dev mailing list
> > bitcoin-dev@lists•linuxfoundation•org
> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation•org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>