Floating fees and SPV clients

Lately I was pondering how to make floating fees and SPV wallets work well together.

I propose the following plan:

1) 0.9 ships with something dead simple, like a command to query what a node estimates and then clients just take the average, or cross-check a centralised estimate against the P2P network. It's fast to implement and simple, but not very secure or decentralised. However it will allow the feature to launch on some kind of reasonable timeframe.

2) We bump the protocol version and the tx message now gets an optional protobuf buffer stuck on the end. The first thing put in this protobuf is a list of the values of the inputs. Using this data, the fee paid by a transaction can be calculated. In step 2 the data is unauthenticated.

3) Some SPV wallets already set themselves up so that they sync with the network in the background, e.g. the Android wallet syncs at least every 24 hours. This should become more common, using scheduler capabilities built into most operating systems. When the wallet syncs with the network, it sets a deliberately very noisy Bloom filter on its peers and waits around for 30-60 seconds or so. The wallet observes some of the broadcasts taking place and records the hashes and associated fees that were paid to disk. Next time it syncs, it includes the observed hashes into the Bloom filter used to download the chain, and thus learns how quickly they confirmed. It can calculate its own fee estimate from that.

4) Finally, when we next hard fork, we make v2 transactions include the output value in the signature, same as the output script (this proposal has been on the forums for a while now). That allows the fee data added in step 2 to be cross-checked against the signatures on the inputs, thus authenticating it.

I think this is a small and easy set of steps that would make it quite hard to attack - malicious nodes could make it appear that some transactions never confirmed thus seeming to force the price up, but it's easy to simply exclude transactions which never confirm at all from the calculations. Plus of course you can cross-check nodes against each other to try and catch nodes that are failing to match transactions properly.

One obvious concern is what to do if nodes don't converge on very similar estimates. Wallets will always want to pay the lowest fee possible, so that means they'll always be riding the very edge of what's acceptable, opening up tx propagation to random flaky failures if fee estimates change whilst a transaction is in progress, or if some nodes don't calculate the same estimates as others.

If a wallet gets a reject message for a tx that has a fee that are by its own estimates acceptable, what should it do? What if only some nodes report that and others don't?

On 12/01/2013 12:51 PM, Mike Hearn wrote:

> I propose the following plan:

Thanks taking the initiative and writing this up!

> If a wallet gets a reject message for a tx that has a fee that are by
> its own estimates acceptable, what should it do? What if only some nodes
> report that and others don't?

As long as the tx is not confirmed (by a broadcast), apps can offer to bump up the fee a little bit.

> As long as the tx is not confirmed (by a broadcast), apps can offer to
> bump up the fee a little bit.

Unfortunately there are risks to that approach.

The most obvious one is that nodes could keep sending reject messages to get wallets to attach ridiculously high fees. If half a wallets peers do this and the other half don’t, then effectively the wallet will double spend against itself. The bad nodes can keep the fat transaction and send it directly to a corrupt miner, no broadcast. If some other miner includes the original normal transaction, no problem, just take it out of the current block. If the corrupt miner finds the current block, they get to claim huge fee premiums.

Quite apart from the problem of malicious nodes/miners, how would you represent this in the wallet GUI? Current wallets are designed on the assumption that 1 payment == 1 transaction == 1 paid fee. If a single payment could have several different fees, and there’s no way to know which you will actually pay until later, then complexity would explode. Even the notion of balance would become even more complicated than it already is.

So I really don’t like the idea of creating different transactions depending on error messages from remote nodes. The only time when it could make sense is if *all* nodes reject a transaction. Then (assuming no MITM) you can assume the first transaction can be thrown away and a new attempt made.

But if you think about what the UI flows for that would look like - it’s just a mess.

There are other risks to fee estimation. Let’s say wallet authors create transactions with exactly the estimated fee needed to get into the next block. But due to mempool skew, estimates vary, and so those transactions don’t propagate cleanly everywhere. Now we have two problems:

1) Unpredictable failure to enter the mempools can lead to double spending and slow confirmations

2) Wallet authors may be tempted to ensure that doesn’t happen by taking the estimate, adding 10% and using that. But then if a bunch of popular wallets all do the same thing, the estimation algorithm might get confused and decide that as everyone seems to be attaching a fee of X+10%, the correct estimate for what fee to attach is X+10%. Then wallets would immediate raise their attached fees again and you’d enter into a infinite upward spiral.

The more I think about this, the more complicated it gets.

It’s tempting to try and just push all the complexity onto the merchant side, but one of the best things about Bitcoin is there isn’t any strong notion of “merchant” - that’s inherent to being peer to peer. So just hand-waving and saying sellers will deal with complicated fee processes is just a punt.

(my post hasn't shown up for an hour, so I'm sending it again)

On 12/01/2013 02:41 PM, Mike Hearn wrote:

>> As long as the tx is not confirmed (by a broadcast), apps can offer to
>> bump up the fee a little bit.
>
> Unfortunately there are risks to that approach.

I assume you're right, since I do not have so much experience with game theory.

About the UI:

Generally, for pending tx I'd like to measure time they're not being broadcast-confirmed and count blocks that they missed being included. Both can be combined into adapting the current generic messages ("This payment should become spendable shortly" for incoming and "This payment has not been transmitted yet" for outgoing transactions). Hint: Statistics could be offered by bitcoinj.

For outgoing transactions, if it is very clear that they're never going to be confirmed, I'd like to see a "Revoke" button. This would have saved us some support hassles with the transmit bugs. It could also offer a "Top up fee" button, which would replace the tx by a new one. I'm aware about a possible double spend but who cares? It doesn't matter which of the two transactions gets into the chain, as long as not both will be included.

> Both can be combined into adapting the current generic messages ("This
> payment should become spendable shortly" for incoming and "This payment
> has not been transmitted yet" for outgoing transactions).

What would the new messages say?

We need to get away from the notion of senders attaching fees anyway. This is the wrong way around because it’s the recipient who cares about double spending risk, not the sender. That’s why merchants keep running into issues with people attaching zero fees. Of course they attach zero fees. They know they aren’t going to double spend. It’s the merchant who cares about getting the security against that.

The UI for sending money should end up dead simple - no mention of fees anywhere, IMO.

The UI for receiving money could be a bit more complicated but even then - I think if ordinary people using smartphone wallets are having to think about how quickly they want their transaction to confirm and adjust fees, etc on the receiving side then we’re getting dangerously close to the usability failure zone.

Unfortunately we lack the protocol pieces to get the right UI here :( Someone needs to sit down and figure out what the UI *should* look like, in the ideal world, and then work backwards to figure out what needs to be done to get us there.

> For outgoing transactions, if it is very clear that they're never going
> to be confirmed, I'd like to see a "Revoke" button.

Disagree. There should never be any cases in which a transaction doesn’t confirm. Period. I know there have been bugs with bitcoinj that could cause this in the past, but they were bugs and they got fixed/will get fixed.

Settlement failure is just unacceptable and building a UI around the possibility will just encourage people to think of it as normal, when it should not be so.

On 12/01/2013 06:19 PM, Mike Hearn wrote:

>> Both can be combined into adapting the current generic messages ("This
>> payment should become spendable shortly" for incoming and "This payment
>> has not been transmitted yet" for outgoing transactions).
>
> What would the new messages say?

Well, for starters I'd suggest something like

"This payment did not become spendable since xxx minutes. Check with the sender if s/he paid the Bitcoin network fee. Check if your internet connection is working properly." (incoming)

"This payment still has not been transmitted. Check if your internet connection is working properly." (outgoing)

> We need to get away from the notion of senders attaching fees anyway.

This is the wrong way around because it’s the recipient who cares about double spending risk, not the sender. That’s why merchants keep running into issues with people attaching zero fees. Of course they attach zero fees. They know they aren’t going to double spend. It’s the merchant who cares about getting the security against that.

Guess you're right. But as you said, we're not there yet.

> The UI for sending money should end up dead simple - no mention of

fees anywhere, IMO.

Agreed, if the sender does not need to pay a fee any more. On the receiving side it of course needs to be mentioned. (Or the other way round, as of today.)

> Unfortunately we lack the protocol pieces to get the right UI here :(

Someone needs to sit down and figure out what the UI *should* look like, in the ideal world, and then work backwards to figure out what needs to be done to get us there.

(The ideal world doesn't need a UI for money.)

>> For outgoing transactions, if it is very clear that they're never going
>> to be confirmed, I'd like to see a "Revoke" button.
>
> Disagree. There should never be any cases in which a transaction

doesn’t confirm. Period. I know there have been bugs with bitcoinj that could cause this in the past, but they were bugs and they got fixed/will get fixed.

>
> Settlement failure is just unacceptable and building a UI around the

possibility will just encourage people to think of it as normal, when it should not be so.

I fully understand your point of view. However, its not the reality currently. (Hopefully it is, after the fixes to bitcoinj.)

> "This payment did not become spendable since xxx minutes. Check with the
> sender if s/he paid the Bitcoin network fee. Check if your internet
> connection is working properly." (incoming)

That seems reasonable.

The other message should be implementable today, I think? If numBroadcastPeers > 0 post 0.10.3 then you know the tx made it out to the internet.

Unfortunately if nodes start to diverge a lot in terms of what they will accept, then “transmitted” is no longer a clean binary yes/no thing. Guess we’ll have to jump that hurdle when we come to it.

> Guess you're right. But as you said, we're not there yet.

The payment protocol at least would need some notion of fee, or possibly (better?) the ability for a recipient to specify some inputs as well as some outputs.

Originally I think we were hoping for child-pays-for-parent. I guess that needs someone to sit down and focus on it for a while, assuming we still think that’s a good idea.

On Sun, Dec 01, 2013 at 06:19:14PM +0100, Mike Hearn wrote:

> > Both can be combined into adapting the current generic messages ("This
> > payment should become spendable shortly" for incoming and "This payment
> > has not been transmitted yet" for outgoing transactions).
>
> What would the new messages say?
>
> We need to get away from the notion of senders attaching fees anyway. This is the wrong way around because it’s the recipient who cares about double spending risk, not the sender. That’s why merchants keep running into issues with people attaching zero fees. Of course they attach zero fees. They know they aren’t going to double spend. It’s the merchant who cares about getting the security against that.
>
> The UI for sending money should end up dead simple - no mention of fees anywhere, IMO.
>
> The UI for receiving money could be a bit more complicated but even then - I think if ordinary people using smartphone wallets are having to think about how quickly they want their transaction to confirm and adjust fees, etc on the receiving side then we’re getting dangerously close to the usability failure zone.
>
> Unfortunately we lack the protocol pieces to get the right UI here :( Someone needs to sit down and figure out what the UI *should* look like, in the ideal world, and then work backwards to figure out what needs to be done to get us there.
>
> > For outgoing transactions, if it is very clear that they're never going
> > to be confirmed, I'd like to see a "Revoke" button.
>
> Disagree. There should never be any cases in which a transaction doesn’t confirm. Period. I know there have been bugs with bitcoinj that could cause this in the past, but they were bugs and they got fixed/will get fixed.
>
> Settlement failure is just unacceptable and building a UI around the possibility will just encourage people to think of it as normal, when it should not be so.

Bitcoin is and always will be limited in capacity - transactions may not confirm in a reasonable about of time because of high-demand and/or DoS attacks. Giving senders and/or receivers the ability to increase fees after the fact is the only way you'll ever be able to deal with these situations. Of course, in those situations revoke isn't going to be 100% reliable until the txins get spent elsewhere, but that just indicates the UI problem is around that kind of functionality is subtle.

re: merchants paying tx fees, child-pays-for-parent is inefficient, and micropayments direct to miners isn't implemented. (though I did write up a rough sketch of how to do that in a decentralized fashion on #bitcoin-dev) Propose something concrete.

> Bitcoin is and always will be limited in capacity - transactions may not
> confirm in a reasonable about of time because of high-demand and/or DoS
> attacks.

I agree in the general case, but I was talking about the mobile wallet case specifically (i.e. people who are sending money between themselves or making small purchases of physical things). I think Bitcoin should be able to scale to handle these sorts of ordinary every-day transactions. Where I’d expect to see transactions falling off the edge is in more specialised cases like very small single micropayments, or “optional” internal transactions like mixing/re/defragmentation of wallets that don’t correspond to an actual payment. Those sorts of transactions would I guess be the first to go when faced with a sudden capacity crunch, but they wouldn’t show up in a mobile wallet UI anyway.

> re: merchants paying tx fees, child-pays-for-parent is inefficient

I know the existing code is, but is that fundamentally the case or just how the code has been written? I haven’t looked at this issue much but I know you’ve worked on it, so I’m curious to learn about why it’s inefficient and whether there are any fixes possible.

On Sun, Dec 01, 2013 at 07:18:07PM +0100, Mike Hearn wrote:

> > Bitcoin is and always will be limited in capacity - transactions may not
> > confirm in a reasonable about of time because of high-demand and/or DoS
> > attacks.
>
> I agree in the general case, but I was talking about the mobile wallet case specifically (i.e. people who are sending money between themselves or making small purchases of physical things). I think Bitcoin should be able to scale to handle these sorts of ordinary every-day transactions. Where I’d expect to see transactions falling off the edge is in more specialised cases like very small single micropayments, or “optional” internal transactions like mixing/re/defragmentation of wallets that don’t correspond to an actual payment. Those sorts of transactions would I guess be the first to go when faced with a sudden capacity crunch, but they wouldn’t show up in a mobile wallet UI anyway.

Maybe, maybe not. We have no idea what fees will be because the system's entire capacity is, and always will be, limited. That's just how fundementally unscalable systems with huge global state work. What demand will be for that limited capacity is unknown.

> > re: merchants paying tx fees, child-pays-for-parent is inefficient
>
> I know the existing code is, but is that fundamentally the case or just how the code has been written? I haven’t looked at this issue much but I know you’ve worked on it, so I’m curious to learn about why it’s inefficient and whether there are any fixes possible.

No, Luke's existing code uses good algorithms with O(n) scaling for n transactions. The inefficiency is needing a second transaction, bloating the blockchain and driving up fees.

First time posting to this mailing list so feel free to ignore me if this is a stupid idea.

On Mon, Dec 2, 2013 at 3:49 AM, Mike Hearn <mike@plan99•net> wrote:

>
> We need to get away from the notion of senders attaching fees anyway. This is the wrong
> way around because it’s the recipient who cares about double spending risk, not the sender.
>

It seems to me that a common problem currently revolves around accepting transactions in retail scenarios, such as paying for a sandwich from Subway. A solution could be to give the vendor responsibility for setting the fee, which means they can choose the trade-off that works best for them in terms of fee size vs. speed of processing.

Idea: Add a "fee" parameter to the payment URI specification. When processing the transaction, the customer's UI should show only the total price, including both the transfer amount and the fee. The vendor only accepts the transaction if the customer uses the right amount and fee. If the fee is too small (for example, the user might be using an older wallet and has selected a fee of zero), the vendor can issue a refund transaction immediately and tell the user to try again.

Pros: - could easily be implemented immediately - old wallets would still be supported by just manually entering the fee as users do now - no greater risk of double spending on either side - maintains the distributed nature of the system - relies on humans to judge the fee (who are much less likely to spiral infinitely upwards) - flexible enough to support varying sizes of transaction and varying degrees of security

Cons - requires the vendor to have sufficient understanding of Bitcoin to make the trade-off - doesn't solve the problem of selecting a fee for transactions between individuals/laymen - doesn't solve fee selection for automated transactions such as mixing/de/refragmentation

Thoughts?

Right, as I said earlier:

"The payment protocol at least would need some notion of fee, or possibly (better?) the ability for a recipient to specify some inputs as well as some outputs."

Having thought about it a bit more, I think it's better to just have a fee field that lets the receiver request the sender to attach the given fee. The outputs would have less value associated with them, so effectively the seller folds the fee into the price. If the seller is charging a round price like 1 mBTC, the user sees "1 mBTC" as the price, even if behind the scenes the created tx only sends 0.99999 BTC

Allowing specification of inputs seems to add too much complexity in other cases, like when value isn't specified at all.

On Mon, Dec 2, 2013 at 2:54 PM, Patrick Mead <patrick@meadia•com•au> wrote:

> First time posting to this mailing list so feel free to ignore me if
> this is a stupid idea.
>
>
> On Mon, Dec 2, 2013 at 3:49 AM, Mike Hearn <mike@plan99•net> wrote:
> >
> > We need to get away from the notion of senders attaching fees anyway.
> This is the wrong
> > way around because it’s the recipient who cares about double spending
> risk, not the sender.
> >
>
>
> It seems to me that a common problem currently revolves around
> accepting transactions in
> retail scenarios, such as paying for a sandwich from Subway. A
> solution could be to give the
> vendor responsibility for setting the fee, which means they can choose
> the trade-off that works
> best for them in terms of fee size vs. speed of processing.
>
> Idea:
> Add a "fee" parameter to the payment URI specification.
> When processing the transaction, the customer's UI should show only
> the total price, including
> both the transfer amount and the fee. The vendor only accepts the
> transaction if the customer
> uses the right amount and fee. If the fee is too small (for example,
> the user might be using an
> older wallet and has selected a fee of zero), the vendor can issue a
> refund transaction
> immediately and tell the user to try again.
>
> Pros:
> - could easily be implemented immediately
> - old wallets would still be supported by just manually entering the
> fee as users do now
> - no greater risk of double spending on either side
> - maintains the distributed nature of the system
> - relies on humans to judge the fee (who are much less likely to
> spiral infinitely upwards)
> - flexible enough to support varying sizes of transaction and varying
> degrees of security
>
> Cons
> - requires the vendor to have sufficient understanding of Bitcoin to
> make the trade-off
> - doesn't solve the problem of selecting a fee for transactions
> between individuals/laymen
> - doesn't solve fee selection for automated transactions such as
> mixing/de/refragmentation
>
>
> Thoughts?
>
>
> ------------------------------------------------------------------------------
> Rapidly troubleshoot problems before they affect your business. Most IT
> organizations don't have a clear picture of how application performance
> affects their revenue. With AppDynamics, you get 100% visibility into your
> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics
> Pro!
> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

On Mon, Dec 2, 2013 at 9:33 AM, Mike Hearn <mike@plan99•net> wrote:

> "The payment protocol at least would need some notion of fee, or possibly
> (better?) the ability for a recipient to specify some inputs as well as some
> outputs."

<vendor hat: on>

BitPay noticed this detail last week. We were noticing that some transactions were not even reaching our bitcoind border routers (edge nodes), due to low/no fees. That led to a long discussion of all things fee-related. SPV fees are a big issue. Getting child-pays-for-parent in some form out to miners is another. Getting a smart, dynamic fee market Gavin mentions is a big need.

PPv1 doesn't have any notion of fee unfortunately. I suppose it could be added easily, but we also need to launch the existing feature set.

There's code pending review to implement PPv1 in bitcoinj, unfortunately it's currently not passing unit tests and the author can't figure out why. I didn't have time to debug it yet myself. I'm hopeful we can get it working and merged by EOY.

It may be time to start talking about timelines for 0.9. I am wondering if floating fees should be broken out of the 0.9 release and launched in a quick 0.10 followup - if that were to be done then I think 0.9 could go to beta relatively soon, like early next year. There have been a lot of improvements already and it'd be a shame to block them all further.

On Mon, Dec 2, 2013 at 3:37 PM, Jeff Garzik <jgarzik@bitpay•com> wrote:

> On Mon, Dec 2, 2013 at 9:33 AM, Mike Hearn <mike@plan99•net> wrote:
> > "The payment protocol at least would need some notion of fee, or possibly
> > (better?) the ability for a recipient to specify some inputs as well as
> some
> > outputs."
>
> <vendor hat: on>
>
> BitPay noticed this detail last week.  We were noticing that some
> transactions were not even reaching our bitcoind border routers (edge
> nodes), due to low/no fees.  That led to a long discussion of all
> things fee-related.  SPV fees are a big issue.  Getting
> child-pays-for-parent in some form out to miners is another.  Getting
> a smart, dynamic fee market Gavin mentions is a big need.
>
> --
> Jeff Garzik
> Bitcoin core developer and open source evangelist
> BitPay, Inc.      https://bitpay.com/
>

Current rough timeline proposed for 0.9 was end-of-January, IIRC.

On Mon, Dec 2, 2013 at 9:44 AM, Mike Hearn <mike@plan99•net> wrote:

> PPv1 doesn't have any notion of fee unfortunately. I suppose it could be
> added easily, but we also need to launch the existing feature set.
>
> There's code pending review to implement PPv1 in bitcoinj, unfortunately
> it's currently not passing unit tests and the author can't figure out why. I
> didn't have time to debug it yet myself. I'm hopeful we can get it working
> and merged by EOY.
>
> It may be time to start talking about timelines for 0.9. I am wondering if
> floating fees should be broken out of the 0.9 release and launched in a
> quick 0.10 followup - if that were to be done then I think 0.9 could go to
> beta relatively soon, like early next year. There have been a lot of
> improvements already and it'd be a shame to block them all further.
>
>
>
> On Mon, Dec 2, 2013 at 3:37 PM, Jeff Garzik <jgarzik@bitpay•com> wrote:
>>
>> On Mon, Dec 2, 2013 at 9:33 AM, Mike Hearn <mike@plan99•net> wrote:
>> > "The payment protocol at least would need some notion of fee, or
>> > possibly
>> > (better?) the ability for a recipient to specify some inputs as well as
>> > some
>> > outputs."
>>
>> <vendor hat: on>
>>
>> BitPay noticed this detail last week.  We were noticing that some
>> transactions were not even reaching our bitcoind border routers (edge
>> nodes), due to low/no fees.  That led to a long discussion of all
>> things fee-related.  SPV fees are a big issue.  Getting
>> child-pays-for-parent in some form out to miners is another.  Getting
>> a smart, dynamic fee market Gavin mentions is a big need.
>>
>> --
>> Jeff Garzik
>> Bitcoin core developer and open source evangelist
>> BitPay, Inc.      https://bitpay.com/
>
>

On Tue, Dec 3, 2013 at 12:44 AM, Mike Hearn <mike@plan99•net> wrote:

> PPv1 doesn't have any notion of fee unfortunately. I suppose it could be
> added easily, but we also need to launch the existing feature set.
>

Lets bang out a merchant-pays-fee extension.

How about:

SPEC:

    optional uint64 allowfee    tag number=1000

Allow up to allowfee satoshis to be deducted from the amount paid to be used to pay Bitcoin network transaction fees. A wallet implementation must not reduce the amount paid for fees more than allowfee, and transaction fees must be equal to or greater than the amount reduced.

:ENDSPEC

Rationale: we don't want wallet software giving users discounts-- sending transactions that are amount-allowfee without paying any fee. We also want to allow users to pay MORE in fees, if they need to (fragmented wallet, maybe, or big CoinJoin transaction) or decide to.

PS: I think there was also consensus that the BIP72 request=... should be shortened to just r=... (save 6 chars in QR codes). Unless somebody objects, I'll change the BIP and the reference implementation code to make it so...

On Tue, Dec 3, 2013 at 2:40 AM, Gavin Andresen <gavinandresen@gmail•com>wrote:

>     optional uint64 allowfee    tag number=1000
>

Let's just use a normal/low tag number. The extensions mechanism is great for people who want to extend the protocol outside the core development process. It'd be weird if nobody ever used the low numbers again though.

Tag numbers are varint encoded so using smaller ones does have a minor efficiency benefit, it's not just aesthetics :)

> Allow up to allowfee satoshis to be deducted from the amount paid to be
> used to pay Bitcoin network transaction fees. A wallet implementation must
> not reduce the amount paid for fees more than allowfee, and transaction
> fees must be equal to or greater than the amount reduced.
>

Hmmm. Why "allow"? Should it not be called min_fee instead? Wallets would have to attach at least that much in fees, right?

Also, why describe it as reducing the amount paid? Which output would be reduced in value? Why not just have it be added to the total value displayed to the user and the outputs are left alone/not reduced.

> We also want to allow users to pay MORE in fees, if they need to
> (fragmented wallet, maybe, or big CoinJoin transaction) or decide to.
>

I like the idea but it seems this gets us back to the original problem - senders don't care about confirmations, ever, not even if they make an annoying set of transactions. The protocol allows users to submit transactions directly to receivers, I guess, if the receiver does not like the transactions they get they could potentially reject the payment. But I'd hope that's really rare.

> PS: I think there was also consensus that the BIP72  request=...   should
> be shortened to just r=... (save 6 chars in QR codes).  Unless somebody
> objects, I'll change the BIP and the reference implementation code to make
> it so...
>

Sweet, thanks!

I dont like the idea of putting the min fee in the hands of the receiver. Seems like that will work against the best interests of senders in the long run.

Why not try a different path of calculating the min fee like difficulty retarget. You can analyse the last 2016 blocks to find the average fee accepted per kb (which would include transactions that were included without fees) and then write that into the block as a soft recommendation that wallets could use in the UI. This way the price can vary up and down according to what people were willing to spend on fees and miners willing to accept.

I absolutely do not trust vendors to set fees. I think it has to be based on what senders are willing to pay and what miners are willing to accept.

Drak

On Tue, Dec 3, 2013 at 11:36 AM, Drak <drak@zikula•org> wrote:

> I dont like the idea of putting the min fee in the hands of the receiver.
> Seems like that will work against the best interests of senders in the long
> run.
>

Senders have no interest in ever attaching any kind of fee, which is one reason we explored child-pays-for-parent for a while. It's not the sender who cares about double spending risk. Left to their own devices, all senders would always attach no fee at all (or rather: whatever the min was to get the transaction relayed to the merchant).

However, receivers do want a fee attached, and ideally we would do this without redundant transactions. Hence, receivers asking senders to attach a fee and effectively folding it into the price that is paid. That is, if you go into a restaurant and the menu says "Burger: 10mBTC" then when you come to pay, what you see on your phone screen is 10mBTC. The fact that actually the shop with receiver 9.9mBTC and the tx fee is 0.1mBTC is hidden in the user interface - creating a situation like many others, where receivers eat a transaction cost. For instance in Europe sales taxes are included in the price, not attached separately later.

There's no need to trust the vendor. If a vendor asks for a ridiculously high tx fee, it will just surface as uncompetitively priced goods/services. Buyers will go elsewhere.

> Why not try a different path of calculating the min fee like difficulty
> retarget. You can analyse the last 2016 blocks to find the average fee
> accepted per kb (which would include transactions that were included
> without fees) and then write that into the block as a soft recommendation
> that wallets could use in the UI. This way the price can vary up and down
> according to what people were willing to spend on fees and miners willing
> to accept.
>

That's what fee estimation does, essentially, minus the encoding into blocks. Once you start getting miners telling people what fees are directly you run into cases where they might try to lie about their behaviour or otherwise influence the average. Querying all nodes avoids that problem.

On Tue, Dec 03, 2013 at 11:40:35AM +1000, Gavin Andresen wrote:

> On Tue, Dec 3, 2013 at 12:44 AM, Mike Hearn <mike@plan99•net> wrote:
>
> > PPv1 doesn't have any notion of fee unfortunately. I suppose it could be
> > added easily, but we also need to launch the existing feature set.
> >
>
> Lets bang out a merchant-pays-fee extension.
>
> How about:
>
> SPEC:
>
>     optional uint64 allowfee    tag number=1000
>
> Allow up to allowfee satoshis to be deducted from the amount paid to be
> used to pay Bitcoin network transaction fees. A wallet implementation must
> not reduce the amount paid for fees more than allowfee, and transaction
> fees must be equal to or greater than the amount reduced.

Fees are per byte of tx data; call it allowfeeperkb, and given that fees are required - the merchant would really rather not waste up to about twice as much on fees for a child-pays-for-parent - it should be called requirefeeperkb.

Back to your point, the merchant wants to limit total fees that have been deducted - 'allowfee' is still a good idea - but only in conjunction with specifying fee-per-kb requirements.

UI once both are implemented is to not show anything in the default case, and explain to the user why they have to pay extra in the unusual case where they are spending a whole bunch of dust.

On 3 December 2013 10:45, Mike Hearn <mike@plan99•net> wrote:

> On Tue, Dec 3, 2013 at 11:36 AM, Drak <drak@zikula•org> wrote:
>
>> I dont like the idea of putting the min fee in the hands of the receiver.
>> Seems like that will work against the best interests of senders in the long
>> run.
>>
>
> Senders have no interest in ever attaching any kind of fee, which is one
> reason we explored child-pays-for-parent for a while. It's not the sender
> who cares about double spending risk. Left to their own devices, all
> senders would always attach no fee at all (or rather: whatever the min was
> to get the transaction relayed to the merchant).
>

I respectfully disagree. Senders need their funds to be received. The incentive is right there. Miners want mining fees. So if you want to pay for something, you need to make sure payment arrives. Senders know that if they exclude the fee it might not arrive at all. Miners increasingly ignore no or low fees. So those two agents together ensure there is a fee more than not. If what you said was true, we would hardly see fees being paid at all, but on the contrary we see lots of fees, and much higher than the minimum 0.0001/kb rate that is currently required.

Merchants will just include ridiculous fees - there are some exchanges that do it already - MtGox being the famous example requires a 0.001 fee 10x higher than the network rate - the CEO does it because he says "it's better". That's not a fee going to MtGox, that is the miner fee and they have no plans to reduce it.

Typically vendor software may not get updated and or lag behind with fat fees never decreasing or decreasing way too slowly - it's just not fluid or dynamic enough when it rests with the vendor.

However, receivers do want a fee attached,

>

No - receivers want to be paid. If they are not paid they wont dispatch the goods or service. Neither party is happy in that circumstance. The incentive that the payment confirms is there naturally.

> That's what fee estimation does, essentially, minus the encoding into
> blocks. Once you start getting miners telling people what fees are directly
> you run into cases where they might try to lie about their behaviour or
> otherwise influence the average. Querying all nodes avoids that problem.
>

Miners cant lie about fees accepted because that's part of the transaction. When Alice sends funds to Bob it includes the fee information and that is included in the block. There is no way to fake it. The average fee paid is provable - so there is no need to query nodes at all, you simply look at the blockchain. You dont even need to write it into the blockchain since it can be calculated from the blockchain, it would just make it simpler.

Drak

Ok, revised spec:

SPEC:

message PaymentDeatils {
    ...
    optional uint64 minfee    tag number=8

Pay at least minfee satoshis in transaction fees. Wallet software should add minfee to the amount the user authorizes and pays, and include at least minfee in the transaction created to pay miner's transaction fees. Wallet software may request that the user pays more, if it must create a complex transaction or judges that minfee is not sufficient for the transaction to be accepted by the network..

:ENDSPEC

Making it fee-per-kilobyte is a bad idea, in my opinion; users don't care how many kilobytes their transactions are, and they will just be confused if they're paying for a 10mBTC burger and are asked to pay 10.00011 or 9.9994 because the merchant has no idea how many kilobytes the paying transaction will be.

On 3 December 2013 11:03, Peter Todd <pete@petertodd•org> wrote:

> UI once both are implemented is to not show anything in the default
> case, and explain to the user why they have to pay extra in the unusual
> case where they are spending a whole bunch of dust.

Yes, that's the other problem with a merchant setting a fee - they have no idea how large the transaction might be. If you spend a bunch of dust the fee could be 2 or 3x the expected fee. Then you might get merchants including higher fees by default to account for this. That means we end up paying more per kb over time.

On Tue, Dec 3, 2013 at 12:07 PM, Gavin Andresen <gavinandresen@gmail•com>wrote:

> Making it fee-per-kilobyte is a bad idea, in my opinion; users don't care
> how many kilobytes their transactions are, and they will just be confused
> if they're paying for a 10mBTC burger and are asked to pay 10.00011 or
> 9.9994 because the merchant has no idea how many kilobytes the paying
> transaction will be.
>

Wouldn't the idea be that the user always sees 10mBTC no matter what, but the receiver may receive less if the user decides to pay with a huge transaction?

It may be acceptable that receivers don't always receive exactly what they requested, at least for person-to-business transactions. For person-to-person transactions of course any fee at all is confusing because you intuitively expect that if you send 1 mBTC, then 1 mBTC will arrive the other end. I wonder if we'll end up in a world where buying things from shops involves paying fees, and (more occasional?) person-to-person transactions tend to be free and people just understand that the money isn't going to be spendable for a while. Or alternatively that wallets let you override the safeguards on spending unconfirmed coins when the user is sure that they trust the sender.

On Sun, Dec 01, 2013 at 12:51:46PM +0100, Mike Hearn wrote:

> 4) Finally, when we next hard fork, we make v2 transactions include the
> output value in the signature, same as the output script (this proposal has
> been on the forums for a while now). That allows the fee data added in step
> 2 to be cross-checked against the signatures on the inputs, thus
> authenticating it.

FTFY s/hard-fork/soft-fork/

The proposals on the forums with regard to a soft-fork are for a new transaction format - to be done in parallel with the old one - that commits to transaction inputs and outputs via a merkle tree extending into the transaction. This data is then committed to via the merge-mine standard.

In addition, all this discussion about trying to figure out transaction fees based on transaction is a bit irrelevant if you suppose a soft-fork. We already know that we need a merkle-sum-tree of fees and transaction size to be able to audit miners and create compact fraud proofs, and using that merkle-sum-tree you can easily calculate sum fees/sum size for the whole block to determine fees.

We know that we can make a good assumption that transactions in blocks will have been broadcast prior by the assumption you and Gavin are making that miners would by far prefer to mine transactions that have already been broadcast so that block propagation via txid lists works. That advantage is on the order of 10x to 100x (or even higher) lower cost per KB to the miner. (if this is not true, let us all know, because then your scalability arguments are bunk with regard to blocksize)

In addition faking fees via non-disclosed high-fee transactions has a cost of about 1% due to the risk your block gets orphaned and the tx fee gets mined by another miner.

Between those two the merkle-sum-trees for fees and size can be used for various levels of average fee for a whole block, plus statistical distributions.

Next it is also desirable for another, related, soft-fork to include a sorted list of txids and/or H(scriptPubKey)'s in a block. (the latter is explicitly part of my TXO commitments proposal) With the txids version especially it's easy and low-bandwidth for SPV nodes to get proof that a given transaction has been mined recently, by asking for proof of inclusion or exclusion to accompany a bloom filter match.

Finally with various anti-sybil techniques that create long-term identities it's easy to show that a peer lied about the data required for fee estimates anyway.

> One obvious concern is what to do if nodes don't converge on very similar
> estimates. Wallets will always want to pay the lowest fee possible, so that
> means they'll always be riding the very edge of what's acceptable, opening
> up tx propagation to random flaky failures if fee estimates change whilst a
> transaction is in progress, or if some nodes don't calculate the same
> estimates as others.

Propagation failures are not a major problem if transactions can be rebroadcast with new, higher, fees; propagation failures can be detected easily and quickly with my proof-of-tx-mining proposal.

On Tue, Dec 03, 2013 at 11:09:51AM +0000, Drak wrote:

> On 3 December 2013 11:03, Peter Todd <pete@petertodd•org> wrote:
>
> > UI once both are implemented is to not show anything in the default
> > case, and explain to the user why they have to pay extra in the unusual
> > case where they are spending a whole bunch of dust.
>
>
> Yes, that's the other problem with a merchant setting a fee - they have no
> idea how large the transaction might be. If you spend a bunch of dust the
> fee could be 2 or 3x the expected fee. Then you might get merchants
> including higher fees by default to account for this. That means we end up
> paying more per kb over time.

Right, which is solved by requiring a fee-per-kb, and only allowing up to a certain amount to be deducted from the amount paid to pay that total fee.

But really, we're better off leaving fees visible to the user in the first place: they're how Bitcoin works and it's not going to change. (was just talking to Taylor from Hive Wallet about that in person actually)

On Tue, Dec 03, 2013 at 12:29:03PM +0100, Mike Hearn wrote:

> On Tue, Dec 3, 2013 at 12:07 PM, Gavin Andresen <gavinandresen@gmail•com>wrote:
>
> > Making it fee-per-kilobyte is a bad idea, in my opinion; users don't care
> > how many kilobytes their transactions are, and they will just be confused
> > if they're paying for a 10mBTC burger and are asked to pay 10.00011 or
> > 9.9994 because the merchant has no idea how many kilobytes the paying
> > transaction will be.
> >
>
> Wouldn't the idea be that the user always sees 10mBTC no matter what, but
> the receiver may receive less if the user decides to pay with a huge
> transaction?
>
> It may be acceptable that receivers don't always receive exactly what they
> requested, at least for person-to-business transactions.  For
> person-to-person transactions of course any fee at all is confusing because
> you intuitively expect that if you send 1 mBTC, then 1 mBTC will arrive the
> other end. I wonder if we'll end up in a world where buying things from
> shops involves paying fees, and (more occasional?) person-to-person
> transactions tend to be free and people just understand that the money
> isn't going to be spendable for a while. Or alternatively that wallets let
> you override the safeguards on spending unconfirmed coins when the user is
> sure that they trust the sender.

Person-to-person payments are an *excellent* argument for keeping fees visible to end-users; people will pay other people commonly in Bitcoin and they will be very confused if those transactions act weirdly differently than payments to merchants.

NAK on unconfirmed overrides - if something goes wrong even by accident it just makes fixing the problem much harder and less intuitive.

>
> Wouldn't the idea be that the user always sees 10mBTC no matter what, but
> the receiver may receive less if the user decides to pay with a huge
> transaction?
>

If users want to pay with a huge transaction then it seems to me the user should cover that cost. Allowing users to pay merchants with 100K transactions full of dust and expecting them to eat the cost seems like a great way to enable bleed-the-merchant-dry attacks.

RE: hiding or showing fees: I pointed out to Peter that there doesn't have to be One True Answer. Let wallets experiment with either hiding or exposing fees, and may the best user experience win.

On Tue, Dec 3, 2013 at 12:41 PM, Gavin Andresen <gavinandresen@gmail•com>wrote:

>
> If users want to pay with a huge transaction then it seems to me the user
> should cover that cost. Allowing users to pay merchants with 100K
> transactions full of dust and expecting them to eat the cost seems like a
> great way to enable bleed-the-merchant-dry attacks.
>

A merchant can always refuse the payment and refund it if that's a practical problem. I doubt it would be though. If a user is trying to buy something from the merchant, they will want it to work, and it'll be up to the developers of the wallet they're using to ensure it never does anything obnoxious or unacceptable that would result in people hating to receive money from that app.

> RE: hiding or showing fees:  I pointed out to Peter that there doesn't
> have to be One True Answer.  Let wallets experiment with either hiding or
> exposing fees, and may the best user experience win.
>

Sure. I think there will be experimentation in this regard.

>
> A merchant can always refuse the payment and refund it if that's a
> practical problem.
>

No, they can't, at least not in bitcoin-qt: when the user pokes the SEND button, the transaction is broadcast on the network, and then the merchant is also told with the Payment/PaymentACK round-trip.

Allowing merchants to cancel (e.g. having a PaymentNACK) makes implementation harder, and brings up nasty issues if we want to allow CoinJoin or CoinJoin-like transactions as payments to merchants. Bitcoin-Qt ALREADY allows you to pay several PaymentRequests with one transaction; handling the case where one merchant gives you a PaymentACK and another gives you (or wants to give you) a PaymentNACK is a nightmare.

On Dec 3, 2013, at 12:29 PM, Mike Hearn <mike@plan99•net> wrote:

> It may be acceptable that receivers don't always receive exactly what they requested, at least for person-to-business transactions.  For person-to-person transactions of course any fee at all is confusing because you intuitively expect that if you send 1 mBTC, then 1 mBTC will arrive the other end. I wonder if we'll end up in a world where buying things from shops involves paying fees, and (more occasional?) person-to-person transactions tend to be free and people just understand that the money isn't going to be spendable for a while.

> person-to-business transactions.  For person-to-person transactions

Why should there be two classes of transactions? Where does paying a local business at a farmer’s stand lie in that realm? Transactions should work the same regardless of who is on the receiving end.

> any fee at all is confusing because you intuitively expect that if you send 1 mBTC, then 1 mBTC will arrive the other end

The paradigm of sending money has an explicit cost is not new... I think people are used to Western Union/PayPal and associated fees, no? It’s okay to have a fee if it’s reasonable, so let’s inform the user what the estimated cost is to send a transaction in a reasonable amount of time.

>  wonder if we'll end up in a world where buying things from shops involves paying fees

I stayed in a hotel for the first night here here in Milan, and there was an (anticipated) surcharge for the use of credit over cash. Again, nothing new here.

Fees are only confusing because existing clients do a terrible job of presenting the information to the user. In Hive Wallet, I’m of the opinion that we should inform the user in an intuitive way to let them make an informed decision.

On 3 December 2013 11:46, Mike Hearn <mike@plan99•net> wrote:

> On Tue, Dec 3, 2013 at 12:41 PM, Gavin Andresen <gavinandresen@gmail•com>wrote:
>>
>> If users want to pay with a huge transaction then it seems to me the user
>> should cover that cost. Allowing users to pay merchants with 100K
>> transactions full of dust and expecting them to eat the cost seems like a
>> great way to enable bleed-the-merchant-dry attacks.
>>
>
> A merchant can always refuse the payment and refund it if that's a
> practical problem. I doubt it would be though. If a user is trying to buy
> something from the merchant, they will want it to work, and it'll be up to
> the developers of the wallet they're using to ensure it never does anything
> obnoxious or unacceptable that would result in people hating to receive
> money from that app.
>

Refunds in this circumstance would be problematic because someone is going to lose because they have to pay the fee. If the sender's money is refunded minus the fee, they will be unhappy. And the merchant will be unhappy about having had an unacceptable transaction they have to send back, and eat a fee for the privilege.

This kind of situation needs to be avoided at all costs.

On Tue, Dec 03, 2013 at 12:57:23PM +0100, Taylor Gerring wrote:

>
> On Dec 3, 2013, at 12:29 PM, Mike Hearn <mike@plan99•net> wrote:
>
> > It may be acceptable that receivers don't always receive exactly what they requested, at least for person-to-business transactions.  For person-to-person transactions of course any fee at all is confusing because you intuitively expect that if you send 1 mBTC, then 1 mBTC will arrive the other end. I wonder if we'll end up in a world where buying things from shops involves paying fees, and (more occasional?) person-to-person transactions tend to be free and people just understand that the money isn't going to be spendable for a while.
>
>
> > person-to-business transactions.  For person-to-person transactions
> Why should there be two classes of transactions? Where does paying a local business at a farmer’s stand lie in that realm? Transactions should work the same regardless of who is on the receiving end.
>
> > any fee at all is confusing because you intuitively expect that if you send 1 mBTC, then 1 mBTC will arrive the other end
> The paradigm of sending money has an explicit cost is not new... I think people are used to Western Union/PayPal and associated fees, no?  It’s okay to have a fee if it’s reasonable, so let’s inform the user what the estimated cost is to send a transaction in a reasonable amount of time.

Indeed.

Transparency on fees is going to be good from a marketing point of view as well: fact is, Bitcoin transations have fees involved, and if we're up-front and honest about those fees and what they are and why, we demystify the system and give people the confidence to tell others about the cost-advantages of Bitcoin, and at the same time, combat fud about fees with accurate and honest information.

Hi all, first post so go easy on me!

Background/Intro: I'm a C/C++ software engineer with a keen interest in Bitcoin working for everyone. I've spent the last couple of months pitching Bitcoin to merchants & end users (previously I mined),

While I agree as Peter said, transparency with fees is good and leaving it to the wallet designer to decide to either show / hide these (as Gavin suggested) there is no denying that users hate fees/charges. That's why in the UK so many online retailers offer free delivery (again perceived as a zero value add charge). Infact, if you ask the average consumer, they only have a mild inkling that there are fees paid by the merchants for using credit cards.

So I'd agree with Gavin's proposed spec in an optional uint64 minfee which senders(wallets) should deduct from the total paid to the receiver. If however the sender is doing a dust collection txn (surely hugely unusual for legit users?) then the sender should pay the additional costs. Does that make "minfee" actually "minfeeperkb"? Perhaps, but wallets should aim to not display such implementation details.

As a newb though, I have to ask, how does the receiver/requester/merchant enforce these fee requests are respected?

On 3 December 2013 12:07, Peter Todd <pete@petertodd•org> wrote:

> On Tue, Dec 03, 2013 at 12:57:23PM +0100, Taylor Gerring wrote:
> >
> > On Dec 3, 2013, at 12:29 PM, Mike Hearn <mike@plan99•net> wrote:
> >
> > > It may be acceptable that receivers don't always receive exactly what
> they requested, at least for person-to-business transactions.  For
> person-to-person transactions of course any fee at all is confusing because
> you intuitively expect that if you send 1 mBTC, then 1 mBTC will arrive the
> other end. I wonder if we'll end up in a world where buying things from
> shops involves paying fees, and (more occasional?) person-to-person
> transactions tend to be free and people just understand that the money
> isn't going to be spendable for a while.
> >
> >
> > > person-to-business transactions.  For person-to-person transactions
> > Why should there be two classes of transactions? Where does paying a
> local business at a farmer’s stand lie in that realm? Transactions should
> work the same regardless of who is on the receiving end.
> >
> > > any fee at all is confusing because you intuitively expect that if you
> send 1 mBTC, then 1 mBTC will arrive the other end
> > The paradigm of sending money has an explicit cost is not new... I think
> people are used to Western Union/PayPal and associated fees, no?  It’s okay
> to have a fee if it’s reasonable, so let’s inform the user what the
> estimated cost is to send a transaction in a reasonable amount of time.
>
> Indeed.
>
> Transparency on fees is going to be good from a marketing point of view
> as well: fact is, Bitcoin transations have fees involved, and if we're
> up-front and honest about those fees and what they are and why, we
> demystify the system and give people the confidence to tell others about
> the cost-advantages of Bitcoin, and at the same time, combat fud about
> fees with accurate and honest information.
>
> --
> 'peter'[:-1]@petertodd.org
> 000000000000000f9102d27cfd61ea9e8bb324593593ca3ce6ba53153ff251b3
>
>
> ------------------------------------------------------------------------------
> Rapidly troubleshoot problems before they affect your business. Most IT
> organizations don't have a clear picture of how application performance
> affects their revenue. With AppDynamics, you get 100% visibility into your
> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics
> Pro!
> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

On Tue, Dec 3, 2013 at 12:57 PM, Taylor Gerring <taylor.gerring@gmail•com>wrote:

> Why should there be two classes of transactions? Where does paying a local
> business at a farmer’s stand lie in that realm? Transactions should work
> the same regardless of who is on the receiving end.
>

Lots and lots of people are psychologically trained to expect that they pay the sticker price for things. Yes in recent times some places have started to show additional fees for using credit cards, but only as a way to try and push people onto cheaper forms of payment, not because customers love surcharges. It's for that reason that many merchants don't do this, even when they could - I pay for things with Maestro Debit all the time and I don't think I've ever seen a surcharge. That system obviously has costs, but they're included.

This is just a basic cultural thing - when I buy something from a shop, the social expectation is that the seller should be grateful for receiving my money. "The customer is always right". When I send money to a friend, the social expectation is different. If my friend said, hey Mike, could you send me that 10 bucks you owe me from last weekend and what he receives is less than 10 bucks, he would probably feel annoyed - if I owe him 10 bucks then I owe him 10 bucks and it's my job the cover the fees. That's why PayPal makes sender pay fees in that case.

Maybe we need new terminology for this. *Interior fees* for included in the price/receiver pays and *exterior fees* for excluded from the price/sender pays?

Fees are only confusing because existing clients do a terrible job of

> presenting the information to the user. In Hive Wallet, I’m of the opinion
> that we should inform the user in an intuitive way to let them make an
> informed decision.
>

Have you thought through the UI for that in detail? How exactly are you going to explain the fee structure? Let the user pick the number of blocks they need to wait for? What's a block? Why should I care? Why shouldn't I just set the slider all the way to the other end and pay no fees at all? Is the merchant going to refuse to take my payment? Gavin just said that's not possible with Bitcoin-Qt. I'm thinking for bitcoinj I might go in a slightly different direction and not broadcast payments submitted via the payment protocol (and definitely not have one wire tx pay multiple payment requests simultaneously, at least not for consumer wallets).

On Dec 3, 2013, at 2:20 PM, Mike Hearn <mike@plan99•net> wrote:

> On Tue, Dec 3, 2013 at 12:57 PM, Taylor Gerring <taylor.gerring@gmail•com> wrote:
> Why should there be two classes of transactions? Where does paying a local business at a farmer’s stand lie in that realm? Transactions should work the same regardless of who is on the receiving end.
>
> Lots and lots of people are psychologically trained to expect that they pay the sticker price for things. Yes in recent times some places have started to show additional fees for using credit cards, but only as a way to try and push people onto cheaper forms of payment, not because customers love surcharges. It's for that reason that many merchants don't do this, even when they could - I pay for things with Maestro Debit all the time and I don't think I've ever seen a surcharge. That system obviously has costs, but they're included.
>
> This is just a basic cultural thing - when I buy something from a shop, the social expectation is that the seller should be grateful for receiving my money. "The customer is always right". When I send money to a friend, the social expectation is different. If my friend said, hey Mike, could you send me that 10 bucks you owe me from last weekend and what he receives is less than 10 bucks, he would probably feel annoyed - if I owe him 10 bucks then I owe him 10 bucks and it's my job the cover the fees. That's why PayPal makes sender pay fees in that case.
>
> Maybe we need new terminology for this. Interior fees for included in the price/receiver pays and exterior fees for excluded from the price/sender pays?
>
> Fees are only confusing because existing clients do a terrible job of presenting the information to the user. In Hive Wallet, I’m of the opinion that we should inform the user in an intuitive way to let them make an informed decision.
>
> Have you thought through the UI for that in detail? How exactly are you going to explain the fee structure? Let the user pick the number of blocks they need to wait for? What's a block? Why should I care? Why shouldn't I just set the slider all the way to the other end and pay no fees at all? Is the merchant going to refuse to take my payment? Gavin just said that's not possible with Bitcoin-Qt. I'm thinking for bitcoinj I might go in a slightly different direction and not broadcast payments submitted via the payment protocol (and definitely not have one wire tx pay multiple payment requests simultaneously, at least not for consumer wallets).
>
>

Most of what you mentioned is entirely culture-dependant. In the majority of North America, sales tax is calculated at the point of sale on top of the advertised price. When my local government increases sales taxes, we feel it BECAUSE we see it. Expose information in a digestible way. Just because you don’t instinctively know how to implement a UI for varying sender fees doesn’t mean that other wallets don’t.

Leave the fee structure alone. Instead, let’s concentrate on how to calculate an accurate assessment of what a reasonable fee is for reliable service and let the software shake out the rest.

Taylor

Heh. People feel rises in sales tax elsewhere too. When VAT rises merchants all raise their prices, they don't normally swallow it (or if they do, they make a big fuss over how awesome they are).

The US system is a complete pain in the ass. You never know how much money you actually need to pay for anything unless you happen to know the local rate and do the multiplication in your head. There's a reason this system is not used in big chunks of the world economy.

> Expose information in a digestible way. Just because you don’t
> instinctively know how to implement a UI for varying sender fees doesn’t
> mean that other wallets don’t.
>

I would love to know how to do it. If you have an intuitive GUI in mind please show us so other wallet authors can copy it :)

The merchant wants to include a fee to ensure the transaction is confirmed which is dependent on the fee per kilobyte, but they don't want to pay unexpectedly high fees. So what about including a min_fee_per_kilobyte and a max_fee in PaymentDetails describing what fees the merchant will pay. The sender would be expected to respect the min_fee_per_kilobyte but if the result exceeds max_fee the sender would be agreeing to pay the extra fee (exterior fees). The sender might also agree to pay fees in excess of min_fee_per_kilobyte.

The sender would deduct the interior or merchant fees from the first output.

The UI could show the payment "price" which would match the sum of original outputs. It would show the merchant fees (interior) and sender fees (exterior) if there are any. The UI should always show fees so users learn to expect them for all transactions.

This should allow the merchant to pay fees in most cases while not having to pay excessive fees if the sender wants to use some large transaction. If max_fee is 0 the sender would be expected to pay all fees.

On 12/03/2013 10:20 AM, Mike Hearn wrote:

> On Tue, Dec 3, 2013 at 12:57 PM, Taylor Gerring
> <taylor.gerring@gmail•com <mailto:taylor.gerring@gmail.com>> wrote:
>
>     Why should there be two classes of transactions? Where does paying
>     a local business at a farmer's stand lie in that realm?
>     Transactions should work the same regardless of who is on the
>     receiving end.
>
>
> Lots and lots of people are psychologically trained to expect that
> they pay the sticker price for things. Yes in recent times some places
> have started to show additional fees for using credit cards, but only
> as a way to try and push people onto cheaper forms of payment, not
> because customers love surcharges. It's for that reason that many
> merchants don't do this, even when they could - I pay for things with
> Maestro Debit all the time and I don't think I've ever seen a
> surcharge. That system obviously has costs, but they're included.
>
> This is just a basic cultural thing - when I buy something from a
> shop, the social expectation is that the seller should be grateful for
> receiving my money. "The customer is always right". When I send money
> to a friend, the social expectation is different. If my friend said,
> hey Mike, could you send me that 10 bucks you owe me from last weekend
> and what he receives is less than 10 bucks, he would probably feel
> annoyed - if I owe him 10 bucks then I owe him 10 bucks and it's my
> job the cover the fees. That's why PayPal makes sender pay fees in
> that case.
>
> Maybe we need new terminology for this. /Interior fees/ for included
> in the price/receiver pays and /exterior fees/ for excluded from the
> price/sender pays?
>
>     Fees are only confusing because existing clients do a terrible job
>     of presenting the information to the user. In Hive Wallet, I'm of
>     the opinion that we should inform the user in an intuitive way to
>     let them make an informed decision.
>
>
> Have you thought through the UI for that in detail? How exactly are
> you going to explain the fee structure? Let the user pick the number
> of blocks they need to wait for? What's a block? Why should I care?
> Why shouldn't I just set the slider all the way to the other end and
> pay no fees at all? Is the merchant going to refuse to take my
> payment? Gavin just said that's not possible with Bitcoin-Qt. I'm
> thinking for bitcoinj I might go in a slightly different direction and
> not broadcast payments submitted via the payment protocol (and
> definitely not have one wire tx pay multiple payment requests
> simultaneously, at least not for consumer wallets).
>
>
>
>
> ------------------------------------------------------------------------------
> Rapidly troubleshoot problems before they affect your business. Most IT
> organizations don't have a clear picture of how application performance
> affects their revenue. With AppDynamics, you get 100% visibility into your
> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
>
>
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development

allowfee:

> Allow up to allowfee satoshis to be deducted from the amount paid to be
> used to pay Bitcoin network transaction fees. A wallet >implementation
> must not reduce the amount paid for fees more than allowfee, and
> transaction fees must be equal to or greater than the >amount reduced.

minfee:

> Pay at least minfee satoshis in transaction fees. Wallet software should
> add minfee to the amount the user authorizes and pays, and >include at
> least minfee in the transaction created to pay miner's transaction fees.
> Wallet software may request that the user pays more, if >it must create
> a complex transaction or judges that minfee is not sufficient for the
> transaction to be accepted by the network..
>

Thanks for the draft specs Gavin. Very clear and precise.

Personally I think 'allowfee' is more useful than 'minfee'. The 'allowfee' tells me something very useful and definitive about what the merchant will let me do when making payment, and if the merchant chooses 'allowfee' intelligently, they can provide real value to their customers without exposing them to undue risk.

A 'minfee' field on the other hand, is just a data point for the wallet software to consider, and likely to be noisy enough that wallets will tend to ignore it. (e.g. like Drak's example of Gox's 0.001 fee)

The sender's wallet software will always be free to choose the fee, and paying less than the 'allowfee' or 'minfee' can still get a TX included in the next block.

I think of the PaymentRequest is a part of the purchase contract. If a payer transmits a transaction before 'expires' but with less than 'minfee', which gets included in the next block, have they failed to meet the terms of payment?

If there is some time criticality, for example to reduce exchange rate risk, then a wallet might need to choose a higher fee to ensure the transaction clears in time. Instead of 'minfee' I'm thinking it would be more appropriate to communicate this using the existing 'expires' field -- in other words, let the merchant express what their requirement is, not tell the wallet how to achieve it.

In the case of a transaction with too-low fee, either the payer can double-spend with a higher fee, or wait longer for the transaction to make it into a block. If it hits the blockchain before the 'expires' time, then the merchant should have no standing to refute it, regardless of the amount of fees paid.

A refund comes into play if a payer reduced the total amount in excess of an agreed upon 'allowfee', or if the transaction doesn't hit the blockchain until after 'expires'. It should be clear in these cases that payer would end up eating the fees in both directions. But then, what if a wallet pays the 'minfee' and broadcasts 1 block before 'expires' but the payment DOESN'T make the block? Is the merchant liable for too-slow transactions due to their own insufficient 'minfee' value?

So I see 'allowfee' as extremely useful, but 'minfee' as somewhat problematic.

Thanks, Jeremy

After reading all 99 messages in this thread, I think allowfee is just about perfect.

It effectively lets merchants to give an allowance against the purchase price for network fees, if they choose. It is still up to the sender (and/or the sender's software) to get the fees right. Sometimes the sender will need to pay more fees than allowed, and sometimes the sender will need to pay less.

We can't solve the fee problem, in general. I'm not sure that we can even define it properly. But this is something that we can do, that will be useful at least occasionally, and that will cause no harm the rest of the time.

P.S. Clever senders can use this to defrag their wallets. Who wants to write the patch for that?

Gavin Andresen wrote:

> On Tue, Dec 3, 2013 at 12:44 AM, Mike Hearn <mike@plan99•net
> <mailto:mike@plan99.net>> wrote:
>
>     PPv1 doesn't have any notion of fee unfortunately. I suppose it
>     could be added easily, but we also need to launch the existing
>     feature set.
>
>
> Lets bang out a merchant-pays-fee extension.
>
> How about:
>
> SPEC:
>
>     optional uint64 allowfee    tag number=1000
>
> Allow up to allowfee satoshis to be deducted from the amount paid to
> be used to pay Bitcoin network transaction fees. A wallet
> implementation must not reduce the amount paid for fees more than
> allowfee, and transaction fees must be equal to or greater than the
> amount reduced.
>
> :ENDSPEC
>
> Rationale: we don't want wallet software giving users discounts--
> sending transactions that are amount-allowfee without paying any fee.
>  We also want to allow users to pay MORE in fees, if they need to
> (fragmented wallet, maybe, or big CoinJoin transaction) or decide to.
>
>
> PS: I think there was also consensus that the BIP72  request=...
> should be shortened to just r=... (save 6 chars in QR codes).  Unless
> somebody objects, I'll change the BIP and the reference implementation
> code to make it so...
>
> --
> --
> Gavin Andresen
>
>
>
> ------------------------------------------------------------------------------
> Rapidly troubleshoot problems before they affect your business. Most IT
> organizations don't have a clear picture of how application performance
> affects their revenue. With AppDynamics, you get 100% visibility into your
> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
>
>
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge•net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development

I think this US/other cultural issue is complicating things more than we appreciate.

I am trying to imagine in my head how all this will work and what it will look like with allow_fee, and I just can't see it. Merchants want customers to pay the sticker price, deviance from that social norm is extremely rare even after the credit card company contracts that required it have been invalidated. The only time it happens to me is when buying flight tickets with credit cards: but it's only for that method, other payment methods are still treated as "free" a.k.a interior fees.

If you walk into a physical shop and try to pay a large bill with bags of pennies, the merchant won't enter into a complicated agreement where they agree to split the cost of processing with you. They will just reject the payment out of hand and tell you to get real. It has to be that way because otherwise the shop would carry the cost of counting all the pennies and hauling them around, not the buyer (who "knows" he put the right number of pennies in the bags).

As a buyer, I do not care about whether my transaction will confirm. If I try to pay with dust, there is no incentive for me to attach a higher fee than allow_fee to make that confirm, especially if the merchant has no way to reject the payment. What's more, as Jeremy points out, no clean fail mechanism means large piles of manual work and lots of disputes due to payments not clearing before the exchange rate shifts and other things like that.

Trying to make the success of payment confirmation a two-person dance seems to have so many edge cases it makes my head hurt. For most pay-to-merchant cases, it has to be the receivers job to get a transaction confirmed, and if the sender doesn't follow the instructions a payment should hard fail and require trying again. If Bitcoin-Qt can't handle that today, that does seem like a problem.

In the case of a transaction with too-low fee, either the payer can

> double-spend with a higher fee

You can't do that. When a tx doesn't have the right fee attached you're out of luck today, except for the fact that some pools run with a custom child pays for parent patch. So respending it would bump priority for some miners and not others.

Mike Hearn <mike@plan99•net> wrote:

>I think this US/other cultural issue is complicating things more than
>we
>appreciate.
>
>I am trying to imagine in my head how all this will work and what it
>will
>look like with allow_fee, and I just can't see it. Merchants want
>customers
>to pay the sticker price, deviance from that social norm is extremely
>rare
>even after the credit card company contracts that required it have been
>invalidated. The only time it happens to me is when buying flight
>tickets
>with credit cards: but it's only for that method, other payment methods
>are
>still treated as "free" a.k.a interior fees.
>
>If you walk into a physical shop and try to pay a large bill with bags
>of
>pennies, the merchant won't enter into a complicated agreement where
>they
>agree to split the cost of processing with you. They will just reject
>the
>payment out of hand and tell you to get real. It has to be that way
>because
>otherwise the shop would carry the cost of counting all the pennies and
>hauling them around, not the buyer (who "knows" he put the right number
>of
>pennies in the bags).
>
>As a buyer, I do not care about whether my transaction will confirm. If
>I
>try to pay with dust, there is no incentive for me to attach a higher
>fee
>than allow_fee to make that confirm, especially if the merchant has no
>way
>to reject the payment. What's more, as Jeremy points out, no clean fail
>mechanism means large piles of manual work and lots of disputes due to
>payments not clearing before the exchange rate shifts and other things
>like
>that.
>
>Trying to make the success of payment confirmation a two-person dance
>seems
>to have so many edge cases it makes my head hurt. For most
>pay-to-merchant
>cases, it has to be the receivers job to get a transaction confirmed,
>and
>if the sender doesn't follow the instructions a payment should hard
>fail
>and require trying again. If Bitcoin-Qt can't handle that today, that
>does
>seem like a problem.
>
>In the case of a transaction with too-low fee, either the payer can
>> double-spend with a higher fee
>
>
>You can't do that. When a tx doesn't have the right fee attached you're
>out
>of luck today, except for the fact that some pools run with a custom
>child
>pays for parent patch. So respending it would bump priority for some
>miners
>and not others.

Here at the dark wallet conf there seems yo be rough consensus that replacement for fee bumping is a good thing and should be supported; I was talking to Taylor from hive specifically yesterday. The code is trivial on the node side of things and doesn't need consent of anymore than a small minority, and coinjoin forces wallets to handle double spends well anyway. I haven't heard anyone caring about zeroconf safety.

I'll be proposing it for "formal" inclusion in our wallet best practices guidelines.

Also fwiw apparently libbitcoin already implements a memory limited mempool and Amir is open to the idea of it using the satoshi consensus critical code for block validity. (therefor fairly safe mining) I wouldn't be surprised if libbitcoin based nodes start getting usage, and with a limited mempool it is very DoS attack safe for them to relay replacements regardless of miner support.

Please don't try and drag this thread off topic. What I said is factually correct. If you want to (again) try and convince people things should work differently, start another thread for that.

On Wed, Dec 04, 2013 at 12:09:42PM +0100, Mike Hearn wrote:

> Please don't try and drag this thread off topic. What I said is factually
> correct. If you want to (again) try and convince people things should work
> differently, start another thread for that.

replace-by-fee is no less speculative than your original proposals; you're also trying to convince people that things should work differently re: fees

On Wed, Dec 4, 2013 at 2:06 PM, Peter Todd <pete@petertodd•org> wrote:

> replace-by-fee is no less speculative than your original proposals;
> you're also trying to convince people that things should work
> differently re: fees

The original proposal I started this thread with hasn't even received comments - presumably it's uncontroversial. The other discussions are about how to handle fees in requests that use the payment protocol, which isn't currently used anywhere so doing things differently isn't possible.

On the other hand you have been talking about a fundamental change to the behaviour of how all Bitcoin nodes operate, which is off topic for this thread.

If you have something specific to say about how floating fees should be managed by SPV wallets or how fees should be negotiated when the payment protocol is in use, this thread is appropriate. Otherwise please take it elsewhere.

On Wed, Dec 04, 2013 at 02:48:08PM +0100, Mike Hearn wrote:

> On Wed, Dec 4, 2013 at 2:06 PM, Peter Todd <pete@petertodd•org> wrote:
>
> > replace-by-fee is no less speculative than your original proposals;
> > you're also trying to convince people that things should work
> > differently re: fees
>
>
> The original proposal I started this thread with hasn't even received
> comments - presumably it's uncontroversial. The other discussions are about
> how to handle fees in requests that use the payment protocol, which isn't
> currently used anywhere so doing things differently isn't possible.
>
> On the other hand you have been talking about a fundamental change to the
> behaviour of how all Bitcoin nodes operate, which is off topic for this
> thread.
>
> If you have something specific to say about how floating fees should be
> managed by SPV wallets or how fees should be negotiated when the payment
> protocol is in use, this thread is appropriate. Otherwise please take it
> elsewhere.

Other than you, replacement for fee changing isn't controversial; I know this because no-one other than you comments on it... just like the fundemental changes involving your proposed hardfork presumably. (which I did comment on)

Besides, "Happily, there does not have to be One Correct Answer here. Let wallets compete, and may the best user experience win..."