BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
13 messages, 7 to 24 Dec 2017. Started by LORD HIS EXCELLENCY JAMES HRMH.
LORD HIS EXCELLENCY JAMES HRMH (9), Spartacus Rex (2), rhavar, Chris Riley
Good afternoon,
The need for this proposal:
We all must learn to admit that transaction bandwidth is still lurking as a serious issue for the operation, reliability, safety, consumer acceptance, uptake and, for the value of Bitcoin.
I recently sent a payment which was not urgent so; I chose three-day target confirmation from the fee recommendation. That transaction has still not confirmed after now more than six days - even waiting twice as long seems quite reasonable to me. That transaction is a valid transaction; it is not rubbish, junk or, spam. Under the current model with transaction bandwidth limitation, the longer a transaction waits, the less likely it is ever to confirm due to rising transaction numbers and being pushed back by transactions with rising fees.
I argue that no transactions are rubbish or junk, only some zero fee transactions might be spam. Having an ever-increasing number of valid transactions that do not confirm as more new transactions with higher fees are created is the opposite of operating a robust, reliable transaction system.
Business cannot operate with a model where transactions may or may not confirm. Even a business choosing a modest fee has no guarantee that their valid transaction will not be shuffled down by new transactions to the realm of never confirming after it is created. Consumers also will not accept this model as Bitcoin expands. If Bitcoin cannot be a reliable payment system for confirmed transactions then consumers, by and large, will simply not accept the model once they understand. Bitcoin will be a dirty payment system, and this will kill the value of Bitcoin.
Under the current system, a minority of transactions will eventually be the lucky few who have fees high enough to escape being pushed down the list.
Once there are more than x transactions (transaction bandwidth limit) every ten minutes, only those choosing twenty-minute confirmation (2 blocks) will have initially at most a fifty percent chance of ever having their payment confirm. Presently, not even using fee recommendations can ensure a sufficiently high fee is paid to ensure transaction confirmation.
I also argue that the current auction model for limited transaction bandwidth is wrong, is not suitable for a reliable transaction system and, is wrong for Bitcoin. All transactions must confirm in due time. Currently, Bitcoin is not a safe way to send payments.
I do not believe that consumers and business are against paying fees, even high fees. What is required is operational reliability.
This great issue needs to be resolved for the safety and reliability of Bitcoin. The time to resolve issues in commerce is before they become great big issues. The time to resolve this issue is now. We must have the foresight to identify and resolve problems before they trip us over. Simply doubling block sizes every so often is reactionary and is not a reliable permanent solution. I have written a BIP proposal for a technical solution but, need your help to write it up to an acceptable standard to be a full BIP.
I have formatted the following with markdown which is human readable so, I hope nobody minds. I have done as much with this proposal as I feel that I am able so far but continue to take your feedback.
# BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
## The problem: Everybody wants value. Miners want to maximize revenue from fees (and we presume, to minimize block size). Consumers need transaction reliability and, (we presume) want low fees.
The current transaction bandwidth limit is a limiting factor for both. As the operational safety of transactions is limited, so is consumer confidence as they realize the issue and, accordingly, uptake is limited. Fees are artificially inflated due to bandwidth limitations while failing to provide a full confirmation service for all transactions.
Current fee recommendations provide no satisfaction for transaction reliability and, as Bitcoin scales, this will worsen.
Bitcoin must be a fully scalable and reliable service, providing full transaction confirmation for every valid transaction.
The possibility to send a transaction with a fee lower than one that is acceptable to allow eventual transaction confirmation should be removed from the protocol and also from the user interface.
## Solution summary: Provide each transaction with an individual transaction priority each time before choosing transactions to include in the current block, the priority being a function of the fee paid (on a curve), and the time waiting in the transaction pool (also on a curve) out to n days (n=60 ?). The transaction priority to serve as the likelihood of a transaction being included in the current block, and for determining the order in which transactions are tried to see if they will be included.
Use a target block size. Determine the target block size using; current transaction pool size x ( 1 / (144 x n days ) ) = number of transactions to be included in the current block. Broadcast the next target block size with the current block when it is solved so that nodes know the next target block size for the block that they are building on.
The curves used for the priority of transactions would have to be appropriate. Perhaps a mathematician with experience in probability can develop the right formulae. My thinking is a steep curve. I suppose that the probability of all transactions should probably account for a sufficient number of inclusions that the target block size is met although, it may not always be. As a suggestion, consider including some zero fee transactions to pad, highest BTC value first?
**Explanation of the operation of priority:**
Quoted text (2 lines)
> If transaction priority is, for example, a number between one (low) and one-hundred (high) it can be directly understood as the percentage chance in one-hundred of a transaction being included in the block. Using probability or likelihood infers that there is some function of random. If random (100) < transaction priority then the transaction is included. >To break it down further, if both the fee on a curve value and the time waiting on a curve value are each a number between one and one-hundred, a rudimentary method may be to simply multiply those two numbers, to find the priority number. For example, a middle fee transaction waiting thirty days (if n = 60 days) may have a value of five for each part (yes, just five, the values are on a curve). When multiplied that will give a priority value of twenty-five, or, a twenty-five percent chance at that moment of being included in the block; it will likely be included in one of the next four blocks, getting more likely each chance. If it is still not included then the value of time waiting will be higher, making for more probability. A very low fee transaction would have a value for the fee of one. It would not be until near sixty-days that the particular low fee transaction has a high likelihood of being included in the block.
I am not concerned with low (or high) transaction fees, the primary reason for addressing the issue is to ensure transactional reliability and scalability while having each transaction confirm in due time.
## Pros: * Maximizes transaction reliability. * Fully scalable. * Maximizes possibility for consumer and business uptake. * Maximizes total fees paid per block without reducing reliability; because of reliability, in time confidence and overall uptake are greater; therefore, more transactions. * Market determines fee paid for transaction priority. * Fee recommendations work all the way out to 30 days or greater. * Provides additional block entropy; greater security since there is less probability of predicting the next block.
## Cons: * Could initially lower total transaction fees per block. * Must be first be programmed.
## Solution operation: This is a simplistic view of the operation. The actual operation will need to be determined in a spec for the programmer.
1. Determine the target block size for the current block. 2. Assign a transaction priority to each transaction in the pool. 3. Select transactions to include in the current block using probability in transaction priority order until the target block size is met. 5. Solve block. 6. Broadcast the next target block size with the current block when it is solved. 7. Block is received. 8. Block verification process. 9. Accept/reject block based on verification result. 10. Repeat.
## Closing comments: It may be possible to verify blocks conform to the proposal by showing that the probability for all transactions included in the block statistically conforms to a probability distribution curve, *if* the individual transaction priority can be recreated. I am not that deep into the mathematics; however, it may also be possible to use a similar method to do this just based on the fee, that statistically, the blocks conform to a fee distribution. Any zero fee transactions would have to be ignored. This solution needs a clever mathematician.
I implore, at the very least, that we use some method that validates full transaction reliability and enables scalability of block sizes. If not this proposal, an alternative.
Regards, Damian Williamson
I should not take it that the lack of critical feedback to this revised proposal is a glowing endorsement. I understand that there would be technical issues to resolve in implementation, but, are there no fundamental errors?
I suppose that it if is difficult to determine how long a transaction has been waiting in the pool then, each node could simply keep track of when a transaction was first seen. This may have implications for a verify routine, however, for example, if a node was offline, how should it differentiate how long each transaction was waiting in that case? If a node was restarted daily would it always think that all transactions had been waiting in the pool less than one day If each node keeps the current transaction pool in a file and updates it, as transactions are included in blocks and, as new transactions appear in the pool, then that would go some way to alleviate the issue, apart from entirely new nodes. There should be no reason the contents of a transaction pool files cannot be shared without agreement as to the transaction pool between nodes, just as nodes transmit new transactions freely.
It has been questioned why miners could not cheat. For the question of how many transactions to include in a block, I say it is a standoff and miners will conform to the proposal, not wanting to leave transactions with valid fees standing, and, not wanting to shrink the transaction pool. In any case, if miners shrink the transaction pool then I am not immediately concerned since it provides a more efficient service. For the question of including transactions according to the proposal, I say if it is possible to keep track of how long transactions are waiting in the pool so that they can be included on a probability curve then it is possible to verify that blocks conform to the proposal, since the input is a probability, the output should conform to a probability curve.
If someone has the necessary skill, would anyone be willing to develop the math necessary for the proposal?
Regards, Damian Williamson
________________________________ From: bitcoin-dev-bounces@lists•linuxfoundation•org <bitcoin-dev-bounces@lists•linuxfoundation•org> on behalf of Damian Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> Sent: Friday, 8 December 2017 8:01 AM To: bitcoin-dev@lists•linuxfoundation•org Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Good afternoon,
The need for this proposal:
We all must learn to admit that transaction bandwidth is still lurking as a serious issue for the operation, reliability, safety, consumer acceptance, uptake and, for the value of Bitcoin.
I recently sent a payment which was not urgent so; I chose three-day target confirmation from the fee recommendation. That transaction has still not confirmed after now more than six days - even waiting twice as long seems quite reasonable to me. That transaction is a valid transaction; it is not rubbish, junk or, spam. Under the current model with transaction bandwidth limitation, the longer a transaction waits, the less likely it is ever to confirm due to rising transaction numbers and being pushed back by transactions with rising fees.
I argue that no transactions are rubbish or junk, only some zero fee transactions might be spam. Having an ever-increasing number of valid transactions that do not confirm as more new transactions with higher fees are created is the opposite of operating a robust, reliable transaction system.
Business cannot operate with a model where transactions may or may not confirm. Even a business choosing a modest fee has no guarantee that their valid transaction will not be shuffled down by new transactions to the realm of never confirming after it is created. Consumers also will not accept this model as Bitcoin expands. If Bitcoin cannot be a reliable payment system for confirmed transactions then consumers, by and large, will simply not accept the model once they understand. Bitcoin will be a dirty payment system, and this will kill the value of Bitcoin.
Under the current system, a minority of transactions will eventually be the lucky few who have fees high enough to escape being pushed down the list.
Once there are more than x transactions (transaction bandwidth limit) every ten minutes, only those choosing twenty-minute confirmation (2 blocks) will have initially at most a fifty percent chance of ever having their payment confirm. Presently, not even using fee recommendations can ensure a sufficiently high fee is paid to ensure transaction confirmation.
I also argue that the current auction model for limited transaction bandwidth is wrong, is not suitable for a reliable transaction system and, is wrong for Bitcoin. All transactions must confirm in due time. Currently, Bitcoin is not a safe way to send payments.
I do not believe that consumers and business are against paying fees, even high fees. What is required is operational reliability.
This great issue needs to be resolved for the safety and reliability of Bitcoin. The time to resolve issues in commerce is before they become great big issues. The time to resolve this issue is now. We must have the foresight to identify and resolve problems before they trip us over. Simply doubling block sizes every so often is reactionary and is not a reliable permanent solution. I have written a BIP proposal for a technical solution but, need your help to write it up to an acceptable standard to be a full BIP.
I have formatted the following with markdown which is human readable so, I hope nobody minds. I have done as much with this proposal as I feel that I am able so far but continue to take your feedback.
# BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
## The problem: Everybody wants value. Miners want to maximize revenue from fees (and we presume, to minimize block size). Consumers need transaction reliability and, (we presume) want low fees.
The current transaction bandwidth limit is a limiting factor for both. As the operational safety of transactions is limited, so is consumer confidence as they realize the issue and, accordingly, uptake is limited. Fees are artificially inflated due to bandwidth limitations while failing to provide a full confirmation service for all transactions.
Current fee recommendations provide no satisfaction for transaction reliability and, as Bitcoin scales, this will worsen.
Bitcoin must be a fully scalable and reliable service, providing full transaction confirmation for every valid transaction.
The possibility to send a transaction with a fee lower than one that is acceptable to allow eventual transaction confirmation should be removed from the protocol and also from the user interface.
## Solution summary: Provide each transaction with an individual transaction priority each time before choosing transactions to include in the current block, the priority being a function of the fee paid (on a curve), and the time waiting in the transaction pool (also on a curve) out to n days (n=60 ?). The transaction priority to serve as the likelihood of a transaction being included in the current block, and for determining the order in which transactions are tried to see if they will be included.
Use a target block size. Determine the target block size using; current transaction pool size x ( 1 / (144 x n days ) ) = number of transactions to be included in the current block. Broadcast the next target block size with the current block when it is solved so that nodes know the next target block size for the block that they are building on.
The curves used for the priority of transactions would have to be appropriate. Perhaps a mathematician with experience in probability can develop the right formulae. My thinking is a steep curve. I suppose that the probability of all transactions should probably account for a sufficient number of inclusions that the target block size is met although, it may not always be. As a suggestion, consider including some zero fee transactions to pad, highest BTC value first?
**Explanation of the operation of priority:**
Quoted text (2 lines)
> If transaction priority is, for example, a number between one (low) and one-hundred (high) it can be directly understood as the percentage chance in one-hundred of a transaction being included in the block. Using probability or likelihood infers that there is some function of random. If random (100) < transaction priority then the transaction is included. >To break it down further, if both the fee on a curve value and the time waiting on a curve value are each a number between one and one-hundred, a rudimentary method may be to simply multiply those two numbers, to find the priority number. For example, a middle fee transaction waiting thirty days (if n = 60 days) may have a value of five for each part (yes, just five, the values are on a curve). When multiplied that will give a priority value of twenty-five, or, a twenty-five percent chance at that moment of being included in the block; it will likely be included in one of the next four blocks, getting more likely each chance. If it is still not included then the value of time waiting will be higher, making for more probability. A very low fee transaction would have a value for the fee of one. It would not be until near sixty-days that the particular low fee transaction has a high likelihood of being included in the block.
I am not concerned with low (or high) transaction fees, the primary reason for addressing the issue is to ensure transactional reliability and scalability while having each transaction confirm in due time.
## Pros: * Maximizes transaction reliability. * Fully scalable. * Maximizes possibility for consumer and business uptake. * Maximizes total fees paid per block without reducing reliability; because of reliability, in time confidence and overall uptake are greater; therefore, more transactions. * Market determines fee paid for transaction priority. * Fee recommendations work all the way out to 30 days or greater. * Provides additional block entropy; greater security since there is less probability of predicting the next block.
## Cons: * Could initially lower total transaction fees per block. * Must be first be programmed.
## Solution operation: This is a simplistic view of the operation. The actual operation will need to be determined in a spec for the programmer.
1. Determine the target block size for the current block. 2. Assign a transaction priority to each transaction in the pool. 3. Select transactions to include in the current block using probability in transaction priority order until the target block size is met. 5. Solve block. 6. Broadcast the next target block size with the current block when it is solved. 7. Block is received. 8. Block verification process. 9. Accept/reject block based on verification result. 10. Repeat.
## Closing comments: It may be possible to verify blocks conform to the proposal by showing that the probability for all transactions included in the block statistically conforms to a probability distribution curve, *if* the individual transaction priority can be recreated. I am not that deep into the mathematics; however, it may also be possible to use a similar method to do this just based on the fee, that statistically, the blocks conform to a fee distribution. Any zero fee transactions would have to be ignored. This solution needs a clever mathematician.
I implore, at the very least, that we use some method that validates full transaction reliability and enables scalability of block sizes. If not this proposal, an alternative.
Regards, Damian Williamson
Quoted text (1 line)
> I understand that there would be technical issues to resolve in implementation, but, are there no fundamental errors?
Unfortunately your proposal is really fundamentally broken, on a few levels. I think you might need to do a bit more research into how bitcoin works before coming up with such improvements =)
But just some quick notes:
* Every node has a (potentially) different mempool, you can't use it to decide consensus values like the max block size.
* Increasing the entropy in a block to make it more unpredictable doesn't really make sense.
* Bitcoin should be roughly incentive compatible. Your proposal explicits asks miners to ignore their best interests, and confirm transactions by "priority". What are you going to do if a "malicious" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a consensus requirement? And even if you miners can still sort their mempool by fee, and then order the top 1MB by priority.
If you could find a good solution that would allow you to know if miners were following your rule or not (and thus ignore it if it doesn't) then you wouldn't even need bitcoin in the first place.
-Ryan
Quoted text (109 lines)
> -------- Original Message -------- > Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks > Local Time: December 15, 2017 3:42 AM > UTC Time: December 15, 2017 9:42 AM > From: bitcoin-dev@lists•linuxfoundation•org > To: Bitcoin Protocol Discussion <bitcoin-dev@lists•linuxfoundation•org> > > I should not take it that the lack of critical feedback to this revised proposal is a glowing endorsement. I understand that there would be technical issues to resolve in implementation, but, are there no fundamental errors? > > I suppose that it if is difficult to determine how long a transaction has been waiting in the pool then, each node could simply keep track of when a transaction was first seen. This may have implications for a verify routine, however, for example, if a node was offline, how should it differentiate how long each transaction was waiting in that case? If a node was restarted daily would it always think that all transactions had been waiting in the pool less than one day If each node keeps the current transaction pool in a file and updates it, as transactions are included in blocks and, as new transactions appear in the pool, then that would go some way to alleviate the issue, apart from entirely new nodes. There should be no reason the contents of a transaction pool files cannot be shared without agreement as to the transaction pool between nodes, just as nodes transmit new transactions freely. > > It has been questioned why miners could not cheat. For the question of how many transactions to include in a block, I say it is a standoff and miners will conform to the proposal, not wanting to leave transactions with valid fees standing, and, not wanting to shrink the transaction pool. In any case, if miners shrink the transaction pool then I am not immediately concerned since it provides a more efficient service. For the question of including transactions according to the proposal, I say if it is possible to keep track of how long transactions are waiting in the pool so that they can be included on a probability curve then it is possible to verify that blocks conform to the proposal, since the input is a probability, the output should conform to a probability curve. > > If someone has the necessary skill, would anyone be willing to develop the math necessary for the proposal? > > Regards, > Damian Williamson > > --------------------------------------------------------------- > > From: bitcoin-dev-bounces@lists•linuxfoundation•org <bitcoin-dev-bounces@lists•linuxfoundation•org> on behalf of Damian Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> > Sent: Friday, 8 December 2017 8:01 AM > To: bitcoin-dev@lists•linuxfoundation•org > Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks > > Good afternoon, > > The need for this proposal: > > We all must learn to admit that transaction bandwidth is still lurking as a serious issue for the operation, reliability, safety, consumer acceptance, uptake and, for the value of Bitcoin. > > I recently sent a payment which was not urgent so; I chose three-day target confirmation from the fee recommendation. That transaction has still not confirmed after now more than six days - even waiting twice as long seems quite reasonable to me. That transaction is a valid transaction; it is not rubbish, junk or, spam. Under the current model with transaction bandwidth limitation, the longer a transaction waits, the less likely it is ever to confirm due to rising transaction numbers and being pushed back by transactions with rising fees. > > I argue that no transactions are rubbish or junk, only some zero fee transactions might be spam. Having an ever-increasing number of valid transactions that do not confirm as more new transactions with higher fees are created is the opposite of operating a robust, reliable transaction system. > > Business cannot operate with a model where transactions may or may not confirm. Even a business choosing a modest fee has no guarantee that their valid transaction will not be shuffled down by new transactions to the realm of never confirming after it is created. Consumers also will not accept this model as Bitcoin expands. If Bitcoin cannot be a reliable payment system for confirmed transactions then consumers, by and large, will simply not accept the model once they understand. Bitcoin will be a dirty payment system, and this will kill the value of Bitcoin. > > Under the current system, a minority of transactions will eventually be the lucky few who have fees high enough to escape being pushed down the list. > > Once there are more than x transactions (transaction bandwidth limit) every ten minutes, only those choosing twenty-minute confirmation (2 blocks) will have initially at most a fifty percent chance of ever having their payment confirm. Presently, not even using fee recommendations can ensure a sufficiently high fee is paid to ensure transaction confirmation. > > I also argue that the current auction model for limited transaction bandwidth is wrong, is not suitable for a reliable transaction system and, is wrong for Bitcoin. All transactions must confirm in due time. Currently, Bitcoin is not a safe way to send payments. > > I do not believe that consumers and business are against paying fees, even high fees. What is required is operational reliability. > > This great issue needs to be resolved for the safety and reliability of Bitcoin. The time to resolve issues in commerce is before they become great big issues. The time to resolve this issue is now. We must have the foresight to identify and resolve problems before they trip us over. Simply doubling block sizes every so often is reactionary and is not a reliable permanent solution. I have written a BIP proposal for a technical solution but, need your help to write it up to an acceptable standard to be a full BIP. > > I have formatted the following with markdown which is human readable so, I hope nobody minds. I have done as much with this proposal as I feel that I am able so far but continue to take your feedback. > > # BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks > > ## The problem: > Everybody wants value. Miners want to maximize revenue from fees (and we presume, to minimize block size). Consumers need transaction reliability and, (we presume) want low fees. > > The current transaction bandwidth limit is a limiting factor for both. As the operational safety of transactions is limited, so is consumer confidence as they realize the issue and, accordingly, uptake is limited. Fees are artificially inflated due to bandwidth limitations while failing to provide a full confirmation service for all transactions. > > Current fee recommendations provide no satisfaction for transaction reliability and, as Bitcoin scales, this will worsen. > > Bitcoin must be a fully scalable and reliable service, providing full transaction confirmation for every valid transaction. > > The possibility to send a transaction with a fee lower than one that is acceptable to allow eventual transaction confirmation should be removed from the protocol and also from the user interface. > > ## Solution summary: > Provide each transaction with an individual transaction priority each time before choosing transactions to include in the current block, the priority being a function of the fee paid (on a curve), and the time waiting in the transaction pool (also on a curve) out to n days (n=60 ?). The transaction priority to serve as the likelihood of a transaction being included in the current block, and for determining the order in which transactions are tried to see if they will be included. > > Use a target block size. Determine the target block size using; current transaction pool size x ( 1 / (144 x n days ) ) = number of transactions to be included in the current block. Broadcast the next target block size with the current block when it is solved so that nodes know the next target block size for the block that they are building on. > > The curves used for the priority of transactions would have to be appropriate. Perhaps a mathematician with experience in probability can develop the right formulae. My thinking is a steep curve. I suppose that the probability of all transactions should probably account for a sufficient number of inclusions that the target block size is met although, it may not always be. As a suggestion, consider including some zero fee transactions to pad, highest BTC value first? > > **Explanation of the operation of priority:** >> If transaction priority is, for example, a number between one (low) and one-hundred (high) it can be directly understood as the percentage chance in one-hundred of a transaction being included in the block. Using probability or likelihood infers that there is some function of random. If random (100) < transaction priority then the transaction is included. > >>To break it down further, if both the fee on a curve value and the time waiting on a curve value are each a number between one and one-hundred, a rudimentary method may be to simply multiply those two numbers, to find the priority number. For example, a middle fee transaction waiting thirty days (if n = 60 days) may have a value of five for each part (yes, just five, the values are on a curve). When multiplied that will give a priority value of twenty-five, or, a twenty-five percent chance at that moment of being included in the block; it will likely be included in one of the next four blocks, getting more likely each chance. If it is still not included then the value of time waiting will be higher, making for more probability. A very low fee transaction would have a value for the fee of one. It would not be until near sixty-days that the particular low fee transaction has a high likelihood of being included in the block. > > I am not concerned with low (or high) transaction fees, the primary reason for addressing the issue is to ensure transactional reliability and scalability while having each transaction confirm in due time. > > ## Pros: > * Maximizes transaction reliability. > * Fully scalable. > * Maximizes possibility for consumer and business uptake. > * Maximizes total fees paid per block without reducing reliability; because of reliability, in time confidence and overall uptake are greater; therefore, more transactions. > * Market determines fee paid for transaction priority. > * Fee recommendations work all the way out to 30 days or greater. > * Provides additional block entropy; greater security since there is less probability of predicting the next block. > > ## Cons: > * Could initially lower total transaction fees per block. > * Must be first be programmed. > > ## Solution operation: > This is a simplistic view of the operation. The actual operation will need to be determined in a spec for the programmer. > > 1. Determine the target block size for the current block. > 2. Assign a transaction priority to each transaction in the pool. > 3. Select transactions to include in the current block using probability in transaction priority order until the target block size is met. > 5. Solve block. > 6. Broadcast the next target block size with the current block when it is solved. > 7. Block is received. > 8. Block verification process. > 9. Accept/reject block based on verification result. > 10. Repeat. > > ## Closing comments: > It may be possible to verify blocks conform to the proposal by showing that the probability for all transactions included in the block statistically conforms to a probability distribution curve, *if* the individual transaction priority can be recreated. I am not that deep into the mathematics; however, it may also be possible to use a similar method to do this just based on the fee, that statistically, the blocks conform to a fee distribution. Any zero fee transactions would have to be ignored. This solution needs a clever mathematician. > > I implore, at the very least, that we use some method that validates full transaction reliability and enables scalability of block sizes. If not this proposal, an alternative. > > Regards, > Damian Williamson
There are really two separate problems to solve.
1. How does Bitcoin scale with fixed block size? 2. How do we ensure that all valid transactions are eventually included in the blockchain?
Those are the two issues that the proposal attempts to address. It makes sense to resolve these two problems together. Using the proposed system for variable block sizes would solve the first problem but there would still be a whole bunch of never confirming transactions. I am not sure how to reliably solve the second problem at scale without first solving the first.
Quoted text (1 line)
>* Every node has a (potentially) different mempool, you can't use it to decide consensus values like the max block size.
I do not suggest a consensus. Depending on which node solves a block the value for next block size will be different. The consensus would be that blocks will adhere to the next block size value transmitted with the current block. It is easy to verify that the consensus is being adhered to once in place.
Quoted text (1 line)
>* Increasing the entropy in a block to make it more unpredictable doesn't really make sense.
Not a necessary function, just an effect of using a probability-based distribution.
Quoted text (1 line)
>* Bitcoin should be roughly incentive compatible. Your proposal explicits asks miners to ignore their best interests, and confirm transactions by "priority". What are you going to do if a "malicious" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a consensus requirement? And even if you miners can still sort their mempool by fee, and then order the top 1MB by priority.
I entirely agree with your sentiment that Bitcoin must be incentive compatible. It is necessary.
It is in only miners immediate interest to make the most profitable block from the available transaction pool. As with so many other things, it is necessary to partially ignore short-term gain for long-term benefit. It is in miners and everybody's long-term interest to have a reliable transaction service. A busy transaction service that confirms lots of transactions per hour will become more profitable as demand increases and more users are prepared to pay for priority. As it is there is currently no way to fully scale because of the transaction bandwidth limit and that is problematic. If all valid transactions must eventually confirm then there must be a way to resolve that problem.
Bitcoin deliberately removes traditional scale by ensuring blocks take ten minutes on average to solve, an ingenious idea and, incentive compatible but, fixed block sizes leaves us with a problem to solve when we want to scale.
Quoted text (1 line)
>If you could find a good solution that would allow you to know if miners were following your rule or not (and thus ignore it if it doesn't) then you wouldn't even need bitcoin in the first place.
I am confident that the math to verify blocks based on the proposal can be developed (and I think it will not be too complex for a mathematician with the relevant experience), however, I am nowhere near experienced enough with probability and statistical analysis to do it. Yes, if Bitcoin doesn't then it might make another great opportunity for an altcoin but I am not even nearly interested in promoting any altcoins.
If not the proposal that I have put forward, then, hopefully, someone can come up with a better solution. The important thing is that the issues are resolved.
Regards,
Damian Williamson
________________________________ From: Rhavar <rhavar@protonmail•com> Sent: Saturday, 16 December 2017 3:38 AM To: Damian Williamson Cc: Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Quoted text (1 line)
> I understand that there would be technical issues to resolve in implementation, but, are there no fundamental errors?
Unfortunately your proposal is really fundamentally broken, on a few levels. I think you might need to do a bit more research into how bitcoin works before coming up with such improvements =)
But just some quick notes:
* Every node has a (potentially) different mempool, you can't use it to decide consensus values like the max block size.
* Increasing the entropy in a block to make it more unpredictable doesn't really make sense.
* Bitcoin should be roughly incentive compatible. Your proposal explicits asks miners to ignore their best interests, and confirm transactions by "priority". What are you going to do if a "malicious" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a consensus requirement? And even if you miners can still sort their mempool by fee, and then order the top 1MB by priority.
If you could find a good solution that would allow you to know if miners were following your rule or not (and thus ignore it if it doesn't) then you wouldn't even need bitcoin in the first place.
-Ryan
-------- Original Message -------- Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks Local Time: December 15, 2017 3:42 AM UTC Time: December 15, 2017 9:42 AM From: bitcoin-dev@lists•linuxfoundation•org To: Bitcoin Protocol Discussion <bitcoin-dev@lists•linuxfoundation•org>
I should not take it that the lack of critical feedback to this revised proposal is a glowing endorsement. I understand that there would be technical issues to resolve in implementation, but, are there no fundamental errors?
I suppose that it if is difficult to determine how long a transaction has been waiting in the pool then, each node could simply keep track of when a transaction was first seen. This may have implications for a verify routine, however, for example, if a node was offline, how should it differentiate how long each transaction was waiting in that case? If a node was restarted daily would it always think that all transactions had been waiting in the pool less than one day If each node keeps the current transaction pool in a file and updates it, as transactions are included in blocks and, as new transactions appear in the pool, then that would go some way to alleviate the issue, apart from entirely new nodes. There should be no reason the contents of a transaction pool files cannot be shared without agreement as to the transaction pool between nodes, just as nodes transmit new transactions freely.
It has been questioned why miners could not cheat. For the question of how many transactions to include in a block, I say it is a standoff and miners will conform to the proposal, not wanting to leave transactions with valid fees standing, and, not wanting to shrink the transaction pool. In any case, if miners shrink the transaction pool then I am not immediately concerned since it provides a more efficient service. For the question of including transactions according to the proposal, I say if it is possible to keep track of how long transactions are waiting in the pool so that they can be included on a probability curve then it is possible to verify that blocks conform to the proposal, since the input is a probability, the output should conform to a probability curve.
If someone has the necessary skill, would anyone be willing to develop the math necessary for the proposal?
Regards, Damian Williamson
________________________________
From: bitcoin-dev-bounces@lists•linuxfoundation•org <bitcoin-dev-bounces@lists•linuxfoundation•org> on behalf of Damian Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> Sent: Friday, 8 December 2017 8:01 AM To: bitcoin-dev@lists•linuxfoundation•org Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Good afternoon,
The need for this proposal:
We all must learn to admit that transaction bandwidth is still lurking as a serious issue for the operation, reliability, safety, consumer acceptance, uptake and, for the value of Bitcoin.
I recently sent a payment which was not urgent so; I chose three-day target confirmation from the fee recommendation. That transaction has still not confirmed after now more than six days - even waiting twice as long seems quite reasonable to me. That transaction is a valid transaction; it is not rubbish, junk or, spam. Under the current model with transaction bandwidth limitation, the longer a transaction waits, the less likely it is ever to confirm due to rising transaction numbers and being pushed back by transactions with rising fees.
I argue that no transactions are rubbish or junk, only some zero fee transactions might be spam. Having an ever-increasing number of valid transactions that do not confirm as more new transactions with higher fees are created is the opposite of operating a robust, reliable transaction system.
Business cannot operate with a model where transactions may or may not confirm. Even a business choosing a modest fee has no guarantee that their valid transaction will not be shuffled down by new transactions to the realm of never confirming after it is created. Consumers also will not accept this model as Bitcoin expands. If Bitcoin cannot be a reliable payment system for confirmed transactions then consumers, by and large, will simply not accept the model once they understand. Bitcoin will be a dirty payment system, and this will kill the value of Bitcoin.
Under the current system, a minority of transactions will eventually be the lucky few who have fees high enough to escape being pushed down the list.
Once there are more than x transactions (transaction bandwidth limit) every ten minutes, only those choosing twenty-minute confirmation (2 blocks) will have initially at most a fifty percent chance of ever having their payment confirm. Presently, not even using fee recommendations can ensure a sufficiently high fee is paid to ensure transaction confirmation.
I also argue that the current auction model for limited transaction bandwidth is wrong, is not suitable for a reliable transaction system and, is wrong for Bitcoin. All transactions must confirm in due time. Currently, Bitcoin is not a safe way to send payments.
I do not believe that consumers and business are against paying fees, even high fees. What is required is operational reliability.
This great issue needs to be resolved for the safety and reliability of Bitcoin. The time to resolve issues in commerce is before they become great big issues. The time to resolve this issue is now. We must have the foresight to identify and resolve problems before they trip us over. Simply doubling block sizes every so often is reactionary and is not a reliable permanent solution. I have written a BIP proposal for a technical solution but, need your help to write it up to an acceptable standard to be a full BIP.
I have formatted the following with markdown which is human readable so, I hope nobody minds. I have done as much with this proposal as I feel that I am able so far but continue to take your feedback.
# BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
## The problem: Everybody wants value. Miners want to maximize revenue from fees (and we presume, to minimize block size). Consumers need transaction reliability and, (we presume) want low fees.
The current transaction bandwidth limit is a limiting factor for both. As the operational safety of transactions is limited, so is consumer confidence as they realize the issue and, accordingly, uptake is limited. Fees are artificially inflated due to bandwidth limitations while failing to provide a full confirmation service for all transactions.
Current fee recommendations provide no satisfaction for transaction reliability and, as Bitcoin scales, this will worsen.
Bitcoin must be a fully scalable and reliable service, providing full transaction confirmation for every valid transaction.
The possibility to send a transaction with a fee lower than one that is acceptable to allow eventual transaction confirmation should be removed from the protocol and also from the user interface.
## Solution summary: Provide each transaction with an individual transaction priority each time before choosing transactions to include in the current block, the priority being a function of the fee paid (on a curve), and the time waiting in the transaction pool (also on a curve) out to n days (n=60 ?). The transaction priority to serve as the likelihood of a transaction being included in the current block, and for determining the order in which transactions are tried to see if they will be included.
Use a target block size. Determine the target block size using; current transaction pool size x ( 1 / (144 x n days ) ) = number of transactions to be included in the current block. Broadcast the next target block size with the current block when it is solved so that nodes know the next target block size for the block that they are building on.
The curves used for the priority of transactions would have to be appropriate. Perhaps a mathematician with experience in probability can develop the right formulae. My thinking is a steep curve. I suppose that the probability of all transactions should probably account for a sufficient number of inclusions that the target block size is met although, it may not always be. As a suggestion, consider including some zero fee transactions to pad, highest BTC value first?
**Explanation of the operation of priority:**
Quoted text (2 lines)
> If transaction priority is, for example, a number between one (low) and one-hundred (high) it can be directly understood as the percentage chance in one-hundred of a transaction being included in the block. Using probability or likelihood infers that there is some function of random. If random (100) < transaction priority then the transaction is included. >To break it down further, if both the fee on a curve value and the time waiting on a curve value are each a number between one and one-hundred, a rudimentary method may be to simply multiply those two numbers, to find the priority number. For example, a middle fee transaction waiting thirty days (if n = 60 days) may have a value of five for each part (yes, just five, the values are on a curve). When multiplied that will give a priority value of twenty-five, or, a twenty-five percent chance at that moment of being included in the block; it will likely be included in one of the next four blocks, getting more likely each chance. If it is still not included then the value of time waiting will be higher, making for more probability. A very low fee transaction would have a value for the fee of one. It would not be until near sixty-days that the particular low fee transaction has a high likelihood of being included in the block.
I am not concerned with low (or high) transaction fees, the primary reason for addressing the issue is to ensure transactional reliability and scalability while having each transaction confirm in due time.
## Pros: * Maximizes transaction reliability. * Fully scalable. * Maximizes possibility for consumer and business uptake. * Maximizes total fees paid per block without reducing reliability; because of reliability, in time confidence and overall uptake are greater; therefore, more transactions. * Market determines fee paid for transaction priority. * Fee recommendations work all the way out to 30 days or greater. * Provides additional block entropy; greater security since there is less probability of predicting the next block.
## Cons: * Could initially lower total transaction fees per block. * Must be first be programmed.
## Solution operation: This is a simplistic view of the operation. The actual operation will need to be determined in a spec for the programmer.
1. Determine the target block size for the current block. 2. Assign a transaction priority to each transaction in the pool. 3. Select transactions to include in the current block using probability in transaction priority order until the target block size is met. 5. Solve block. 6. Broadcast the next target block size with the current block when it is solved. 7. Block is received. 8. Block verification process. 9. Accept/reject block based on verification result. 10. Repeat.
## Closing comments: It may be possible to verify blocks conform to the proposal by showing that the probability for all transactions included in the block statistically conforms to a probability distribution curve, *if* the individual transaction priority can be recreated. I am not that deep into the mathematics; however, it may also be possible to use a similar method to do this just based on the fee, that statistically, the blocks conform to a fee distribution. Any zero fee transactions would have to be ignored. This solution needs a clever mathematician.
I implore, at the very least, that we use some method that validates full transaction reliability and enables scalability of block sizes. If not this proposal, an alternative.
Regards, Damian Williamson
I do not know why people make the leap that the proposal requires a consensus on the transaction pool. It does not.
It may be helpful to have the discussion from the previous thread linked here.
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-December/015370.html
Where I speak of validating that a block conforms to the broadcast next block size, I do not propose validating the number broadcast for the next block size itself, only that the next generated block is that size.
Regards,
Damian Williamson
________________________________ From: Damian Williamson <willtech@live•com•au> Sent: Saturday, 16 December 2017 7:59 AM To: Rhavar Cc: Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
There are really two separate problems to solve.
1. How does Bitcoin scale with fixed block size? 2. How do we ensure that all valid transactions are eventually included in the blockchain?
Those are the two issues that the proposal attempts to address. It makes sense to resolve these two problems together. Using the proposed system for variable block sizes would solve the first problem but there would still be a whole bunch of never confirming transactions. I am not sure how to reliably solve the second problem at scale without first solving the first.
Quoted text (1 line)
>* Every node has a (potentially) different mempool, you can't use it to decide consensus values like the max block size.
I do not suggest a consensus. Depending on which node solves a block the value for next block size will be different. The consensus would be that blocks will adhere to the next block size value transmitted with the current block. It is easy to verify that the consensus is being adhered to once in place.
Quoted text (1 line)
>* Increasing the entropy in a block to make it more unpredictable doesn't really make sense.
Not a necessary function, just an effect of using a probability-based distribution.
Quoted text (1 line)
>* Bitcoin should be roughly incentive compatible. Your proposal explicits asks miners to ignore their best interests, and confirm transactions by "priority". What are you going to do if a "malicious" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a consensus requirement? And even if you miners can still sort their mempool by fee, and then order the top 1MB by priority.
I entirely agree with your sentiment that Bitcoin must be incentive compatible. It is necessary.
It is in only miners immediate interest to make the most profitable block from the available transaction pool. As with so many other things, it is necessary to partially ignore short-term gain for long-term benefit. It is in miners and everybody's long-term interest to have a reliable transaction service. A busy transaction service that confirms lots of transactions per hour will become more profitable as demand increases and more users are prepared to pay for priority. As it is there is currently no way to fully scale because of the transaction bandwidth limit and that is problematic. If all valid transactions must eventually confirm then there must be a way to resolve that problem.
Bitcoin deliberately removes traditional scale by ensuring blocks take ten minutes on average to solve, an ingenious idea and, incentive compatible but, fixed block sizes leaves us with a problem to solve when we want to scale.
Quoted text (1 line)
>If you could find a good solution that would allow you to know if miners were following your rule or not (and thus ignore it if it doesn't) then you wouldn't even need bitcoin in the first place.
I am confident that the math to verify blocks based on the proposal can be developed (and I think it will not be too complex for a mathematician with the relevant experience), however, I am nowhere near experienced enough with probability and statistical analysis to do it. Yes, if Bitcoin doesn't then it might make another great opportunity for an altcoin but I am not even nearly interested in promoting any altcoins.
If not the proposal that I have put forward, then, hopefully, someone can come up with a better solution. The important thing is that the issues are resolved.
Regards,
Damian Williamson
________________________________ From: Rhavar <rhavar@protonmail•com> Sent: Saturday, 16 December 2017 3:38 AM To: Damian Williamson Cc: Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Quoted text (1 line)
> I understand that there would be technical issues to resolve in implementation, but, are there no fundamental errors?
Unfortunately your proposal is really fundamentally broken, on a few levels. I think you might need to do a bit more research into how bitcoin works before coming up with such improvements =)
But just some quick notes:
* Every node has a (potentially) different mempool, you can't use it to decide consensus values like the max block size.
* Increasing the entropy in a block to make it more unpredictable doesn't really make sense.
* Bitcoin should be roughly incentive compatible. Your proposal explicits asks miners to ignore their best interests, and confirm transactions by "priority". What are you going to do if a "malicious" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a consensus requirement? And even if you miners can still sort their mempool by fee, and then order the top 1MB by priority.
If you could find a good solution that would allow you to know if miners were following your rule or not (and thus ignore it if it doesn't) then you wouldn't even need bitcoin in the first place.
-Ryan
-------- Original Message -------- Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks Local Time: December 15, 2017 3:42 AM UTC Time: December 15, 2017 9:42 AM From: bitcoin-dev@lists•linuxfoundation•org To: Bitcoin Protocol Discussion <bitcoin-dev@lists•linuxfoundation•org>
I should not take it that the lack of critical feedback to this revised proposal is a glowing endorsement. I understand that there would be technical issues to resolve in implementation, but, are there no fundamental errors?
I suppose that it if is difficult to determine how long a transaction has been waiting in the pool then, each node could simply keep track of when a transaction was first seen. This may have implications for a verify routine, however, for example, if a node was offline, how should it differentiate how long each transaction was waiting in that case? If a node was restarted daily would it always think that all transactions had been waiting in the pool less than one day If each node keeps the current transaction pool in a file and updates it, as transactions are included in blocks and, as new transactions appear in the pool, then that would go some way to alleviate the issue, apart from entirely new nodes. There should be no reason the contents of a transaction pool files cannot be shared without agreement as to the transaction pool between nodes, just as nodes transmit new transactions freely.
It has been questioned why miners could not cheat. For the question of how many transactions to include in a block, I say it is a standoff and miners will conform to the proposal, not wanting to leave transactions with valid fees standing, and, not wanting to shrink the transaction pool. In any case, if miners shrink the transaction pool then I am not immediately concerned since it provides a more efficient service. For the question of including transactions according to the proposal, I say if it is possible to keep track of how long transactions are waiting in the pool so that they can be included on a probability curve then it is possible to verify that blocks conform to the proposal, since the input is a probability, the output should conform to a probability curve.
If someone has the necessary skill, would anyone be willing to develop the math necessary for the proposal?
Regards, Damian Williamson
________________________________
From: bitcoin-dev-bounces@lists•linuxfoundation•org <bitcoin-dev-bounces@lists•linuxfoundation•org> on behalf of Damian Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> Sent: Friday, 8 December 2017 8:01 AM To: bitcoin-dev@lists•linuxfoundation•org Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Good afternoon,
The need for this proposal:
We all must learn to admit that transaction bandwidth is still lurking as a serious issue for the operation, reliability, safety, consumer acceptance, uptake and, for the value of Bitcoin.
I recently sent a payment which was not urgent so; I chose three-day target confirmation from the fee recommendation. That transaction has still not confirmed after now more than six days - even waiting twice as long seems quite reasonable to me. That transaction is a valid transaction; it is not rubbish, junk or, spam. Under the current model with transaction bandwidth limitation, the longer a transaction waits, the less likely it is ever to confirm due to rising transaction numbers and being pushed back by transactions with rising fees.
I argue that no transactions are rubbish or junk, only some zero fee transactions might be spam. Having an ever-increasing number of valid transactions that do not confirm as more new transactions with higher fees are created is the opposite of operating a robust, reliable transaction system.
Business cannot operate with a model where transactions may or may not confirm. Even a business choosing a modest fee has no guarantee that their valid transaction will not be shuffled down by new transactions to the realm of never confirming after it is created. Consumers also will not accept this model as Bitcoin expands. If Bitcoin cannot be a reliable payment system for confirmed transactions then consumers, by and large, will simply not accept the model once they understand. Bitcoin will be a dirty payment system, and this will kill the value of Bitcoin.
Under the current system, a minority of transactions will eventually be the lucky few who have fees high enough to escape being pushed down the list.
Once there are more than x transactions (transaction bandwidth limit) every ten minutes, only those choosing twenty-minute confirmation (2 blocks) will have initially at most a fifty percent chance of ever having their payment confirm. Presently, not even using fee recommendations can ensure a sufficiently high fee is paid to ensure transaction confirmation.
I also argue that the current auction model for limited transaction bandwidth is wrong, is not suitable for a reliable transaction system and, is wrong for Bitcoin. All transactions must confirm in due time. Currently, Bitcoin is not a safe way to send payments.
I do not believe that consumers and business are against paying fees, even high fees. What is required is operational reliability.
This great issue needs to be resolved for the safety and reliability of Bitcoin. The time to resolve issues in commerce is before they become great big issues. The time to resolve this issue is now. We must have the foresight to identify and resolve problems before they trip us over. Simply doubling block sizes every so often is reactionary and is not a reliable permanent solution. I have written a BIP proposal for a technical solution but, need your help to write it up to an acceptable standard to be a full BIP.
I have formatted the following with markdown which is human readable so, I hope nobody minds. I have done as much with this proposal as I feel that I am able so far but continue to take your feedback.
# BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
## The problem: Everybody wants value. Miners want to maximize revenue from fees (and we presume, to minimize block size). Consumers need transaction reliability and, (we presume) want low fees.
The current transaction bandwidth limit is a limiting factor for both. As the operational safety of transactions is limited, so is consumer confidence as they realize the issue and, accordingly, uptake is limited. Fees are artificially inflated due to bandwidth limitations while failing to provide a full confirmation service for all transactions.
Current fee recommendations provide no satisfaction for transaction reliability and, as Bitcoin scales, this will worsen.
Bitcoin must be a fully scalable and reliable service, providing full transaction confirmation for every valid transaction.
The possibility to send a transaction with a fee lower than one that is acceptable to allow eventual transaction confirmation should be removed from the protocol and also from the user interface.
## Solution summary: Provide each transaction with an individual transaction priority each time before choosing transactions to include in the current block, the priority being a function of the fee paid (on a curve), and the time waiting in the transaction pool (also on a curve) out to n days (n=60 ?). The transaction priority to serve as the likelihood of a transaction being included in the current block, and for determining the order in which transactions are tried to see if they will be included.
Use a target block size. Determine the target block size using; current transaction pool size x ( 1 / (144 x n days ) ) = number of transactions to be included in the current block. Broadcast the next target block size with the current block when it is solved so that nodes know the next target block size for the block that they are building on.
The curves used for the priority of transactions would have to be appropriate. Perhaps a mathematician with experience in probability can develop the right formulae. My thinking is a steep curve. I suppose that the probability of all transactions should probably account for a sufficient number of inclusions that the target block size is met although, it may not always be. As a suggestion, consider including some zero fee transactions to pad, highest BTC value first?
**Explanation of the operation of priority:**
Quoted text (2 lines)
> If transaction priority is, for example, a number between one (low) and one-hundred (high) it can be directly understood as the percentage chance in one-hundred of a transaction being included in the block. Using probability or likelihood infers that there is some function of random. If random (100) < transaction priority then the transaction is included. >To break it down further, if both the fee on a curve value and the time waiting on a curve value are each a number between one and one-hundred, a rudimentary method may be to simply multiply those two numbers, to find the priority number. For example, a middle fee transaction waiting thirty days (if n = 60 days) may have a value of five for each part (yes, just five, the values are on a curve). When multiplied that will give a priority value of twenty-five, or, a twenty-five percent chance at that moment of being included in the block; it will likely be included in one of the next four blocks, getting more likely each chance. If it is still not included then the value of time waiting will be higher, making for more probability. A very low fee transaction would have a value for the fee of one. It would not be until near sixty-days that the particular low fee transaction has a high likelihood of being included in the block.
I am not concerned with low (or high) transaction fees, the primary reason for addressing the issue is to ensure transactional reliability and scalability while having each transaction confirm in due time.
## Pros: * Maximizes transaction reliability. * Fully scalable. * Maximizes possibility for consumer and business uptake. * Maximizes total fees paid per block without reducing reliability; because of reliability, in time confidence and overall uptake are greater; therefore, more transactions. * Market determines fee paid for transaction priority. * Fee recommendations work all the way out to 30 days or greater. * Provides additional block entropy; greater security since there is less probability of predicting the next block.
## Cons: * Could initially lower total transaction fees per block. * Must be first be programmed.
## Solution operation: This is a simplistic view of the operation. The actual operation will need to be determined in a spec for the programmer.
1. Determine the target block size for the current block. 2. Assign a transaction priority to each transaction in the pool. 3. Select transactions to include in the current block using probability in transaction priority order until the target block size is met. 5. Solve block. 6. Broadcast the next target block size with the current block when it is solved. 7. Block is received. 8. Block verification process. 9. Accept/reject block based on verification result. 10. Repeat.
## Closing comments: It may be possible to verify blocks conform to the proposal by showing that the probability for all transactions included in the block statistically conforms to a probability distribution curve, *if* the individual transaction priority can be recreated. I am not that deep into the mathematics; however, it may also be possible to use a similar method to do this just based on the fee, that statistically, the blocks conform to a fee distribution. Any zero fee transactions would have to be ignored. This solution needs a clever mathematician.
I implore, at the very least, that we use some method that validates full transaction reliability and enables scalability of block sizes. If not this proposal, an alternative.
Regards, Damian Williamson
Regarding "problem" #2 where you say "How do we ensure that all valid transactions are eventually included in the blockchain?": I do not believe that all people would (a) agree this is a problem or (b) that we do want to *ENSURE* that *ALL* valid transactions are eventually included in the blockchain. There are many *valid* transactions that oftentimes miners do not (and should not) wish to require be confirmed and included in the blockchain. Spam transactions for example can be valid, but used to attack bitcoin by using no or low fee. Any valid transaction MAY be included by a miner, but requiring it in some fashion at this point would open the network to other attack vectors. Perhaps you meant it a different way.
On Fri, Dec 15, 2017 at 3:59 PM, Damian Williamson via bitcoin-dev < bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (6 lines)
> > There are really two separate problems to solve. > > > How does Bitcoin scale with fixed block size? > How do we ensure that all valid transactions are eventually included in
the blockchain?
Quoted text (3 lines)
> > > Those are the two issues that the proposal attempts to address. It makes
sense to resolve these two problems together. Using the proposed system for variable block sizes would solve the first problem but there would still be a whole bunch of never confirming transactions. I am not sure how to reliably solve the second problem at scale without first solving the first.
Quoted text (3 lines)
> > > >* Every node has a (potentially) different mempool, you can't use it to
decide consensus values like the max block size.
Quoted text (3 lines)
> > > I do not suggest a consensus. Depending on which node solves a block the
value for next block size will be different. The consensus would be that blocks will adhere to the next block size value transmitted with the current block. It is easy to verify that the consensus is being adhered to once in place.
Quoted text (2 lines)
> > >* Increasing the entropy in a block to make it more unpredictable
doesn't really make sense.
Quoted text (2 lines)
> > Not a necessary function, just an effect of using a probability-based
distribution.
Quoted text (2 lines)
> > >* Bitcoin should be roughly incentive compatible. Your proposal
explicits asks miners to ignore their best interests, and confirm transactions by "priority". What are you going to do if a "malicious" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a consensus requirement? And even if you miners can still sort their mempool by fee, and then order the top 1MB by priority.
Quoted text (2 lines)
> > I entirely agree with your sentiment that Bitcoin must be incentive
compatible. It is necessary.
Quoted text (2 lines)
> > It is in only miners immediate interest to make the most profitable block
from the available transaction pool. As with so many other things, it is necessary to partially ignore short-term gain for long-term benefit. It is in miners and everybody's long-term interest to have a reliable transaction service. A busy transaction service that confirms lots of transactions per hour will become more profitable as demand increases and more users are prepared to pay for priority. As it is there is currently no way to fully scale because of the transaction bandwidth limit and that is problematic. If all valid transactions must eventually confirm then there must be a way to resolve that problem.
Quoted text (2 lines)
> > Bitcoin deliberately removes traditional scale by ensuring blocks take
ten minutes on average to solve, an ingenious idea and, incentive compatible but, fixed block sizes leaves us with a problem to solve when we want to scale.
Quoted text (2 lines)
> > >If you could find a good solution that would allow you to know if miners
were following your rule or not (and thus ignore it if it doesn't) then you wouldn't even need bitcoin in the first place.
Quoted text (2 lines)
> > I am confident that the math to verify blocks based on the proposal can
be developed (and I think it will not be too complex for a mathematician with the relevant experience), however, I am nowhere near experienced enough with probability and statistical analysis to do it. Yes, if Bitcoin doesn't then it might make another great opportunity for an altcoin but I am not even nearly interested in promoting any altcoins.
Quoted text (3 lines)
> > > If not the proposal that I have put forward, then, hopefully, someone can
come up with a better solution. The important thing is that the issues are resolved.
Quoted text (14 lines)
> > > Regards, > > Damian Williamson > > > > ________________________________ > From: Rhavar <rhavar@protonmail•com> > Sent: Saturday, 16 December 2017 3:38 AM > To: Damian Williamson > Cc: Bitcoin Protocol Discussion > Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use
Transaction Priority For Ordering Transactions In Blocks
Quoted text (2 lines)
> > > I understand that there would be technical issues to resolve in
implementation, but, are there no fundamental errors?
Quoted text (2 lines)
> > Unfortunately your proposal is really fundamentally broken, on a few
levels. I think you might need to do a bit more research into how bitcoin works before coming up with such improvements =)
Quoted text (4 lines)
> > But just some quick notes: > > * Every node has a (potentially) different mempool, you can't use it to
decide consensus values like the max block size.
Quoted text (2 lines)
> > * Increasing the entropy in a block to make it more unpredictable doesn't
really make sense.
Quoted text (2 lines)
> > * Bitcoin should be roughly incentive compatible. Your proposal explicits
asks miners to ignore their best interests, and confirm transactions by "priority". What are you going to do if a "malicious" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a consensus requirement? And even if you miners can still sort their mempool by fee, and then order the top 1MB by priority.
Quoted text (2 lines)
> > If you could find a good solution that would allow you to know if miners
were following your rule or not (and thus ignore it if it doesn't) then you wouldn't even need bitcoin in the first place.
Quoted text (9 lines)
> > > > > -Ryan > > > -------- Original Message -------- > Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction
Priority For Ordering Transactions In Blocks
Quoted text (8 lines)
> Local Time: December 15, 2017 3:42 AM > UTC Time: December 15, 2017 9:42 AM > From: bitcoin-dev@lists•linuxfoundation•org > To: Bitcoin Protocol Discussion <bitcoin-dev@lists•linuxfoundation•org> > > > > I should not take it that the lack of critical feedback to this revised
proposal is a glowing endorsement. I understand that there would be technical issues to resolve in implementation, but, are there no fundamental errors?
Quoted text (2 lines)
> > I suppose that it if is difficult to determine how long a transaction has
been waiting in the pool then, each node could simply keep track of when a transaction was first seen. This may have implications for a verify routine, however, for example, if a node was offline, how should it differentiate how long each transaction was waiting in that case? If a node was restarted daily would it always think that all transactions had been waiting in the pool less than one day If each node keeps the current transaction pool in a file and updates it, as transactions are included in blocks and, as new transactions appear in the pool, then that would go some way to alleviate the issue, apart from entirely new nodes. There should be no reason the contents of a transaction pool files cannot be shared without agreement as to the transaction pool between nodes, just as nodes transmit new transactions freely.
Quoted text (2 lines)
> > It has been questioned why miners could not cheat. For the question of
how many transactions to include in a block, I say it is a standoff and miners will conform to the proposal, not wanting to leave transactions with valid fees standing, and, not wanting to shrink the transaction pool. In any case, if miners shrink the transaction pool then I am not immediately concerned since it provides a more efficient service. For the question of including transactions according to the proposal, I say if it is possible to keep track of how long transactions are waiting in the pool so that they can be included on a probability curve then it is possible to verify that blocks conform to the proposal, since the input is a probability, the output should conform to a probability curve.
Quoted text (3 lines)
> > > If someone has the necessary skill, would anyone be willing to develop
the math necessary for the proposal?
Quoted text (8 lines)
> > Regards, > Damian Williamson > > > ________________________________ > > From: bitcoin-dev-bounces@lists•linuxfoundation•org <
bitcoin-dev-bounces@lists•linuxfoundation•org> on behalf of Damian Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org>
Quoted text (3 lines)
> Sent: Friday, 8 December 2017 8:01 AM > To: bitcoin-dev@lists•linuxfoundation•org > Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction
Priority For Ordering Transactions In Blocks
Quoted text (8 lines)
> > > > Good afternoon, > > The need for this proposal: > > We all must learn to admit that transaction bandwidth is still lurking as
a serious issue for the operation, reliability, safety, consumer acceptance, uptake and, for the value of Bitcoin.
Quoted text (2 lines)
> > I recently sent a payment which was not urgent so; I chose three-day
target confirmation from the fee recommendation. That transaction has still not confirmed after now more than six days - even waiting twice as long seems quite reasonable to me. That transaction is a valid transaction; it is not rubbish, junk or, spam. Under the current model with transaction bandwidth limitation, the longer a transaction waits, the less likely it is ever to confirm due to rising transaction numbers and being pushed back by transactions with rising fees.
Quoted text (2 lines)
> > I argue that no transactions are rubbish or junk, only some zero fee
transactions might be spam. Having an ever-increasing number of valid transactions that do not confirm as more new transactions with higher fees are created is the opposite of operating a robust, reliable transaction system.
Quoted text (2 lines)
> > Business cannot operate with a model where transactions may or may not
confirm. Even a business choosing a modest fee has no guarantee that their valid transaction will not be shuffled down by new transactions to the realm of never confirming after it is created. Consumers also will not accept this model as Bitcoin expands. If Bitcoin cannot be a reliable payment system for confirmed transactions then consumers, by and large, will simply not accept the model once they understand. Bitcoin will be a dirty payment system, and this will kill the value of Bitcoin.
Quoted text (2 lines)
> > Under the current system, a minority of transactions will eventually be
the lucky few who have fees high enough to escape being pushed down the list.
Quoted text (2 lines)
> > Once there are more than x transactions (transaction bandwidth limit)
every ten minutes, only those choosing twenty-minute confirmation (2 blocks) will have initially at most a fifty percent chance of ever having their payment confirm. Presently, not even using fee recommendations can ensure a sufficiently high fee is paid to ensure transaction confirmation.
Quoted text (2 lines)
> > I also argue that the current auction model for limited transaction
bandwidth is wrong, is not suitable for a reliable transaction system and, is wrong for Bitcoin. All transactions must confirm in due time. Currently, Bitcoin is not a safe way to send payments.
Quoted text (2 lines)
> > I do not believe that consumers and business are against paying fees,
even high fees. What is required is operational reliability.
Quoted text (2 lines)
> > This great issue needs to be resolved for the safety and reliability of
Bitcoin. The time to resolve issues in commerce is before they become great big issues. The time to resolve this issue is now. We must have the foresight to identify and resolve problems before they trip us over. Simply doubling block sizes every so often is reactionary and is not a reliable permanent solution. I have written a BIP proposal for a technical solution but, need your help to write it up to an acceptable standard to be a full BIP.
Quoted text (2 lines)
> > I have formatted the following with markdown which is human readable so,
I hope nobody minds. I have done as much with this proposal as I feel that I am able so far but continue to take your feedback.
Quoted text (2 lines)
> > # BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering
Transactions In Blocks
Quoted text (3 lines)
> > ## The problem: > Everybody wants value. Miners want to maximize revenue from fees (and we
presume, to minimize block size). Consumers need transaction reliability and, (we presume) want low fees.
Quoted text (2 lines)
> > The current transaction bandwidth limit is a limiting factor for both. As
the operational safety of transactions is limited, so is consumer confidence as they realize the issue and, accordingly, uptake is limited. Fees are artificially inflated due to bandwidth limitations while failing to provide a full confirmation service for all transactions.
Quoted text (2 lines)
> > Current fee recommendations provide no satisfaction for transaction
reliability and, as Bitcoin scales, this will worsen.
Quoted text (2 lines)
> > Bitcoin must be a fully scalable and reliable service, providing full
transaction confirmation for every valid transaction.
Quoted text (2 lines)
> > The possibility to send a transaction with a fee lower than one that is
acceptable to allow eventual transaction confirmation should be removed from the protocol and also from the user interface.
Quoted text (3 lines)
> > ## Solution summary: > Provide each transaction with an individual transaction priority each
time before choosing transactions to include in the current block, the priority being a function of the fee paid (on a curve), and the time waiting in the transaction pool (also on a curve) out to n days (n=60 ?). The transaction priority to serve as the likelihood of a transaction being included in the current block, and for determining the order in which transactions are tried to see if they will be included.
Quoted text (2 lines)
> > Use a target block size. Determine the target block size using; current
transaction pool size x ( 1 / (144 x n days ) ) = number of transactions to be included in the current block. Broadcast the next target block size with the current block when it is solved so that nodes know the next target block size for the block that they are building on.
Quoted text (2 lines)
> > The curves used for the priority of transactions would have to be
appropriate. Perhaps a mathematician with experience in probability can develop the right formulae. My thinking is a steep curve. I suppose that the probability of all transactions should probably account for a sufficient number of inclusions that the target block size is met although, it may not always be. As a suggestion, consider including some zero fee transactions to pad, highest BTC value first?
Quoted text (3 lines)
> > **Explanation of the operation of priority:** > > If transaction priority is, for example, a number between one (low) and
one-hundred (high) it can be directly understood as the percentage chance in one-hundred of a transaction being included in the block. Using probability or likelihood infers that there is some function of random. If random (100) < transaction priority then the transaction is included.
Quoted text (2 lines)
> > >To break it down further, if both the fee on a curve value and the time
waiting on a curve value are each a number between one and one-hundred, a rudimentary method may be to simply multiply those two numbers, to find the priority number. For example, a middle fee transaction waiting thirty days (if n = 60 days) may have a value of five for each part (yes, just five, the values are on a curve). When multiplied that will give a priority value of twenty-five, or, a twenty-five percent chance at that moment of being included in the block; it will likely be included in one of the next four blocks, getting more likely each chance. If it is still not included then the value of time waiting will be higher, making for more probability. A very low fee transaction would have a value for the fee of one. It would not be until near sixty-days that the particular low fee transaction has a high likelihood of being included in the block.
Quoted text (2 lines)
> > I am not concerned with low (or high) transaction fees, the primary
reason for addressing the issue is to ensure transactional reliability and scalability while having each transaction confirm in due time.
Quoted text (6 lines)
> > ## Pros: > * Maximizes transaction reliability. > * Fully scalable. > * Maximizes possibility for consumer and business uptake. > * Maximizes total fees paid per block without reducing reliability;
because of reliability, in time confidence and overall uptake are greater; therefore, more transactions.
Quoted text (3 lines)
> * Market determines fee paid for transaction priority. > * Fee recommendations work all the way out to 30 days or greater. > * Provides additional block entropy; greater security since there is less
probability of predicting the next block.
Quoted text (7 lines)
> > ## Cons: > * Could initially lower total transaction fees per block. > * Must be first be programmed. > > ## Solution operation: > This is a simplistic view of the operation. The actual operation will
need to be determined in a spec for the programmer.
Quoted text (4 lines)
> > 1. Determine the target block size for the current block. > 2. Assign a transaction priority to each transaction in the pool. > 3. Select transactions to include in the current block using probability
in transaction priority order until the target block size is met.
Quoted text (2 lines)
> 5. Solve block. > 6. Broadcast the next target block size with the current block when it is
solved.
Quoted text (7 lines)
> 7. Block is received. > 8. Block verification process. > 9. Accept/reject block based on verification result. > 10. Repeat. > > ## Closing comments: > It may be possible to verify blocks conform to the proposal by showing
that the probability for all transactions included in the block statistically conforms to a probability distribution curve, *if* the individual transaction priority can be recreated. I am not that deep into the mathematics; however, it may also be possible to use a similar method to do this just based on the fee, that statistically, the blocks conform to a fee distribution. Any zero fee transactions would have to be ignored. This solution needs a clever mathematician.
Quoted text (2 lines)
> > I implore, at the very least, that we use some method that validates full
transaction reliability and enables scalability of block sizes. If not this proposal, an alternative.
Quoted text (12 lines)
> > Regards, > Damian Williamson > > > > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >
Thank you for your constructive feedback. I now see that the proposal introduces a potential issue.
It is difficult to define then, what is a valid transaction? Clearly, my definition was insufficient.
Regards,
Damian Williamson
________________________________ From: Chris Riley <criley@gmail•com> Sent: Monday, 18 December 2017 11:09 PM To: Damian Williamson; Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Regarding "problem" #2 where you say "How do we ensure that all valid transactions are eventually included in the blockchain?": I do not believe that all people would (a) agree this is a problem or (b) that we do want to *ENSURE* that *ALL* valid transactions are eventually included in the blockchain. There are many *valid* transactions that oftentimes miners do not (and should not) wish to require be confirmed and included in the blockchain. Spam transactions for example can be valid, but used to attack bitcoin by using no or low fee. Any valid transaction MAY be included by a miner, but requiring it in some fashion at this point would open the network to other attack vectors. Perhaps you meant it a different way.
On Fri, Dec 15, 2017 at 3:59 PM, Damian Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
Quoted text (193 lines)
> > There are really two separate problems to solve. > > > How does Bitcoin scale with fixed block size? > How do we ensure that all valid transactions are eventually included in the blockchain? > > > Those are the two issues that the proposal attempts to address. It makes sense to resolve these two problems together. Using the proposed system for variable block sizes would solve the first problem but there would still be a whole bunch of never confirming transactions. I am not sure how to reliably solve the second problem at scale without first solving the first. > > > >* Every node has a (potentially) different mempool, you can't use it to decide consensus values like the max block size. > > > I do not suggest a consensus. Depending on which node solves a block the value for next block size will be different. The consensus would be that blocks will adhere to the next block size value transmitted with the current block. It is easy to verify that the consensus is being adhered to once in place. > > >* Increasing the entropy in a block to make it more unpredictable doesn't really make sense. > > Not a necessary function, just an effect of using a probability-based distribution. > > >* Bitcoin should be roughly incentive compatible. Your proposal explicits asks miners to ignore their best interests, and confirm transactions by "priority". What are you going to do if a "malicious" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a consensus requirement? And even if you miners can still sort their mempool by fee, and then order the top 1MB by priority. > > I entirely agree with your sentiment that Bitcoin must be incentive compatible. It is necessary. > > It is in only miners immediate interest to make the most profitable block from the available transaction pool. As with so many other things, it is necessary to partially ignore short-term gain for long-term benefit. It is in miners and everybody's long-term interest to have a reliable transaction service. A busy transaction service that confirms lots of transactions per hour will become more profitable as demand increases and more users are prepared to pay for priority. As it is there is currently no way to fully scale because of the transaction bandwidth limit and that is problematic. If all valid transactions must eventually confirm then there must be a way to resolve that problem. > > Bitcoin deliberately removes traditional scale by ensuring blocks take ten minutes on average to solve, an ingenious idea and, incentive compatible but, fixed block sizes leaves us with a problem to solve when we want to scale. > > >If you could find a good solution that would allow you to know if miners were following your rule or not (and thus ignore it if it doesn't) then you wouldn't even need bitcoin in the first place. > > I am confident that the math to verify blocks based on the proposal can be developed (and I think it will not be too complex for a mathematician with the relevant experience), however, I am nowhere near experienced enough with probability and statistical analysis to do it. Yes, if Bitcoin doesn't then it might make another great opportunity for an altcoin but I am not even nearly interested in promoting any altcoins. > > > If not the proposal that I have put forward, then, hopefully, someone can come up with a better solution. The important thing is that the issues are resolved. > > > Regards, > > Damian Williamson > > > > ________________________________ > From: Rhavar <rhavar@protonmail•com<mailto:rhavar@protonmail.com>> > Sent: Saturday, 16 December 2017 3:38 AM > To: Damian Williamson > Cc: Bitcoin Protocol Discussion > Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks > > > I understand that there would be technical issues to resolve in implementation, but, are there no fundamental errors? > > Unfortunately your proposal is really fundamentally broken, on a few levels. I think you might need to do a bit more research into how bitcoin works before coming up with such improvements =) > > But just some quick notes: > > * Every node has a (potentially) different mempool, you can't use it to decide consensus values like the max block size. > > * Increasing the entropy in a block to make it more unpredictable doesn't really make sense. > > * Bitcoin should be roughly incentive compatible. Your proposal explicits asks miners to ignore their best interests, and confirm transactions by "priority". What are you going to do if a "malicious" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a consensus requirement? And even if you miners can still sort their mempool by fee, and then order the top 1MB by priority. > > If you could find a good solution that would allow you to know if miners were following your rule or not (and thus ignore it if it doesn't) then you wouldn't even need bitcoin in the first place. > > > > > -Ryan > > > -------- Original Message -------- > Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks > Local Time: December 15, 2017 3:42 AM > UTC Time: December 15, 2017 9:42 AM > From: bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org> > To: Bitcoin Protocol Discussion <bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org>> > > > > I should not take it that the lack of critical feedback to this revised proposal is a glowing endorsement. I understand that there would be technical issues to resolve in implementation, but, are there no fundamental errors? > > I suppose that it if is difficult to determine how long a transaction has been waiting in the pool then, each node could simply keep track of when a transaction was first seen. This may have implications for a verify routine, however, for example, if a node was offline, how should it differentiate how long each transaction was waiting in that case? If a node was restarted daily would it always think that all transactions had been waiting in the pool less than one day If each node keeps the current transaction pool in a file and updates it, as transactions are included in blocks and, as new transactions appear in the pool, then that would go some way to alleviate the issue, apart from entirely new nodes. There should be no reason the contents of a transaction pool files cannot be shared without agreement as to the transaction pool between nodes, just as nodes transmit new transactions freely. > > It has been questioned why miners could not cheat. For the question of how many transactions to include in a block, I say it is a standoff and miners will conform to the proposal, not wanting to leave transactions with valid fees standing, and, not wanting to shrink the transaction pool. In any case, if miners shrink the transaction pool then I am not immediately concerned since it provides a more efficient service. For the question of including transactions according to the proposal, I say if it is possible to keep track of how long transactions are waiting in the pool so that they can be included on a probability curve then it is possible to verify that blocks conform to the proposal, since the input is a probability, the output should conform to a probability curve. > > > If someone has the necessary skill, would anyone be willing to develop the math necessary for the proposal? > > Regards, > Damian Williamson > > > ________________________________ > > From: bitcoin-dev-bounces@lists•linuxfoundation•org<mailto:bitcoin-dev-bounces@lists.linuxfoundation.org> <bitcoin-dev-bounces@lists•linuxfoundation•org<mailto:bitcoin-dev-bounces@lists.linuxfoundation.org>> on behalf of Damian Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org>> > Sent: Friday, 8 December 2017 8:01 AM > To: bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org> > Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks > > > > Good afternoon, > > The need for this proposal: > > We all must learn to admit that transaction bandwidth is still lurking as a serious issue for the operation, reliability, safety, consumer acceptance, uptake and, for the value of Bitcoin. > > I recently sent a payment which was not urgent so; I chose three-day target confirmation from the fee recommendation. That transaction has still not confirmed after now more than six days - even waiting twice as long seems quite reasonable to me. That transaction is a valid transaction; it is not rubbish, junk or, spam. Under the current model with transaction bandwidth limitation, the longer a transaction waits, the less likely it is ever to confirm due to rising transaction numbers and being pushed back by transactions with rising fees. > > I argue that no transactions are rubbish or junk, only some zero fee transactions might be spam. Having an ever-increasing number of valid transactions that do not confirm as more new transactions with higher fees are created is the opposite of operating a robust, reliable transaction system. > > Business cannot operate with a model where transactions may or may not confirm. Even a business choosing a modest fee has no guarantee that their valid transaction will not be shuffled down by new transactions to the realm of never confirming after it is created. Consumers also will not accept this model as Bitcoin expands. If Bitcoin cannot be a reliable payment system for confirmed transactions then consumers, by and large, will simply not accept the model once they understand. Bitcoin will be a dirty payment system, and this will kill the value of Bitcoin. > > Under the current system, a minority of transactions will eventually be the lucky few who have fees high enough to escape being pushed down the list. > > Once there are more than x transactions (transaction bandwidth limit) every ten minutes, only those choosing twenty-minute confirmation (2 blocks) will have initially at most a fifty percent chance of ever having their payment confirm. Presently, not even using fee recommendations can ensure a sufficiently high fee is paid to ensure transaction confirmation. > > I also argue that the current auction model for limited transaction bandwidth is wrong, is not suitable for a reliable transaction system and, is wrong for Bitcoin. All transactions must confirm in due time. Currently, Bitcoin is not a safe way to send payments. > > I do not believe that consumers and business are against paying fees, even high fees. What is required is operational reliability. > > This great issue needs to be resolved for the safety and reliability of Bitcoin. The time to resolve issues in commerce is before they become great big issues. The time to resolve this issue is now. We must have the foresight to identify and resolve problems before they trip us over. Simply doubling block sizes every so often is reactionary and is not a reliable permanent solution. I have written a BIP proposal for a technical solution but, need your help to write it up to an acceptable standard to be a full BIP. > > I have formatted the following with markdown which is human readable so, I hope nobody minds. I have done as much with this proposal as I feel that I am able so far but continue to take your feedback. > > # BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks > > ## The problem: > Everybody wants value. Miners want to maximize revenue from fees (and we presume, to minimize block size). Consumers need transaction reliability and, (we presume) want low fees. > > The current transaction bandwidth limit is a limiting factor for both. As the operational safety of transactions is limited, so is consumer confidence as they realize the issue and, accordingly, uptake is limited. Fees are artificially inflated due to bandwidth limitations while failing to provide a full confirmation service for all transactions. > > Current fee recommendations provide no satisfaction for transaction reliability and, as Bitcoin scales, this will worsen. > > Bitcoin must be a fully scalable and reliable service, providing full transaction confirmation for every valid transaction. > > The possibility to send a transaction with a fee lower than one that is acceptable to allow eventual transaction confirmation should be removed from the protocol and also from the user interface. > > ## Solution summary: > Provide each transaction with an individual transaction priority each time before choosing transactions to include in the current block, the priority being a function of the fee paid (on a curve), and the time waiting in the transaction pool (also on a curve) out to n days (n=60 ?). The transaction priority to serve as the likelihood of a transaction being included in the current block, and for determining the order in which transactions are tried to see if they will be included. > > Use a target block size. Determine the target block size using; current transaction pool size x ( 1 / (144 x n days ) ) = number of transactions to be included in the current block. Broadcast the next target block size with the current block when it is solved so that nodes know the next target block size for the block that they are building on. > > The curves used for the priority of transactions would have to be appropriate. Perhaps a mathematician with experience in probability can develop the right formulae. My thinking is a steep curve. I suppose that the probability of all transactions should probably account for a sufficient number of inclusions that the target block size is met although, it may not always be. As a suggestion, consider including some zero fee transactions to pad, highest BTC value first? > > **Explanation of the operation of priority:** > > If transaction priority is, for example, a number between one (low) and one-hundred (high) it can be directly understood as the percentage chance in one-hundred of a transaction being included in the block. Using probability or likelihood infers that there is some function of random. If random (100) < transaction priority then the transaction is included. > > >To break it down further, if both the fee on a curve value and the time waiting on a curve value are each a number between one and one-hundred, a rudimentary method may be to simply multiply those two numbers, to find the priority number. For example, a middle fee transaction waiting thirty days (if n = 60 days) may have a value of five for each part (yes, just five, the values are on a curve). When multiplied that will give a priority value of twenty-five, or, a twenty-five percent chance at that moment of being included in the block; it will likely be included in one of the next four blocks, getting more likely each chance. If it is still not included then the value of time waiting will be higher, making for more probability. A very low fee transaction would have a value for the fee of one. It would not be until near sixty-days that the particular low fee transaction has a high likelihood of being included in the block. > > I am not concerned with low (or high) transaction fees, the primary reason for addressing the issue is to ensure transactional reliability and scalability while having each transaction confirm in due time. > > ## Pros: > * Maximizes transaction reliability. > * Fully scalable. > * Maximizes possibility for consumer and business uptake. > * Maximizes total fees paid per block without reducing reliability; because of reliability, in time confidence and overall uptake are greater; therefore, more transactions. > * Market determines fee paid for transaction priority. > * Fee recommendations work all the way out to 30 days or greater. > * Provides additional block entropy; greater security since there is less probability of predicting the next block. > > ## Cons: > * Could initially lower total transaction fees per block. > * Must be first be programmed. > > ## Solution operation: > This is a simplistic view of the operation. The actual operation will need to be determined in a spec for the programmer. > > 1. Determine the target block size for the current block. > 2. Assign a transaction priority to each transaction in the pool. > 3. Select transactions to include in the current block using probability in transaction priority order until the target block size is met. > 5. Solve block. > 6. Broadcast the next target block size with the current block when it is solved. > 7. Block is received. > 8. Block verification process. > 9. Accept/reject block based on verification result. > 10. Repeat. > > ## Closing comments: > It may be possible to verify blocks conform to the proposal by showing that the probability for all transactions included in the block statistically conforms to a probability distribution curve, *if* the individual transaction priority can be recreated. I am not that deep into the mathematics; however, it may also be possible to use a similar method to do this just based on the fee, that statistically, the blocks conform to a fee distribution. Any zero fee transactions would have to be ignored. This solution needs a clever mathematician. > > I implore, at the very least, that we use some method that validates full transaction reliability and enables scalability of block sizes. If not this proposal, an alternative. > > Regards, > Damian Williamson > > > > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >
Thank you for your constructive feedback. I now see that the proposal introduces a potential issue.
Quoted text (1 line)
>Finally in terms of the broad goal, having block size based on the number of transactions is NOT something desirable in the first place, even if it did work. That’s effectively the same as an infinite block size since anyone anywhere can create transactions in the mempool at no cost.
Do you have any critical suggestion as to how transaction bandwidth limit could be addressed, it will eventually become an issue if nothing is changed regardless of how high fees go?
Regards,
Damian Williamson
________________________________ From: Mark Friedenbach <mark@friedenbach•org> Sent: Tuesday, 19 December 2017 3:08 AM To: Damian Williamson Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Damian, you seem to be misunderstanding that either
(1) the strong form of your proposal requires validating the commitment to the mempool properties, in which case the mempool becomes consensus critical (an impossible requirement); or
(2) in the weak form where the current block is dependent on the commitment in the last block only it is becomes a miner-selected field they can freely parameterize with no repercussions for setting values totally independent of the actual mempool.
If you want to make the block size dependent on the properties of the mempool in a consensus critical way, flex cap achieves this. If you want to make the contents or properties of the mempool known to well-connected nodes, weak blocks achieves that. But you can’t stick the mempool in consensus because it fundamentally is not something the nodes have consensus over. That’s a chicken-and-the-egg assumption.
Finally in terms of the broad goal, having block size based on the number of transactions is NOT something desirable in the first place, even if it did work. That’s effectively the same as an infinite block size since anyone anywhere can create transactions in the mempool at no cost.
On Dec 16, 2017, at 8:14 PM, Damian Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
I do not know why people make the leap that the proposal requires a consensus on the transaction pool. It does not.
It may be helpful to have the discussion from the previous thread linked here. https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-December/015370.html
Where I speak of validating that a block conforms to the broadcast next block size, I do not propose validating the number broadcast for the next block size itself, only that the next generated block is that size.
Regards, Damian Williamson
________________________________ From: Damian Williamson <willtech@live•com•au<mailto:willtech@live.com.au>> Sent: Saturday, 16 December 2017 7:59 AM To: Rhavar Cc: Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
There are really two separate problems to solve.
1. How does Bitcoin scale with fixed block size? 2. How do we ensure that all valid transactions are eventually included in the blockchain?
Those are the two issues that the proposal attempts to address. It makes sense to resolve these two problems together. Using the proposed system for variable block sizes would solve the first problem but there would still be a whole bunch of never confirming transactions. I am not sure how to reliably solve the second problem at scale without first solving the first.
Quoted text (1 line)
>* Every node has a (potentially) different mempool, you can't use it to decide consensus values like the max block size.
I do not suggest a consensus. Depending on which node solves a block the value for next block size will be different. The consensus would be that blocks will adhere to the next block size value transmitted with the current block. It is easy to verify that the consensus is being adhered to once in place.
Quoted text (1 line)
>* Increasing the entropy in a block to make it more unpredictable doesn't really make sense.
Not a necessary function, just an effect of using a probability-based distribution.
Quoted text (1 line)
>* Bitcoin should be roughly incentive compatible. Your proposal explicits asks miners to ignore their best interests, and confirm transactions by "priority". What are you going to do if a "malicious" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a consensus requirement? And even if you miners can still sort their mempool by fee, and then order the top 1MB by priority.
I entirely agree with your sentiment that Bitcoin must be incentive compatible. It is necessary.
It is in only miners immediate interest to make the most profitable block from the available transaction pool. As with so many other things, it is necessary to partially ignore short-term gain for long-term benefit. It is in miners and everybody's long-term interest to have a reliable transaction service. A busy transaction service that confirms lots of transactions per hour will become more profitable as demand increases and more users are prepared to pay for priority. As it is there is currently no way to fully scale because of the transaction bandwidth limit and that is problematic. If all valid transactions must eventually confirm then there must be a way to resolve that problem.
Bitcoin deliberately removes traditional scale by ensuring blocks take ten minutes on average to solve, an ingenious idea and, incentive compatible but, fixed block sizes leaves us with a problem to solve when we want to scale.
Quoted text (1 line)
>If you could find a good solution that would allow you to know if miners were following your rule or not (and thus ignore it if it doesn't) then you wouldn't even need bitcoin in the first place.
I am confident that the math to verify blocks based on the proposal can be developed (and I think it will not be too complex for a mathematician with the relevant experience), however, I am nowhere near experienced enough with probability and statistical analysis to do it. Yes, if Bitcoin doesn't then it might make another great opportunity for an altcoin but I am not even nearly interested in promoting any altcoins.
If not the proposal that I have put forward, then, hopefully, someone can come up with a better solution. The important thing is that the issues are resolved.
Regards, Damian Williamson
________________________________ From: Rhavar <rhavar@protonmail•com<mailto:rhavar@protonmail.com>> Sent: Saturday, 16 December 2017 3:38 AM To: Damian Williamson Cc: Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Quoted text (1 line)
> I understand that there would be technical issues to resolve in implementation, but, are there no fundamental errors?
Unfortunately your proposal is really fundamentally broken, on a few levels. I think you might need to do a bit more research into how bitcoin works before coming up with such improvements =)
But just some quick notes:
* Every node has a (potentially) different mempool, you can't use it to decide consensus values like the max block size.
* Increasing the entropy in a block to make it more unpredictable doesn't really make sense.
* Bitcoin should be roughly incentive compatible. Your proposal explicits asks miners to ignore their best interests, and confirm transactions by "priority". What are you going to do if a "malicious" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a consensus requirement? And even if you miners can still sort their mempool by fee, and then order the top 1MB by priority.
If you could find a good solution that would allow you to know if miners were following your rule or not (and thus ignore it if it doesn't) then you wouldn't even need bitcoin in the first place.
-Ryan
-------- Original Message -------- Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks Local Time: December 15, 2017 3:42 AM UTC Time: December 15, 2017 9:42 AM From: bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org> To: Bitcoin Protocol Discussion <bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org>>
I should not take it that the lack of critical feedback to this revised proposal is a glowing endorsement. I understand that there would be technical issues to resolve in implementation, but, are there no fundamental errors?
I suppose that it if is difficult to determine how long a transaction has been waiting in the pool then, each node could simply keep track of when a transaction was first seen. This may have implications for a verify routine, however, for example, if a node was offline, how should it differentiate how long each transaction was waiting in that case? If a node was restarted daily would it always think that all transactions had been waiting in the pool less than one day If each node keeps the current transaction pool in a file and updates it, as transactions are included in blocks and, as new transactions appear in the pool, then that would go some way to alleviate the issue, apart from entirely new nodes. There should be no reason the contents of a transaction pool files cannot be shared without agreement as to the transaction pool between nodes, just as nodes transmit new transactions freely.
It has been questioned why miners could not cheat. For the question of how many transactions to include in a block, I say it is a standoff and miners will conform to the proposal, not wanting to leave transactions with valid fees standing, and, not wanting to shrink the transaction pool. In any case, if miners shrink the transaction pool then I am not immediately concerned since it provides a more efficient service. For the question of including transactions according to the proposal, I say if it is possible to keep track of how long transactions are waiting in the pool so that they can be included on a probability curve then it is possible to verify that blocks conform to the proposal, since the input is a probability, the output should conform to a probability curve.
If someone has the necessary skill, would anyone be willing to develop the math necessary for the proposal?
Regards, Damian Williamson
________________________________
From: bitcoin-dev-bounces@lists•linuxfoundation•org<mailto:bitcoin-dev-bounces@lists.linuxfoundation.org> <bitcoin-dev-bounces@lists•linuxfoundation•org<mailto:bitcoin-dev-bounces@lists.linuxfoundation.org>> on behalf of Damian Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org>> Sent: Friday, 8 December 2017 8:01 AM To: bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org> Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Good afternoon,
The need for this proposal:
We all must learn to admit that transaction bandwidth is still lurking as a serious issue for the operation, reliability, safety, consumer acceptance, uptake and, for the value of Bitcoin.
I recently sent a payment which was not urgent so; I chose three-day target confirmation from the fee recommendation. That transaction has still not confirmed after now more than six days - even waiting twice as long seems quite reasonable to me. That transaction is a valid transaction; it is not rubbish, junk or, spam. Under the current model with transaction bandwidth limitation, the longer a transaction waits, the less likely it is ever to confirm due to rising transaction numbers and being pushed back by transactions with rising fees.
I argue that no transactions are rubbish or junk, only some zero fee transactions might be spam. Having an ever-increasing number of valid transactions that do not confirm as more new transactions with higher fees are created is the opposite of operating a robust, reliable transaction system.
Business cannot operate with a model where transactions may or may not confirm. Even a business choosing a modest fee has no guarantee that their valid transaction will not be shuffled down by new transactions to the realm of never confirming after it is created. Consumers also will not accept this model as Bitcoin expands. If Bitcoin cannot be a reliable payment system for confirmed transactions then consumers, by and large, will simply not accept the model once they understand. Bitcoin will be a dirty payment system, and this will kill the value of Bitcoin.
Under the current system, a minority of transactions will eventually be the lucky few who have fees high enough to escape being pushed down the list.
Once there are more than x transactions (transaction bandwidth limit) every ten minutes, only those choosing twenty-minute confirmation (2 blocks) will have initially at most a fifty percent chance of ever having their payment confirm. Presently, not even using fee recommendations can ensure a sufficiently high fee is paid to ensure transaction confirmation.
I also argue that the current auction model for limited transaction bandwidth is wrong, is not suitable for a reliable transaction system and, is wrong for Bitcoin. All transactions must confirm in due time. Currently, Bitcoin is not a safe way to send payments.
I do not believe that consumers and business are against paying fees, even high fees. What is required is operational reliability.
This great issue needs to be resolved for the safety and reliability of Bitcoin. The time to resolve issues in commerce is before they become great big issues. The time to resolve this issue is now. We must have the foresight to identify and resolve problems before they trip us over. Simply doubling block sizes every so often is reactionary and is not a reliable permanent solution. I have written a BIP proposal for a technical solution but, need your help to write it up to an acceptable standard to be a full BIP.
I have formatted the following with markdown which is human readable so, I hope nobody minds. I have done as much with this proposal as I feel that I am able so far but continue to take your feedback.
# BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
## The problem: Everybody wants value. Miners want to maximize revenue from fees (and we presume, to minimize block size). Consumers need transaction reliability and, (we presume) want low fees.
The current transaction bandwidth limit is a limiting factor for both. As the operational safety of transactions is limited, so is consumer confidence as they realize the issue and, accordingly, uptake is limited. Fees are artificially inflated due to bandwidth limitations while failing to provide a full confirmation service for all transactions.
Current fee recommendations provide no satisfaction for transaction reliability and, as Bitcoin scales, this will worsen.
Bitcoin must be a fully scalable and reliable service, providing full transaction confirmation for every valid transaction.
The possibility to send a transaction with a fee lower than one that is acceptable to allow eventual transaction confirmation should be removed from the protocol and also from the user interface.
## Solution summary: Provide each transaction with an individual transaction priority each time before choosing transactions to include in the current block, the priority being a function of the fee paid (on a curve), and the time waiting in the transaction pool (also on a curve) out to n days (n=60 ?). The transaction priority to serve as the likelihood of a transaction being included in the current block, and for determining the order in which transactions are tried to see if they will be included.
Use a target block size. Determine the target block size using; current transaction pool size x ( 1 / (144 x n days ) ) = number of transactions to be included in the current block. Broadcast the next target block size with the current block when it is solved so that nodes know the next target block size for the block that they are building on.
The curves used for the priority of transactions would have to be appropriate. Perhaps a mathematician with experience in probability can develop the right formulae. My thinking is a steep curve. I suppose that the probability of all transactions should probably account for a sufficient number of inclusions that the target block size is met although, it may not always be. As a suggestion, consider including some zero fee transactions to pad, highest BTC value first?
**Explanation of the operation of priority:**
Quoted text (2 lines)
> If transaction priority is, for example, a number between one (low) and one-hundred (high) it can be directly understood as the percentage chance in one-hundred of a transaction being included in the block. Using probability or likelihood infers that there is some function of random. If random (100) < transaction priority then the transaction is included. >To break it down further, if both the fee on a curve value and the time waiting on a curve value are each a number between one and one-hundred, a rudimentary method may be to simply multiply those two numbers, to find the priority number. For example, a middle fee transaction waiting thirty days (if n = 60 days) may have a value of five for each part (yes, just five, the values are on a curve). When multiplied that will give a priority value of twenty-five, or, a twenty-five percent chance at that moment of being included in the block; it will likely be included in one of the next four blocks, getting more likely each chance. If it is still not included then the value of time waiting will be higher, making for more probability. A very low fee transaction would have a value for the fee of one. It would not be until near sixty-days that the particular low fee transaction has a high likelihood of being included in the block.
I am not concerned with low (or high) transaction fees, the primary reason for addressing the issue is to ensure transactional reliability and scalability while having each transaction confirm in due time.
## Pros: * Maximizes transaction reliability. * Fully scalable. * Maximizes possibility for consumer and business uptake. * Maximizes total fees paid per block without reducing reliability; because of reliability, in time confidence and overall uptake are greater; therefore, more transactions. * Market determines fee paid for transaction priority. * Fee recommendations work all the way out to 30 days or greater. * Provides additional block entropy; greater security since there is less probability of predicting the next block.
## Cons: * Could initially lower total transaction fees per block. * Must be first be programmed.
## Solution operation: This is a simplistic view of the operation. The actual operation will need to be determined in a spec for the programmer.
1. Determine the target block size for the current block. 2. Assign a transaction priority to each transaction in the pool. 3. Select transactions to include in the current block using probability in transaction priority order until the target block size is met. 5. Solve block. 6. Broadcast the next target block size with the current block when it is solved. 7. Block is received. 8. Block verification process. 9. Accept/reject block based on verification result. 10. Repeat.
## Closing comments: It may be possible to verify blocks conform to the proposal by showing that the probability for all transactions included in the block statistically conforms to a probability distribution curve, *if* the individual transaction priority can be recreated. I am not that deep into the mathematics; however, it may also be possible to use a similar method to do this just based on the fee, that statistically, the blocks conform to a fee distribution. Any zero fee transactions would have to be ignored. This solution needs a clever mathematician.
I implore, at the very least, that we use some method that validates full transaction reliability and enables scalability of block sizes. If not this proposal, an alternative.
Regards, Damian Williamson
If the cash value of Bitcoin was high enough and zero fee transactions were never accepted and not counted when calculating the transaction pool size then I do not think it would be such an issue. Why is it even possible to create zero fee transactions?
Regards,
Damian Williamson
________________________________ From: bitcoin-dev-bounces@lists•linuxfoundation•org <bitcoin-dev-bounces@lists•linuxfoundation•org> on behalf of Damian Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> Sent: Tuesday, 19 December 2017 6:51 PM To: Mark Friedenbach Cc: bitcoin-dev@lists•linuxfoundation•org Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Thank you for your constructive feedback. I now see that the proposal introduces a potential issue.
Quoted text (1 line)
>Finally in terms of the broad goal, having block size based on the number of transactions is NOT something desirable in the first place, even if it did work. That’s effectively the same as an infinite block size since anyone anywhere can create transactions in the mempool at no cost.
Do you have any critical suggestion as to how transaction bandwidth limit could be addressed, it will eventually become an issue if nothing is changed regardless of how high fees go?
Regards,
Damian Williamson
________________________________ From: Mark Friedenbach <mark@friedenbach•org> Sent: Tuesday, 19 December 2017 3:08 AM To: Damian Williamson Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Damian, you seem to be misunderstanding that either
(1) the strong form of your proposal requires validating the commitment to the mempool properties, in which case the mempool becomes consensus critical (an impossible requirement); or
(2) in the weak form where the current block is dependent on the commitment in the last block only it is becomes a miner-selected field they can freely parameterize with no repercussions for setting values totally independent of the actual mempool.
If you want to make the block size dependent on the properties of the mempool in a consensus critical way, flex cap achieves this. If you want to make the contents or properties of the mempool known to well-connected nodes, weak blocks achieves that. But you can’t stick the mempool in consensus because it fundamentally is not something the nodes have consensus over. That’s a chicken-and-the-egg assumption.
Finally in terms of the broad goal, having block size based on the number of transactions is NOT something desirable in the first place, even if it did work. That’s effectively the same as an infinite block size since anyone anywhere can create transactions in the mempool at no cost.
On Dec 16, 2017, at 8:14 PM, Damian Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
I do not know why people make the leap that the proposal requires a consensus on the transaction pool. It does not.
It may be helpful to have the discussion from the previous thread linked here. https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-December/015370.html
Where I speak of validating that a block conforms to the broadcast next block size, I do not propose validating the number broadcast for the next block size itself, only that the next generated block is that size.
Regards, Damian Williamson
________________________________ From: Damian Williamson <willtech@live•com•au<mailto:willtech@live.com.au>> Sent: Saturday, 16 December 2017 7:59 AM To: Rhavar Cc: Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
There are really two separate problems to solve.
1. How does Bitcoin scale with fixed block size? 2. How do we ensure that all valid transactions are eventually included in the blockchain?
Those are the two issues that the proposal attempts to address. It makes sense to resolve these two problems together. Using the proposed system for variable block sizes would solve the first problem but there would still be a whole bunch of never confirming transactions. I am not sure how to reliably solve the second problem at scale without first solving the first.
Quoted text (1 line)
>* Every node has a (potentially) different mempool, you can't use it to decide consensus values like the max block size.
I do not suggest a consensus. Depending on which node solves a block the value for next block size will be different. The consensus would be that blocks will adhere to the next block size value transmitted with the current block. It is easy to verify that the consensus is being adhered to once in place.
Quoted text (1 line)
>* Increasing the entropy in a block to make it more unpredictable doesn't really make sense.
Not a necessary function, just an effect of using a probability-based distribution.
Quoted text (1 line)
>* Bitcoin should be roughly incentive compatible. Your proposal explicits asks miners to ignore their best interests, and confirm transactions by "priority". What are you going to do if a "malicious" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a consensus requirement? And even if you miners can still sort their mempool by fee, and then order the top 1MB by priority.
I entirely agree with your sentiment that Bitcoin must be incentive compatible. It is necessary.
It is in only miners immediate interest to make the most profitable block from the available transaction pool. As with so many other things, it is necessary to partially ignore short-term gain for long-term benefit. It is in miners and everybody's long-term interest to have a reliable transaction service. A busy transaction service that confirms lots of transactions per hour will become more profitable as demand increases and more users are prepared to pay for priority. As it is there is currently no way to fully scale because of the transaction bandwidth limit and that is problematic. If all valid transactions must eventually confirm then there must be a way to resolve that problem.
Bitcoin deliberately removes traditional scale by ensuring blocks take ten minutes on average to solve, an ingenious idea and, incentive compatible but, fixed block sizes leaves us with a problem to solve when we want to scale.
Quoted text (1 line)
>If you could find a good solution that would allow you to know if miners were following your rule or not (and thus ignore it if it doesn't) then you wouldn't even need bitcoin in the first place.
I am confident that the math to verify blocks based on the proposal can be developed (and I think it will not be too complex for a mathematician with the relevant experience), however, I am nowhere near experienced enough with probability and statistical analysis to do it. Yes, if Bitcoin doesn't then it might make another great opportunity for an altcoin but I am not even nearly interested in promoting any altcoins.
If not the proposal that I have put forward, then, hopefully, someone can come up with a better solution. The important thing is that the issues are resolved.
Regards, Damian Williamson
________________________________ From: Rhavar <rhavar@protonmail•com<mailto:rhavar@protonmail.com>> Sent: Saturday, 16 December 2017 3:38 AM To: Damian Williamson Cc: Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Quoted text (1 line)
> I understand that there would be technical issues to resolve in implementation, but, are there no fundamental errors?
Unfortunately your proposal is really fundamentally broken, on a few levels. I think you might need to do a bit more research into how bitcoin works before coming up with such improvements =)
But just some quick notes:
* Every node has a (potentially) different mempool, you can't use it to decide consensus values like the max block size.
* Increasing the entropy in a block to make it more unpredictable doesn't really make sense.
* Bitcoin should be roughly incentive compatible. Your proposal explicits asks miners to ignore their best interests, and confirm transactions by "priority". What are you going to do if a "malicious" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a consensus requirement? And even if you miners can still sort their mempool by fee, and then order the top 1MB by priority.
If you could find a good solution that would allow you to know if miners were following your rule or not (and thus ignore it if it doesn't) then you wouldn't even need bitcoin in the first place.
-Ryan
-------- Original Message -------- Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks Local Time: December 15, 2017 3:42 AM UTC Time: December 15, 2017 9:42 AM From: bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org> To: Bitcoin Protocol Discussion <bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org>>
I should not take it that the lack of critical feedback to this revised proposal is a glowing endorsement. I understand that there would be technical issues to resolve in implementation, but, are there no fundamental errors?
I suppose that it if is difficult to determine how long a transaction has been waiting in the pool then, each node could simply keep track of when a transaction was first seen. This may have implications for a verify routine, however, for example, if a node was offline, how should it differentiate how long each transaction was waiting in that case? If a node was restarted daily would it always think that all transactions had been waiting in the pool less than one day If each node keeps the current transaction pool in a file and updates it, as transactions are included in blocks and, as new transactions appear in the pool, then that would go some way to alleviate the issue, apart from entirely new nodes. There should be no reason the contents of a transaction pool files cannot be shared without agreement as to the transaction pool between nodes, just as nodes transmit new transactions freely.
It has been questioned why miners could not cheat. For the question of how many transactions to include in a block, I say it is a standoff and miners will conform to the proposal, not wanting to leave transactions with valid fees standing, and, not wanting to shrink the transaction pool. In any case, if miners shrink the transaction pool then I am not immediately concerned since it provides a more efficient service. For the question of including transactions according to the proposal, I say if it is possible to keep track of how long transactions are waiting in the pool so that they can be included on a probability curve then it is possible to verify that blocks conform to the proposal, since the input is a probability, the output should conform to a probability curve.
If someone has the necessary skill, would anyone be willing to develop the math necessary for the proposal?
Regards, Damian Williamson
________________________________
From: bitcoin-dev-bounces@lists•linuxfoundation•org<mailto:bitcoin-dev-bounces@lists.linuxfoundation.org> <bitcoin-dev-bounces@lists•linuxfoundation•org<mailto:bitcoin-dev-bounces@lists.linuxfoundation.org>> on behalf of Damian Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org>> Sent: Friday, 8 December 2017 8:01 AM To: bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org> Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Good afternoon,
The need for this proposal:
We all must learn to admit that transaction bandwidth is still lurking as a serious issue for the operation, reliability, safety, consumer acceptance, uptake and, for the value of Bitcoin.
I recently sent a payment which was not urgent so; I chose three-day target confirmation from the fee recommendation. That transaction has still not confirmed after now more than six days - even waiting twice as long seems quite reasonable to me. That transaction is a valid transaction; it is not rubbish, junk or, spam. Under the current model with transaction bandwidth limitation, the longer a transaction waits, the less likely it is ever to confirm due to rising transaction numbers and being pushed back by transactions with rising fees.
I argue that no transactions are rubbish or junk, only some zero fee transactions might be spam. Having an ever-increasing number of valid transactions that do not confirm as more new transactions with higher fees are created is the opposite of operating a robust, reliable transaction system.
Business cannot operate with a model where transactions may or may not confirm. Even a business choosing a modest fee has no guarantee that their valid transaction will not be shuffled down by new transactions to the realm of never confirming after it is created. Consumers also will not accept this model as Bitcoin expands. If Bitcoin cannot be a reliable payment system for confirmed transactions then consumers, by and large, will simply not accept the model once they understand. Bitcoin will be a dirty payment system, and this will kill the value of Bitcoin.
Under the current system, a minority of transactions will eventually be the lucky few who have fees high enough to escape being pushed down the list.
Once there are more than x transactions (transaction bandwidth limit) every ten minutes, only those choosing twenty-minute confirmation (2 blocks) will have initially at most a fifty percent chance of ever having their payment confirm. Presently, not even using fee recommendations can ensure a sufficiently high fee is paid to ensure transaction confirmation.
I also argue that the current auction model for limited transaction bandwidth is wrong, is not suitable for a reliable transaction system and, is wrong for Bitcoin. All transactions must confirm in due time. Currently, Bitcoin is not a safe way to send payments.
I do not believe that consumers and business are against paying fees, even high fees. What is required is operational reliability.
This great issue needs to be resolved for the safety and reliability of Bitcoin. The time to resolve issues in commerce is before they become great big issues. The time to resolve this issue is now. We must have the foresight to identify and resolve problems before they trip us over. Simply doubling block sizes every so often is reactionary and is not a reliable permanent solution. I have written a BIP proposal for a technical solution but, need your help to write it up to an acceptable standard to be a full BIP.
I have formatted the following with markdown which is human readable so, I hope nobody minds. I have done as much with this proposal as I feel that I am able so far but continue to take your feedback.
# BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
## The problem: Everybody wants value. Miners want to maximize revenue from fees (and we presume, to minimize block size). Consumers need transaction reliability and, (we presume) want low fees.
The current transaction bandwidth limit is a limiting factor for both. As the operational safety of transactions is limited, so is consumer confidence as they realize the issue and, accordingly, uptake is limited. Fees are artificially inflated due to bandwidth limitations while failing to provide a full confirmation service for all transactions.
Current fee recommendations provide no satisfaction for transaction reliability and, as Bitcoin scales, this will worsen.
Bitcoin must be a fully scalable and reliable service, providing full transaction confirmation for every valid transaction.
The possibility to send a transaction with a fee lower than one that is acceptable to allow eventual transaction confirmation should be removed from the protocol and also from the user interface.
## Solution summary: Provide each transaction with an individual transaction priority each time before choosing transactions to include in the current block, the priority being a function of the fee paid (on a curve), and the time waiting in the transaction pool (also on a curve) out to n days (n=60 ?). The transaction priority to serve as the likelihood of a transaction being included in the current block, and for determining the order in which transactions are tried to see if they will be included.
Use a target block size. Determine the target block size using; current transaction pool size x ( 1 / (144 x n days ) ) = number of transactions to be included in the current block. Broadcast the next target block size with the current block when it is solved so that nodes know the next target block size for the block that they are building on.
The curves used for the priority of transactions would have to be appropriate. Perhaps a mathematician with experience in probability can develop the right formulae. My thinking is a steep curve. I suppose that the probability of all transactions should probably account for a sufficient number of inclusions that the target block size is met although, it may not always be. As a suggestion, consider including some zero fee transactions to pad, highest BTC value first?
**Explanation of the operation of priority:**
Quoted text (2 lines)
> If transaction priority is, for example, a number between one (low) and one-hundred (high) it can be directly understood as the percentage chance in one-hundred of a transaction being included in the block. Using probability or likelihood infers that there is some function of random. If random (100) < transaction priority then the transaction is included. >To break it down further, if both the fee on a curve value and the time waiting on a curve value are each a number between one and one-hundred, a rudimentary method may be to simply multiply those two numbers, to find the priority number. For example, a middle fee transaction waiting thirty days (if n = 60 days) may have a value of five for each part (yes, just five, the values are on a curve). When multiplied that will give a priority value of twenty-five, or, a twenty-five percent chance at that moment of being included in the block; it will likely be included in one of the next four blocks, getting more likely each chance. If it is still not included then the value of time waiting will be higher, making for more probability. A very low fee transaction would have a value for the fee of one. It would not be until near sixty-days that the particular low fee transaction has a high likelihood of being included in the block.
I am not concerned with low (or high) transaction fees, the primary reason for addressing the issue is to ensure transactional reliability and scalability while having each transaction confirm in due time.
## Pros: * Maximizes transaction reliability. * Fully scalable. * Maximizes possibility for consumer and business uptake. * Maximizes total fees paid per block without reducing reliability; because of reliability, in time confidence and overall uptake are greater; therefore, more transactions. * Market determines fee paid for transaction priority. * Fee recommendations work all the way out to 30 days or greater. * Provides additional block entropy; greater security since there is less probability of predicting the next block.
## Cons: * Could initially lower total transaction fees per block. * Must be first be programmed.
## Solution operation: This is a simplistic view of the operation. The actual operation will need to be determined in a spec for the programmer.
1. Determine the target block size for the current block. 2. Assign a transaction priority to each transaction in the pool. 3. Select transactions to include in the current block using probability in transaction priority order until the target block size is met. 5. Solve block. 6. Broadcast the next target block size with the current block when it is solved. 7. Block is received. 8. Block verification process. 9. Accept/reject block based on verification result. 10. Repeat.
## Closing comments: It may be possible to verify blocks conform to the proposal by showing that the probability for all transactions included in the block statistically conforms to a probability distribution curve, *if* the individual transaction priority can be recreated. I am not that deep into the mathematics; however, it may also be possible to use a similar method to do this just based on the fee, that statistically, the blocks conform to a fee distribution. Any zero fee transactions would have to be ignored. This solution needs a clever mathematician.
I implore, at the very least, that we use some method that validates full transaction reliability and enables scalability of block sizes. If not this proposal, an alternative.
Regards, Damian Williamson
Hi Damian,
Thought I'd chip in. This is a hard fork scenario. This system has flaws, they all do.
If you had a fixed fee per block, so that every txn in that block paid the same fee, that might make it easier to include all txns eventually, as you envisage.
The fee could be calculated as the average of the amount txns are prepared to pay in the last 1000 blocks.
A txn would say ' I'll pay up to X bitcoins ' and as long as that is more than the value required for the block your txn can be added. This is to ensure you don't pay more than you are willing. It also ensures that putting an enormous fee will not ensure your txn is processed quickly..
Calculating what the outputs are given a variable fee needs a new mechanism all of it's own, but I'm sure it's possible.
The simple fact is that there is currently no known system that works as well as the current system..
But there are other systems.
On Dec 22, 2017 15:09, "Damian Williamson via bitcoin-dev" < bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (471 lines)
> If the cash value of Bitcoin was high enough and zero fee transactions > were never accepted and not counted when calculating the transaction pool > size then I do not think it would be such an issue. Why is it even possible > to create zero fee transactions? > > > Regards, > > Damian Williamson > > ------------------------------ > *From:* bitcoin-dev-bounces@lists•linuxfoundation•org < > bitcoin-dev-bounces@lists•linuxfoundation•org> on behalf of Damian > Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> > *Sent:* Tuesday, 19 December 2017 6:51 PM > *To:* Mark Friedenbach > *Cc:* bitcoin-dev@lists•linuxfoundation•org > *Subject:* [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use > Transaction Priority For Ordering Transactions In Blocks > > > Thank you for your constructive feedback. I now see that the proposal > introduces a potential issue. > > > >Finally in terms of the broad goal, having block size based on the number > of transactions is NOT something desirable in the first place, even if it > did work. That’s effectively the same as an infinite block size since > anyone anywhere can create transactions in the mempool at no cost. > > > Do you have any critical suggestion as to how transaction bandwidth limit > could be addressed, it will eventually become an issue if nothing is > changed regardless of how high fees go? > > > Regards, > Damian Williamson > > > > ------------------------------ > *From:* Mark Friedenbach <mark@friedenbach•org> > *Sent:* Tuesday, 19 December 2017 3:08 AM > *To:* Damian Williamson > *Subject:* Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use > Transaction Priority For Ordering Transactions In Blocks > > Damian, you seem to be misunderstanding that either > > (1) the strong form of your proposal requires validating the commitment to > the mempool properties, in which case the mempool becomes consensus > critical (an impossible requirement); or > > (2) in the weak form where the current block is dependent on the > commitment in the last block only it is becomes a miner-selected field they > can freely parameterize with no repercussions for setting values totally > independent of the actual mempool. > > If you want to make the block size dependent on the properties of the > mempool in a consensus critical way, flex cap achieves this. If you want to > make the contents or properties of the mempool known to well-connected > nodes, weak blocks achieves that. But you can’t stick the mempool in > consensus because it fundamentally is not something the nodes have > consensus over. That’s a chicken-and-the-egg assumption. > > Finally in terms of the broad goal, having block size based on the number > of transactions is NOT something desirable in the first place, even if it > did work. That’s effectively the same as an infinite block size since > anyone anywhere can create transactions in the mempool at no cost. > > On Dec 16, 2017, at 8:14 PM, Damian Williamson via bitcoin-dev < > bitcoin-dev@lists•linuxfoundation•org> wrote: > > I do not know why people make the leap that the proposal requires a > consensus on the transaction pool. It does not. > > It may be helpful to have the discussion from the previous thread linked > here. > https://lists.linuxfoundation.org/pipermail/bitcoin-dev/ > 2017-December/015370.html > > Where I speak of validating that a block conforms to the broadcast next > block size, I do not propose validating the number broadcast for the next > block size itself, only that the next generated block is that size. > > Regards, > Damian Williamson > > > ------------------------------ > *From:* Damian Williamson <willtech@live•com•au> > *Sent:* Saturday, 16 December 2017 7:59 AM > *To:* Rhavar > *Cc:* Bitcoin Protocol Discussion > *Subject:* Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use > Transaction Priority For Ordering Transactions In Blocks > > There are really two separate problems to solve. > > > 1. How does Bitcoin scale with fixed block size? > 2. How do we ensure that all valid transactions are eventually > included in the blockchain? > > > Those are the two issues that the proposal attempts to address. It makes > sense to resolve these two problems together. Using the proposed system for > variable block sizes would solve the first problem but there would still be > a whole bunch of never confirming transactions. I am not sure how to > reliably solve the second problem at scale without first solving the first. > > >* Every node has a (potentially) different mempool, you can't use it to > decide consensus values like the max block size. > > I do not suggest a consensus. Depending on which node solves a block the > value for next block size will be different. The consensus would be that > blocks will adhere to the next block size value transmitted with the > current block. It is easy to verify that the consensus is being adhered to > once in place. > > >* Increasing the entropy in a block to make it more unpredictable doesn't > really make sense. > > Not a necessary function, just an effect of using a probability-based > distribution. > > >* Bitcoin should be roughly incentive compatible. Your proposal explicits > asks miners to ignore their best interests, and confirm transactions by > "priority". What are you going to do if a "malicious" miner decides to go > after their profits and order by what makes them the most money. Add > "ordered by priority" as a consensus requirement? And even if you miners > can still sort their mempool by fee, and then order the top 1MB by priority. > > I entirely agree with your sentiment that Bitcoin must be incentive > compatible. It is necessary. > > It is in only miners immediate interest to make the most profitable block > from the available transaction pool. As with so many other things, it is > necessary to partially ignore short-term gain for long-term benefit. It is > in miners and everybody's long-term interest to have a reliable transaction > service. A busy transaction service that confirms lots of transactions per > hour will become more profitable as demand increases and more users are > prepared to pay for priority. As it is there is currently no way to fully > scale because of the transaction bandwidth limit and that is problematic. > If all valid transactions must eventually confirm then there must be a way > to resolve that problem. > > Bitcoin deliberately removes traditional scale by ensuring blocks take ten > minutes on average to solve, an ingenious idea and, incentive compatible > but, fixed block sizes leaves us with a problem to solve when we want to > scale. > > >If you could find a good solution that would allow you to know if miners > were following your rule or not (and thus ignore it if it doesn't) then you > wouldn't even need bitcoin in the first place. > > I am confident that the math to verify blocks based on the proposal can be > developed (and I think it will not be too complex for a mathematician with > the relevant experience), however, I am nowhere near experienced enough > with probability and statistical analysis to do it. Yes, if Bitcoin doesn't > then it might make another great opportunity for an altcoin but I am not > even nearly interested in promoting any altcoins. > > > If not the proposal that I have put forward, then, hopefully, someone can > come up with a better solution. The important thing is that the issues are > resolved. > > Regards, > Damian Williamson > > > ------------------------------ > *From:* Rhavar <rhavar@protonmail•com> > *Sent:* Saturday, 16 December 2017 3:38 AM > *To:* Damian Williamson > *Cc:* Bitcoin Protocol Discussion > *Subject:* Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use > Transaction Priority For Ordering Transactions In Blocks > > > I understand that there would be technical issues to resolve in > implementation, but, are there no fundamental errors? > > Unfortunately your proposal is really fundamentally broken, on a few > levels. I think you might need to do a bit more research into how bitcoin > works before coming up with such improvements =) > > But just some quick notes: > > * Every node has a (potentially) different mempool, you can't use it to > decide consensus values like the max block size. > > * Increasing the entropy in a block to make it more unpredictable doesn't > really make sense. > > * Bitcoin should be roughly incentive compatible. Your proposal explicits > asks miners to ignore their best interests, and confirm transactions by > "priority". What are you going to do if a "malicious" miner decides to go > after their profits and order by what makes them the most money. Add > "ordered by priority" as a consensus requirement? And even if you miners > can still sort their mempool by fee, and then order the top 1MB by priority. > > If you could find a good solution that would allow you to know if miners > were following your rule or not (and thus ignore it if it doesn't) then you > wouldn't even need bitcoin in the first place. > > > > > -Ryan > > > -------- Original Message -------- > Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction > Priority For Ordering Transactions In Blocks > Local Time: December 15, 2017 3:42 AM > UTC Time: December 15, 2017 9:42 AM > From: bitcoin-dev@lists•linuxfoundation•org > To: Bitcoin Protocol Discussion <bitcoin-dev@lists•linuxfoundation•org> > > > > I should not take it that the lack of critical feedback to this revised > proposal is a glowing endorsement. I understand that there would be > technical issues to resolve in implementation, but, are there no > fundamental errors? > > I suppose that it if is difficult to determine how long a transaction has > been waiting in the pool then, each node could simply keep track of when a > transaction was first seen. This may have implications for a verify > routine, however, for example, if a node was offline, how should it > differentiate how long each transaction was waiting in that case? If a node > was restarted daily would it always think that all transactions had been > waiting in the pool less than one day If each node keeps the current > transaction pool in a file and updates it, as transactions are included in > blocks and, as new transactions appear in the pool, then that would go some > way to alleviate the issue, apart from entirely new nodes. There should be > no reason the contents of a transaction pool files cannot be shared without > agreement as to the transaction pool between nodes, just as nodes > transmit new transactions freely. > > It has been questioned why miners could not cheat. For the question of how > many transactions to include in a block, I say it is a standoff and miners > will conform to the proposal, not wanting to leave transactions with valid > fees standing, and, not wanting to shrink the transaction pool. In any > case, if miners shrink the transaction pool then I am not immediately > concerned since it provides a more efficient service. For the question of > including transactions according to the proposal, I say if it is possible > to keep track of how long transactions are waiting in the pool so that they > can be included on a probability curve then it is possible to verify that > blocks conform to the proposal, since the input is a probability, the > output should conform to a probability curve. > > > If someone has the necessary skill, would anyone be willing to develop the > math necessary for the proposal? > > Regards, > Damian Williamson > > > ------------------------------ > > *From:* bitcoin-dev-bounces@lists•linuxfoundation•org <bit > coin-dev-bounces@lists•linuxfoundation•org> on behalf of Damian > Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> > *Sent:* Friday, 8 December 2017 8:01 AM > *To:* bitcoin-dev@lists•linuxfoundation•org > *Subject:* [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use > Transaction Priority For Ordering Transactions In Blocks > > > Good afternoon, > > The need for this proposal: > > We all must learn to admit that transaction bandwidth is still lurking as > a serious issue for the operation, reliability, safety, consumer > acceptance, uptake and, for the value of Bitcoin. > > I recently sent a payment which was not urgent so; I chose three-day > target confirmation from the fee recommendation. That transaction has still > not confirmed after now more than six days - even waiting twice as long > seems quite reasonable to me. That transaction is a valid transaction; it > is not rubbish, junk or, spam. Under the current model with transaction > bandwidth limitation, the longer a transaction waits, the less likely it is > ever to confirm due to rising transaction numbers and being pushed back by > transactions with rising fees. > > I argue that no transactions are rubbish or junk, only some zero fee > transactions might be spam. Having an ever-increasing number of valid > transactions that do not confirm as more new transactions with higher fees > are created is the opposite of operating a robust, reliable transaction > system. > > Business cannot operate with a model where transactions may or may not > confirm. Even a business choosing a modest fee has no guarantee that their > valid transaction will not be shuffled down by new transactions to the > realm of never confirming after it is created. Consumers also will not > accept this model as Bitcoin expands. If Bitcoin cannot be a reliable > payment system for confirmed transactions then consumers, by and large, > will simply not accept the model once they understand. Bitcoin will be a > dirty payment system, and this will kill the value of Bitcoin. > > Under the current system, a minority of transactions will eventually be > the lucky few who have fees high enough to escape being pushed down the > list. > > Once there are more than x transactions (transaction bandwidth limit) > every ten minutes, only those choosing twenty-minute confirmation (2 > blocks) will have initially at most a fifty percent chance of ever having > their payment confirm. Presently, not even using fee recommendations can > ensure a sufficiently high fee is paid to ensure transaction confirmation. > > I also argue that the current auction model for limited transaction > bandwidth is wrong, is not suitable for a reliable transaction system and, > is wrong for Bitcoin. All transactions must confirm in due time. Currently, > Bitcoin is not a safe way to send payments. > > I do not believe that consumers and business are against paying fees, even > high fees. What is required is operational reliability. > > This great issue needs to be resolved for the safety and reliability of > Bitcoin. The time to resolve issues in commerce is before they become great > big issues. The time to resolve this issue is now. We must have the > foresight to identify and resolve problems before they trip us over. > Simply doubling block sizes every so often is reactionary and is not a > reliable permanent solution. I have written a BIP proposal for a technical > solution but, need your help to write it up to an acceptable standard to be > a full BIP. > > I have formatted the following with markdown which is human readable so, I > hope nobody minds. I have done as much with this proposal as I feel that I > am able so far but continue to take your feedback. > > # BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering > Transactions In Blocks > > ## The problem: > Everybody wants value. Miners want to maximize revenue from fees (and we > presume, to minimize block size). Consumers need transaction reliability > and, (we presume) want low fees. > > The current transaction bandwidth limit is a limiting factor for both. As > the operational safety of transactions is limited, so is consumer > confidence as they realize the issue and, accordingly, uptake is limited. > Fees are artificially inflated due to bandwidth limitations while failing > to provide a full confirmation service for all transactions. > > Current fee recommendations provide no satisfaction for transaction > reliability and, as Bitcoin scales, this will worsen. > > Bitcoin must be a fully scalable and reliable service, providing full > transaction confirmation for every valid transaction. > > The possibility to send a transaction with a fee lower than one that is > acceptable to allow eventual transaction confirmation should be removed > from the protocol and also from the user interface. > > ## Solution summary: > Provide each transaction with an individual transaction priority each time > before choosing transactions to include in the current block, the priority > being a function of the fee paid (on a curve), and the time waiting in the > transaction pool (also on a curve) out to n days (n=60 ?). The transaction > priority to serve as the likelihood of a transaction being included in the > current block, and for determining the order in which transactions are > tried to see if they will be included. > > Use a target block size. Determine the target block size using; current > transaction pool size x ( 1 / (144 x n days ) ) = number of transactions to > be included in the current block. Broadcast the next target block size with > the current block when it is solved so that nodes know the next target > block size for the block that they are building on. > > The curves used for the priority of transactions would have to be > appropriate. Perhaps a mathematician with experience in probability can > develop the right formulae. My thinking is a steep curve. I suppose that > the probability of all transactions should probably account for a > sufficient number of inclusions that the target block size is met although, > it may not always be. As a suggestion, consider including some zero fee > transactions to pad, highest BTC value first? > > **Explanation of the operation of priority:** > > If transaction priority is, for example, a number between one (low) and > one-hundred (high) it can be directly understood as the percentage chance > in one-hundred of a transaction being included in the block. Using > probability or likelihood infers that there is some function of random. If > random (100) < transaction priority then the transaction is included. > > >To break it down further, if both the fee on a curve value and the time > waiting on a curve value are each a number between one and one-hundred, a > rudimentary method may be to simply multiply those two numbers, to find the > priority number. For example, a middle fee transaction waiting thirty days > (if n = 60 days) may have a value of five for each part (yes, just five, > the values are on a curve). When multiplied that will give a priority value > of twenty-five, or, a twenty-five percent chance at that moment of being > included in the block; it will likely be included in one of the next four > blocks, getting more likely each chance. If it is still not included then > the value of time waiting will be higher, making for more probability. A > very low fee transaction would have a value for the fee of one. It would > not be until near sixty-days that the particular low fee transaction has a > high likelihood of being included in the block. > > I am not concerned with low (or high) transaction fees, the primary reason > for addressing the issue is to ensure transactional reliability and > scalability while having each transaction confirm in due time. > > ## Pros: > * Maximizes transaction reliability. > * Fully scalable. > * Maximizes possibility for consumer and business uptake. > * Maximizes total fees paid per block without reducing reliability; > because of reliability, in time confidence and overall uptake are greater; > therefore, more transactions. > * Market determines fee paid for transaction priority. > * Fee recommendations work all the way out to 30 days or greater. > * Provides additional block entropy; greater security since there is less > probability of predicting the next block. > > ## Cons: > * Could initially lower total transaction fees per block. > * Must be first be programmed. > > ## Solution operation: > This is a simplistic view of the operation. The actual operation will need > to be determined in a spec for the programmer. > > 1. Determine the target block size for the current block. > 2. Assign a transaction priority to each transaction in the pool. > 3. Select transactions to include in the current block using probability > in transaction priority order until the target block size is met. > 5. Solve block. > 6. Broadcast the next target block size with the current block when it is > solved. > 7. Block is received. > 8. Block verification process. > 9. Accept/reject block based on verification result. > 10. Repeat. > > ## Closing comments: > It may be possible to verify blocks conform to the proposal by showing > that the probability for all transactions included in the block > statistically conforms to a probability distribution curve, *if* the > individual transaction priority can be recreated. I am not that deep into > the mathematics; however, it may also be possible to use a similar method > to do this just based on the fee, that statistically, the blocks conform to > a fee distribution. Any zero fee transactions would have to be ignored. > This solution needs a clever mathematician. > > I implore, at the very least, that we use some method that validates full > transaction reliability and enables scalability of block sizes. If not this > proposal, an alternative. > > Regards, > Damian Williamson > > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > > > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > >
I suppose what I intended is (2) the weak form but, what is essentially needed is (1) the strong form. The answer may be somewhere in-between.
I do not see that an entire consensus for the mempool is needed, each node just needs a loose understanding of the average number of non-zero fee transactions in the mempool.
As a pre-rollout, it would be possible to give each node a serial ID and, calculate the average number of non-zero fee transactions from the information it has and, say every ten minutes, distribute information it has about the number of transactions in the mempool. Each node would be able to form its own picture of the average number of non-zero fee transactions in the mempool.
At rollout, this information would be the basis a node would use when a block is solved to provide the next expected block size. This would still not stop cheating by providing especially a number lower than the proposal would allow for, to game the system and hike fees. If miners will not act in the long-term interest of the stability and operation of the system then they should be ignored. If most miners will adhere to the proposal then the average effect would be stability in the operation of the proposal, having a few or even several nodes posting low numbers for the number of transactions expected in the next expected block size would not destroy the operation. If some node posted an insanely high number for next expected block size that resulted in the mempool being emptied then the proposal would be offended but I do not actually care. If no number is posted, just create a block the appropriate size ensure conformity. Nodes that have not adopted the proposal could just continue to create 1MB blocks.
Actually, the operation could be simplified using the distributed information directly to just create blocks of the appropriate size with no need to provide next block size. Flexible block size.
The proposal should also specify a minimum number of transactions to include for the next block to give at a minimum a 1MB block.
I currently have no information on flex cap, do you have a link?
Regards,
Damian Williamson
________________________________ From: Mark Friedenbach <mark@friedenbach•org> Sent: Tuesday, 19 December 2017 3:08 AM To: Damian Williamson Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Damian, you seem to be misunderstanding that either
(1) the strong form of your proposal requires validating the commitment to the mempool properties, in which case the mempool becomes consensus critical (an impossible requirement); or
(2) in the weak form where the current block is dependent on the commitment in the last block only it is becomes a miner-selected field they can freely parameterize with no repercussions for setting values totally independent of the actual mempool.
If you want to make the block size dependent on the properties of the mempool in a consensus critical way, flex cap achieves this. If you want to make the contents or properties of the mempool known to well-connected nodes, weak blocks achieves that. But you can’t stick the mempool in consensus because it fundamentally is not something the nodes have consensus over. That’s a chicken-and-the-egg assumption.
Finally in terms of the broad goal, having block size based on the number of transactions is NOT something desirable in the first place, even if it did work. That’s effectively the same as an infinite block size since anyone anywhere can create transactions in the mempool at no cost.
On Dec 16, 2017, at 8:14 PM, Damian Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
I do not know why people make the leap that the proposal requires a consensus on the transaction pool. It does not.
It may be helpful to have the discussion from the previous thread linked here. https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-December/015370.html
Where I speak of validating that a block conforms to the broadcast next block size, I do not propose validating the number broadcast for the next block size itself, only that the next generated block is that size.
Regards, Damian Williamson
________________________________ From: Damian Williamson <willtech@live•com•au<mailto:willtech@live.com.au>> Sent: Saturday, 16 December 2017 7:59 AM To: Rhavar Cc: Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
There are really two separate problems to solve.
1. How does Bitcoin scale with fixed block size? 2. How do we ensure that all valid transactions are eventually included in the blockchain?
Those are the two issues that the proposal attempts to address. It makes sense to resolve these two problems together. Using the proposed system for variable block sizes would solve the first problem but there would still be a whole bunch of never confirming transactions. I am not sure how to reliably solve the second problem at scale without first solving the first.
Quoted text (1 line)
>* Every node has a (potentially) different mempool, you can't use it to decide consensus values like the max block size.
I do not suggest a consensus. Depending on which node solves a block the value for next block size will be different. The consensus would be that blocks will adhere to the next block size value transmitted with the current block. It is easy to verify that the consensus is being adhered to once in place.
Quoted text (1 line)
>* Increasing the entropy in a block to make it more unpredictable doesn't really make sense.
Not a necessary function, just an effect of using a probability-based distribution.
Quoted text (1 line)
>* Bitcoin should be roughly incentive compatible. Your proposal explicits asks miners to ignore their best interests, and confirm transactions by "priority". What are you going to do if a "malicious" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a consensus requirement? And even if you miners can still sort their mempool by fee, and then order the top 1MB by priority.
I entirely agree with your sentiment that Bitcoin must be incentive compatible. It is necessary.
It is in only miners immediate interest to make the most profitable block from the available transaction pool. As with so many other things, it is necessary to partially ignore short-term gain for long-term benefit. It is in miners and everybody's long-term interest to have a reliable transaction service. A busy transaction service that confirms lots of transactions per hour will become more profitable as demand increases and more users are prepared to pay for priority. As it is there is currently no way to fully scale because of the transaction bandwidth limit and that is problematic. If all valid transactions must eventually confirm then there must be a way to resolve that problem.
Bitcoin deliberately removes traditional scale by ensuring blocks take ten minutes on average to solve, an ingenious idea and, incentive compatible but, fixed block sizes leaves us with a problem to solve when we want to scale.
Quoted text (1 line)
>If you could find a good solution that would allow you to know if miners were following your rule or not (and thus ignore it if it doesn't) then you wouldn't even need bitcoin in the first place.
I am confident that the math to verify blocks based on the proposal can be developed (and I think it will not be too complex for a mathematician with the relevant experience), however, I am nowhere near experienced enough with probability and statistical analysis to do it. Yes, if Bitcoin doesn't then it might make another great opportunity for an altcoin but I am not even nearly interested in promoting any altcoins.
If not the proposal that I have put forward, then, hopefully, someone can come up with a better solution. The important thing is that the issues are resolved.
Regards, Damian Williamson
________________________________ From: Rhavar <rhavar@protonmail•com<mailto:rhavar@protonmail.com>> Sent: Saturday, 16 December 2017 3:38 AM To: Damian Williamson Cc: Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Quoted text (1 line)
> I understand that there would be technical issues to resolve in implementation, but, are there no fundamental errors?
Unfortunately your proposal is really fundamentally broken, on a few levels. I think you might need to do a bit more research into how bitcoin works before coming up with such improvements =)
But just some quick notes:
* Every node has a (potentially) different mempool, you can't use it to decide consensus values like the max block size.
* Increasing the entropy in a block to make it more unpredictable doesn't really make sense.
* Bitcoin should be roughly incentive compatible. Your proposal explicits asks miners to ignore their best interests, and confirm transactions by "priority". What are you going to do if a "malicious" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a consensus requirement? And even if you miners can still sort their mempool by fee, and then order the top 1MB by priority.
If you could find a good solution that would allow you to know if miners were following your rule or not (and thus ignore it if it doesn't) then you wouldn't even need bitcoin in the first place.
-Ryan
-------- Original Message -------- Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks Local Time: December 15, 2017 3:42 AM UTC Time: December 15, 2017 9:42 AM From: bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org> To: Bitcoin Protocol Discussion <bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org>>
I should not take it that the lack of critical feedback to this revised proposal is a glowing endorsement. I understand that there would be technical issues to resolve in implementation, but, are there no fundamental errors?
I suppose that it if is difficult to determine how long a transaction has been waiting in the pool then, each node could simply keep track of when a transaction was first seen. This may have implications for a verify routine, however, for example, if a node was offline, how should it differentiate how long each transaction was waiting in that case? If a node was restarted daily would it always think that all transactions had been waiting in the pool less than one day If each node keeps the current transaction pool in a file and updates it, as transactions are included in blocks and, as new transactions appear in the pool, then that would go some way to alleviate the issue, apart from entirely new nodes. There should be no reason the contents of a transaction pool files cannot be shared without agreement as to the transaction pool between nodes, just as nodes transmit new transactions freely.
It has been questioned why miners could not cheat. For the question of how many transactions to include in a block, I say it is a standoff and miners will conform to the proposal, not wanting to leave transactions with valid fees standing, and, not wanting to shrink the transaction pool. In any case, if miners shrink the transaction pool then I am not immediately concerned since it provides a more efficient service. For the question of including transactions according to the proposal, I say if it is possible to keep track of how long transactions are waiting in the pool so that they can be included on a probability curve then it is possible to verify that blocks conform to the proposal, since the input is a probability, the output should conform to a probability curve.
If someone has the necessary skill, would anyone be willing to develop the math necessary for the proposal?
Regards, Damian Williamson
________________________________
From: bitcoin-dev-bounces@lists•linuxfoundation•org<mailto:bitcoin-dev-bounces@lists.linuxfoundation.org> <bitcoin-dev-bounces@lists•linuxfoundation•org<mailto:bitcoin-dev-bounces@lists.linuxfoundation.org>> on behalf of Damian Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org>> Sent: Friday, 8 December 2017 8:01 AM To: bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org> Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Good afternoon,
The need for this proposal:
We all must learn to admit that transaction bandwidth is still lurking as a serious issue for the operation, reliability, safety, consumer acceptance, uptake and, for the value of Bitcoin.
I recently sent a payment which was not urgent so; I chose three-day target confirmation from the fee recommendation. That transaction has still not confirmed after now more than six days - even waiting twice as long seems quite reasonable to me. That transaction is a valid transaction; it is not rubbish, junk or, spam. Under the current model with transaction bandwidth limitation, the longer a transaction waits, the less likely it is ever to confirm due to rising transaction numbers and being pushed back by transactions with rising fees.
I argue that no transactions are rubbish or junk, only some zero fee transactions might be spam. Having an ever-increasing number of valid transactions that do not confirm as more new transactions with higher fees are created is the opposite of operating a robust, reliable transaction system.
Business cannot operate with a model where transactions may or may not confirm. Even a business choosing a modest fee has no guarantee that their valid transaction will not be shuffled down by new transactions to the realm of never confirming after it is created. Consumers also will not accept this model as Bitcoin expands. If Bitcoin cannot be a reliable payment system for confirmed transactions then consumers, by and large, will simply not accept the model once they understand. Bitcoin will be a dirty payment system, and this will kill the value of Bitcoin.
Under the current system, a minority of transactions will eventually be the lucky few who have fees high enough to escape being pushed down the list.
Once there are more than x transactions (transaction bandwidth limit) every ten minutes, only those choosing twenty-minute confirmation (2 blocks) will have initially at most a fifty percent chance of ever having their payment confirm. Presently, not even using fee recommendations can ensure a sufficiently high fee is paid to ensure transaction confirmation.
I also argue that the current auction model for limited transaction bandwidth is wrong, is not suitable for a reliable transaction system and, is wrong for Bitcoin. All transactions must confirm in due time. Currently, Bitcoin is not a safe way to send payments.
I do not believe that consumers and business are against paying fees, even high fees. What is required is operational reliability.
This great issue needs to be resolved for the safety and reliability of Bitcoin. The time to resolve issues in commerce is before they become great big issues. The time to resolve this issue is now. We must have the foresight to identify and resolve problems before they trip us over. Simply doubling block sizes every so often is reactionary and is not a reliable permanent solution. I have written a BIP proposal for a technical solution but, need your help to write it up to an acceptable standard to be a full BIP.
I have formatted the following with markdown which is human readable so, I hope nobody minds. I have done as much with this proposal as I feel that I am able so far but continue to take your feedback.
# BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
## The problem: Everybody wants value. Miners want to maximize revenue from fees (and we presume, to minimize block size). Consumers need transaction reliability and, (we presume) want low fees.
The current transaction bandwidth limit is a limiting factor for both. As the operational safety of transactions is limited, so is consumer confidence as they realize the issue and, accordingly, uptake is limited. Fees are artificially inflated due to bandwidth limitations while failing to provide a full confirmation service for all transactions.
Current fee recommendations provide no satisfaction for transaction reliability and, as Bitcoin scales, this will worsen.
Bitcoin must be a fully scalable and reliable service, providing full transaction confirmation for every valid transaction.
The possibility to send a transaction with a fee lower than one that is acceptable to allow eventual transaction confirmation should be removed from the protocol and also from the user interface.
## Solution summary: Provide each transaction with an individual transaction priority each time before choosing transactions to include in the current block, the priority being a function of the fee paid (on a curve), and the time waiting in the transaction pool (also on a curve) out to n days (n=60 ?). The transaction priority to serve as the likelihood of a transaction being included in the current block, and for determining the order in which transactions are tried to see if they will be included.
Use a target block size. Determine the target block size using; current transaction pool size x ( 1 / (144 x n days ) ) = number of transactions to be included in the current block. Broadcast the next target block size with the current block when it is solved so that nodes know the next target block size for the block that they are building on.
The curves used for the priority of transactions would have to be appropriate. Perhaps a mathematician with experience in probability can develop the right formulae. My thinking is a steep curve. I suppose that the probability of all transactions should probably account for a sufficient number of inclusions that the target block size is met although, it may not always be. As a suggestion, consider including some zero fee transactions to pad, highest BTC value first?
**Explanation of the operation of priority:**
Quoted text (2 lines)
> If transaction priority is, for example, a number between one (low) and one-hundred (high) it can be directly understood as the percentage chance in one-hundred of a transaction being included in the block. Using probability or likelihood infers that there is some function of random. If random (100) < transaction priority then the transaction is included. >To break it down further, if both the fee on a curve value and the time waiting on a curve value are each a number between one and one-hundred, a rudimentary method may be to simply multiply those two numbers, to find the priority number. For example, a middle fee transaction waiting thirty days (if n = 60 days) may have a value of five for each part (yes, just five, the values are on a curve). When multiplied that will give a priority value of twenty-five, or, a twenty-five percent chance at that moment of being included in the block; it will likely be included in one of the next four blocks, getting more likely each chance. If it is still not included then the value of time waiting will be higher, making for more probability. A very low fee transaction would have a value for the fee of one. It would not be until near sixty-days that the particular low fee transaction has a high likelihood of being included in the block.
I am not concerned with low (or high) transaction fees, the primary reason for addressing the issue is to ensure transactional reliability and scalability while having each transaction confirm in due time.
## Pros: * Maximizes transaction reliability. * Fully scalable. * Maximizes possibility for consumer and business uptake. * Maximizes total fees paid per block without reducing reliability; because of reliability, in time confidence and overall uptake are greater; therefore, more transactions. * Market determines fee paid for transaction priority. * Fee recommendations work all the way out to 30 days or greater. * Provides additional block entropy; greater security since there is less probability of predicting the next block.
## Cons: * Could initially lower total transaction fees per block. * Must be first be programmed.
## Solution operation: This is a simplistic view of the operation. The actual operation will need to be determined in a spec for the programmer.
1. Determine the target block size for the current block. 2. Assign a transaction priority to each transaction in the pool. 3. Select transactions to include in the current block using probability in transaction priority order until the target block size is met. 5. Solve block. 6. Broadcast the next target block size with the current block when it is solved. 7. Block is received. 8. Block verification process. 9. Accept/reject block based on verification result. 10. Repeat.
## Closing comments: It may be possible to verify blocks conform to the proposal by showing that the probability for all transactions included in the block statistically conforms to a probability distribution curve, *if* the individual transaction priority can be recreated. I am not that deep into the mathematics; however, it may also be possible to use a similar method to do this just based on the fee, that statistically, the blocks conform to a fee distribution. Any zero fee transactions would have to be ignored. This solution needs a clever mathematician.
I implore, at the very least, that we use some method that validates full transaction reliability and enables scalability of block sizes. If not this proposal, an alternative.
Regards, Damian Williamson
Quoted text (2 lines)
>.. This system has flaws, they all do. >The simple fact is that there is currently no known system that works as well as the current system..
Alright, but, we seem to agree, the current system also has flaws. The transaction bandwidth limit is a serious issue for transactional reliability.
What you have proposed is interesting but seems to do nothing for the issue of transaction bandwidth, which seems to be approaching its threshold:
https://bitinfocharts.com/comparison/bitcoin-transactions.html
Regards,
Damian Williamson
________________________________ From: Spartacus Rex <spartacusrex99@gmail•com> Sent: Saturday, 23 December 2017 5:07:49 AM To: Damian Williamson; Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Hi Damian,
Thought I'd chip in. This is a hard fork scenario. This system has flaws, they all do.
If you had a fixed fee per block, so that every txn in that block paid the same fee, that might make it easier to include all txns eventually, as you envisage.
The fee could be calculated as the average of the amount txns are prepared to pay in the last 1000 blocks.
A txn would say ' I'll pay up to X bitcoins ' and as long as that is more than the value required for the block your txn can be added. This is to ensure you don't pay more than you are willing. It also ensures that putting an enormous fee will not ensure your txn is processed quickly..
Calculating what the outputs are given a variable fee needs a new mechanism all of it's own, but I'm sure it's possible.
The simple fact is that there is currently no known system that works as well as the current system..
But there are other systems.
On Dec 22, 2017 15:09, "Damian Williamson via bitcoin-dev" <bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
If the cash value of Bitcoin was high enough and zero fee transactions were never accepted and not counted when calculating the transaction pool size then I do not think it would be such an issue. Why is it even possible to create zero fee transactions?
Regards,
Damian Williamson
________________________________ From: bitcoin-dev-bounces@lists•linuxfoundation•org<mailto:bitcoin-dev-bounces@lists.linuxfoundation.org> <bitcoin-dev-bounces@lists•linuxfoundation•org<mailto:bitcoin-dev-bounces@lists.linuxfoundation.org>> on behalf of Damian Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org>> Sent: Tuesday, 19 December 2017 6:51 PM To: Mark Friedenbach Cc: bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org> Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Thank you for your constructive feedback. I now see that the proposal introduces a potential issue.
Quoted text (1 line)
>Finally in terms of the broad goal, having block size based on the number of transactions is NOT something desirable in the first place, even if it did work. That’s effectively the same as an infinite block size since anyone anywhere can create transactions in the mempool at no cost.
Do you have any critical suggestion as to how transaction bandwidth limit could be addressed, it will eventually become an issue if nothing is changed regardless of how high fees go?
Regards,
Damian Williamson
________________________________ From: Mark Friedenbach <mark@friedenbach•org<mailto:mark@friedenbach.org>> Sent: Tuesday, 19 December 2017 3:08 AM To: Damian Williamson Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Damian, you seem to be misunderstanding that either
(1) the strong form of your proposal requires validating the commitment to the mempool properties, in which case the mempool becomes consensus critical (an impossible requirement); or
(2) in the weak form where the current block is dependent on the commitment in the last block only it is becomes a miner-selected field they can freely parameterize with no repercussions for setting values totally independent of the actual mempool.
If you want to make the block size dependent on the properties of the mempool in a consensus critical way, flex cap achieves this. If you want to make the contents or properties of the mempool known to well-connected nodes, weak blocks achieves that. But you can’t stick the mempool in consensus because it fundamentally is not something the nodes have consensus over. That’s a chicken-and-the-egg assumption.
Finally in terms of the broad goal, having block size based on the number of transactions is NOT something desirable in the first place, even if it did work. That’s effectively the same as an infinite block size since anyone anywhere can create transactions in the mempool at no cost.
On Dec 16, 2017, at 8:14 PM, Damian Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
I do not know why people make the leap that the proposal requires a consensus on the transaction pool. It does not.
It may be helpful to have the discussion from the previous thread linked here. https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-December/015370.html
Where I speak of validating that a block conforms to the broadcast next block size, I do not propose validating the number broadcast for the next block size itself, only that the next generated block is that size.
Regards, Damian Williamson
________________________________ From: Damian Williamson <willtech@live•com•au<mailto:willtech@live.com.au>> Sent: Saturday, 16 December 2017 7:59 AM To: Rhavar Cc: Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
There are really two separate problems to solve.
1. How does Bitcoin scale with fixed block size? 2. How do we ensure that all valid transactions are eventually included in the blockchain?
Those are the two issues that the proposal attempts to address. It makes sense to resolve these two problems together. Using the proposed system for variable block sizes would solve the first problem but there would still be a whole bunch of never confirming transactions. I am not sure how to reliably solve the second problem at scale without first solving the first.
Quoted text (1 line)
>* Every node has a (potentially) different mempool, you can't use it to decide consensus values like the max block size.
I do not suggest a consensus. Depending on which node solves a block the value for next block size will be different. The consensus would be that blocks will adhere to the next block size value transmitted with the current block. It is easy to verify that the consensus is being adhered to once in place.
Quoted text (1 line)
>* Increasing the entropy in a block to make it more unpredictable doesn't really make sense.
Not a necessary function, just an effect of using a probability-based distribution.
Quoted text (1 line)
>* Bitcoin should be roughly incentive compatible. Your proposal explicits asks miners to ignore their best interests, and confirm transactions by "priority". What are you going to do if a "malicious" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a consensus requirement? And even if you miners can still sort their mempool by fee, and then order the top 1MB by priority.
I entirely agree with your sentiment that Bitcoin must be incentive compatible. It is necessary.
It is in only miners immediate interest to make the most profitable block from the available transaction pool. As with so many other things, it is necessary to partially ignore short-term gain for long-term benefit. It is in miners and everybody's long-term interest to have a reliable transaction service. A busy transaction service that confirms lots of transactions per hour will become more profitable as demand increases and more users are prepared to pay for priority. As it is there is currently no way to fully scale because of the transaction bandwidth limit and that is problematic. If all valid transactions must eventually confirm then there must be a way to resolve that problem.
Bitcoin deliberately removes traditional scale by ensuring blocks take ten minutes on average to solve, an ingenious idea and, incentive compatible but, fixed block sizes leaves us with a problem to solve when we want to scale.
Quoted text (1 line)
>If you could find a good solution that would allow you to know if miners were following your rule or not (and thus ignore it if it doesn't) then you wouldn't even need bitcoin in the first place.
I am confident that the math to verify blocks based on the proposal can be developed (and I think it will not be too complex for a mathematician with the relevant experience), however, I am nowhere near experienced enough with probability and statistical analysis to do it. Yes, if Bitcoin doesn't then it might make another great opportunity for an altcoin but I am not even nearly interested in promoting any altcoins.
If not the proposal that I have put forward, then, hopefully, someone can come up with a better solution. The important thing is that the issues are resolved.
Regards, Damian Williamson
________________________________ From: Rhavar <rhavar@protonmail•com<mailto:rhavar@protonmail.com>> Sent: Saturday, 16 December 2017 3:38 AM To: Damian Williamson Cc: Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Quoted text (1 line)
> I understand that there would be technical issues to resolve in implementation, but, are there no fundamental errors?
Unfortunately your proposal is really fundamentally broken, on a few levels. I think you might need to do a bit more research into how bitcoin works before coming up with such improvements =)
But just some quick notes:
* Every node has a (potentially) different mempool, you can't use it to decide consensus values like the max block size.
* Increasing the entropy in a block to make it more unpredictable doesn't really make sense.
* Bitcoin should be roughly incentive compatible. Your proposal explicits asks miners to ignore their best interests, and confirm transactions by "priority". What are you going to do if a "malicious" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a consensus requirement? And even if you miners can still sort their mempool by fee, and then order the top 1MB by priority.
If you could find a good solution that would allow you to know if miners were following your rule or not (and thus ignore it if it doesn't) then you wouldn't even need bitcoin in the first place.
-Ryan
-------- Original Message -------- Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks Local Time: December 15, 2017 3:42 AM UTC Time: December 15, 2017 9:42 AM From: bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org> To: Bitcoin Protocol Discussion <bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org>>
I should not take it that the lack of critical feedback to this revised proposal is a glowing endorsement. I understand that there would be technical issues to resolve in implementation, but, are there no fundamental errors?
I suppose that it if is difficult to determine how long a transaction has been waiting in the pool then, each node could simply keep track of when a transaction was first seen. This may have implications for a verify routine, however, for example, if a node was offline, how should it differentiate how long each transaction was waiting in that case? If a node was restarted daily would it always think that all transactions had been waiting in the pool less than one day If each node keeps the current transaction pool in a file and updates it, as transactions are included in blocks and, as new transactions appear in the pool, then that would go some way to alleviate the issue, apart from entirely new nodes. There should be no reason the contents of a transaction pool files cannot be shared without agreement as to the transaction pool between nodes, just as nodes transmit new transactions freely.
It has been questioned why miners could not cheat. For the question of how many transactions to include in a block, I say it is a standoff and miners will conform to the proposal, not wanting to leave transactions with valid fees standing, and, not wanting to shrink the transaction pool. In any case, if miners shrink the transaction pool then I am not immediately concerned since it provides a more efficient service. For the question of including transactions according to the proposal, I say if it is possible to keep track of how long transactions are waiting in the pool so that they can be included on a probability curve then it is possible to verify that blocks conform to the proposal, since the input is a probability, the output should conform to a probability curve.
If someone has the necessary skill, would anyone be willing to develop the math necessary for the proposal?
Regards, Damian Williamson
________________________________
From: bitcoin-dev-bounces@lists•linuxfoundation•org<mailto:bitcoin-dev-bounces@lists.linuxfoundation.org> <bitcoin-dev-bounces@lists•linuxfoundation•org<mailto:bitcoin-dev-bounces@lists.linuxfoundation.org>> on behalf of Damian Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org>> Sent: Friday, 8 December 2017 8:01 AM To: bitcoin-dev@lists•linuxfoundation•org<mailto:bitcoin-dev@lists.linuxfoundation.org> Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Good afternoon,
The need for this proposal:
We all must learn to admit that transaction bandwidth is still lurking as a serious issue for the operation, reliability, safety, consumer acceptance, uptake and, for the value of Bitcoin.
I recently sent a payment which was not urgent so; I chose three-day target confirmation from the fee recommendation. That transaction has still not confirmed after now more than six days - even waiting twice as long seems quite reasonable to me. That transaction is a valid transaction; it is not rubbish, junk or, spam. Under the current model with transaction bandwidth limitation, the longer a transaction waits, the less likely it is ever to confirm due to rising transaction numbers and being pushed back by transactions with rising fees.
I argue that no transactions are rubbish or junk, only some zero fee transactions might be spam. Having an ever-increasing number of valid transactions that do not confirm as more new transactions with higher fees are created is the opposite of operating a robust, reliable transaction system.
Business cannot operate with a model where transactions may or may not confirm. Even a business choosing a modest fee has no guarantee that their valid transaction will not be shuffled down by new transactions to the realm of never confirming after it is created. Consumers also will not accept this model as Bitcoin expands. If Bitcoin cannot be a reliable payment system for confirmed transactions then consumers, by and large, will simply not accept the model once they understand. Bitcoin will be a dirty payment system, and this will kill the value of Bitcoin.
Under the current system, a minority of transactions will eventually be the lucky few who have fees high enough to escape being pushed down the list.
Once there are more than x transactions (transaction bandwidth limit) every ten minutes, only those choosing twenty-minute confirmation (2 blocks) will have initially at most a fifty percent chance of ever having their payment confirm. Presently, not even using fee recommendations can ensure a sufficiently high fee is paid to ensure transaction confirmation.
I also argue that the current auction model for limited transaction bandwidth is wrong, is not suitable for a reliable transaction system and, is wrong for Bitcoin. All transactions must confirm in due time. Currently, Bitcoin is not a safe way to send payments.
I do not believe that consumers and business are against paying fees, even high fees. What is required is operational reliability.
This great issue needs to be resolved for the safety and reliability of Bitcoin. The time to resolve issues in commerce is before they become great big issues. The time to resolve this issue is now. We must have the foresight to identify and resolve problems before they trip us over. Simply doubling block sizes every so often is reactionary and is not a reliable permanent solution. I have written a BIP proposal for a technical solution but, need your help to write it up to an acceptable standard to be a full BIP.
I have formatted the following with markdown which is human readable so, I hope nobody minds. I have done as much with this proposal as I feel that I am able so far but continue to take your feedback.
# BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
## The problem: Everybody wants value. Miners want to maximize revenue from fees (and we presume, to minimize block size). Consumers need transaction reliability and, (we presume) want low fees.
The current transaction bandwidth limit is a limiting factor for both. As the operational safety of transactions is limited, so is consumer confidence as they realize the issue and, accordingly, uptake is limited. Fees are artificially inflated due to bandwidth limitations while failing to provide a full confirmation service for all transactions.
Current fee recommendations provide no satisfaction for transaction reliability and, as Bitcoin scales, this will worsen.
Bitcoin must be a fully scalable and reliable service, providing full transaction confirmation for every valid transaction.
The possibility to send a transaction with a fee lower than one that is acceptable to allow eventual transaction confirmation should be removed from the protocol and also from the user interface.
## Solution summary: Provide each transaction with an individual transaction priority each time before choosing transactions to include in the current block, the priority being a function of the fee paid (on a curve), and the time waiting in the transaction pool (also on a curve) out to n days (n=60 ?). The transaction priority to serve as the likelihood of a transaction being included in the current block, and for determining the order in which transactions are tried to see if they will be included.
Use a target block size. Determine the target block size using; current transaction pool size x ( 1 / (144 x n days ) ) = number of transactions to be included in the current block. Broadcast the next target block size with the current block when it is solved so that nodes know the next target block size for the block that they are building on.
The curves used for the priority of transactions would have to be appropriate. Perhaps a mathematician with experience in probability can develop the right formulae. My thinking is a steep curve. I suppose that the probability of all transactions should probably account for a sufficient number of inclusions that the target block size is met although, it may not always be. As a suggestion, consider including some zero fee transactions to pad, highest BTC value first?
**Explanation of the operation of priority:**
Quoted text (2 lines)
> If transaction priority is, for example, a number between one (low) and one-hundred (high) it can be directly understood as the percentage chance in one-hundred of a transaction being included in the block. Using probability or likelihood infers that there is some function of random. If random (100) < transaction priority then the transaction is included. >To break it down further, if both the fee on a curve value and the time waiting on a curve value are each a number between one and one-hundred, a rudimentary method may be to simply multiply those two numbers, to find the priority number. For example, a middle fee transaction waiting thirty days (if n = 60 days) may have a value of five for each part (yes, just five, the values are on a curve). When multiplied that will give a priority value of twenty-five, or, a twenty-five percent chance at that moment of being included in the block; it will likely be included in one of the next four blocks, getting more likely each chance. If it is still not included then the value of time waiting will be higher, making for more probability. A very low fee transaction would have a value for the fee of one. It would not be until near sixty-days that the particular low fee transaction has a high likelihood of being included in the block.
I am not concerned with low (or high) transaction fees, the primary reason for addressing the issue is to ensure transactional reliability and scalability while having each transaction confirm in due time.
## Pros: * Maximizes transaction reliability. * Fully scalable. * Maximizes possibility for consumer and business uptake. * Maximizes total fees paid per block without reducing reliability; because of reliability, in time confidence and overall uptake are greater; therefore, more transactions. * Market determines fee paid for transaction priority. * Fee recommendations work all the way out to 30 days or greater. * Provides additional block entropy; greater security since there is less probability of predicting the next block.
## Cons: * Could initially lower total transaction fees per block. * Must be first be programmed.
## Solution operation: This is a simplistic view of the operation. The actual operation will need to be determined in a spec for the programmer.
1. Determine the target block size for the current block. 2. Assign a transaction priority to each transaction in the pool. 3. Select transactions to include in the current block using probability in transaction priority order until the target block size is met. 5. Solve block. 6. Broadcast the next target block size with the current block when it is solved. 7. Block is received. 8. Block verification process. 9. Accept/reject block based on verification result. 10. Repeat.
## Closing comments: It may be possible to verify blocks conform to the proposal by showing that the probability for all transactions included in the block statistically conforms to a probability distribution curve, *if* the individual transaction priority can be recreated. I am not that deep into the mathematics; however, it may also be possible to use a similar method to do this just based on the fee, that statistically, the blocks conform to a fee distribution. Any zero fee transactions would have to be ignored. This solution needs a clever mathematician.
I implore, at the very least, that we use some method that validates full transaction reliability and enables scalability of block sizes. If not this proposal, an alternative.
Regards, Damian Williamson
..What you have proposed is interesting but seems to do nothing for the issue of transaction bandwidth, which seems to be approaching its threshold: ..
This system just shows one way of changing the way a miner calculates txn priority.
A miners should always do what makes him the most money, so an old txn will never get priority if a newer one offering more fees comes along. This is why some txns will never get confirmation.
In this system a txn cannot just pay more fees, as you all pay the same fees in a block, so an old txn that has a high enough threshold will be worth just as much to a miner as any txn coming later on.
This way you can be sure that your txn will confirm at some point, and not just be relegated to the 'never' confirmed pile.
On Dec 24, 2017 03:44, "Damian Williamson" <willtech@live•com•au> wrote:
Quoted text (2 lines)
>.. This system has flaws, they all do. >The simple fact is that there is currently no known system that works as
well as the current system..
Alright, but, we seem to agree, the current system also has flaws. The transaction bandwidth limit is a serious issue for transactional reliability.
What you have proposed is interesting but seems to do nothing for the issue of transaction bandwidth, which seems to be approaching its threshold:
https://bitinfocharts.com/comparison/bitcoin-transactions.html
Regards,
Damian Williamson ------------------------------ *From:* Spartacus Rex <spartacusrex99@gmail•com> *Sent:* Saturday, 23 December 2017 5:07:49 AM *To:* Damian Williamson; Bitcoin Protocol Discussion
*Subject:* Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Hi Damian,
Thought I'd chip in. This is a hard fork scenario. This system has flaws, they all do.
If you had a fixed fee per block, so that every txn in that block paid the same fee, that might make it easier to include all txns eventually, as you envisage.
The fee could be calculated as the average of the amount txns are prepared to pay in the last 1000 blocks.
A txn would say ' I'll pay up to X bitcoins ' and as long as that is more than the value required for the block your txn can be added. This is to ensure you don't pay more than you are willing. It also ensures that putting an enormous fee will not ensure your txn is processed quickly..
Calculating what the outputs are given a variable fee needs a new mechanism all of it's own, but I'm sure it's possible.
The simple fact is that there is currently no known system that works as well as the current system..
But there are other systems.
On Dec 22, 2017 15:09, "Damian Williamson via bitcoin-dev" < bitcoin-dev@lists•linuxfoundation•org> wrote:
If the cash value of Bitcoin was high enough and zero fee transactions were never accepted and not counted when calculating the transaction pool size then I do not think it would be such an issue. Why is it even possible to create zero fee transactions?
Regards,
Damian Williamson
------------------------------ *From:* bitcoin-dev-bounces@lists•linuxfoundation•org < bitcoin-dev-bounces@lists•linuxfoundation•org> on behalf of Damian Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> *Sent:* Tuesday, 19 December 2017 6:51 PM *To:* Mark Friedenbach *Cc:* bitcoin-dev@lists•linuxfoundation•org *Subject:* [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Thank you for your constructive feedback. I now see that the proposal introduces a potential issue.
Quoted text (1 line)
>Finally in terms of the broad goal, having block size based on the number
of transactions is NOT something desirable in the first place, even if it did work. That’s effectively the same as an infinite block size since anyone anywhere can create transactions in the mempool at no cost.
Do you have any critical suggestion as to how transaction bandwidth limit could be addressed, it will eventually become an issue if nothing is changed regardless of how high fees go?
Regards, Damian Williamson
------------------------------ *From:* Mark Friedenbach <mark@friedenbach•org> *Sent:* Tuesday, 19 December 2017 3:08 AM *To:* Damian Williamson *Subject:* Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Damian, you seem to be misunderstanding that either
(1) the strong form of your proposal requires validating the commitment to the mempool properties, in which case the mempool becomes consensus critical (an impossible requirement); or
(2) in the weak form where the current block is dependent on the commitment in the last block only it is becomes a miner-selected field they can freely parameterize with no repercussions for setting values totally independent of the actual mempool.
If you want to make the block size dependent on the properties of the mempool in a consensus critical way, flex cap achieves this. If you want to make the contents or properties of the mempool known to well-connected nodes, weak blocks achieves that. But you can’t stick the mempool in consensus because it fundamentally is not something the nodes have consensus over. That’s a chicken-and-the-egg assumption.
Finally in terms of the broad goal, having block size based on the number of transactions is NOT something desirable in the first place, even if it did work. That’s effectively the same as an infinite block size since anyone anywhere can create transactions in the mempool at no cost.
On Dec 16, 2017, at 8:14 PM, Damian Williamson via bitcoin-dev < bitcoin-dev@lists•linuxfoundation•org> wrote:
I do not know why people make the leap that the proposal requires a consensus on the transaction pool. It does not.
It may be helpful to have the discussion from the previous thread linked here. https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017 -December/015370.html
Where I speak of validating that a block conforms to the broadcast next block size, I do not propose validating the number broadcast for the next block size itself, only that the next generated block is that size.
Regards, Damian Williamson
------------------------------ *From:* Damian Williamson <willtech@live•com•au> *Sent:* Saturday, 16 December 2017 7:59 AM *To:* Rhavar *Cc:* Bitcoin Protocol Discussion *Subject:* Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
There are really two separate problems to solve.
1. How does Bitcoin scale with fixed block size? 2. How do we ensure that all valid transactions are eventually included in the blockchain?
Those are the two issues that the proposal attempts to address. It makes sense to resolve these two problems together. Using the proposed system for variable block sizes would solve the first problem but there would still be a whole bunch of never confirming transactions. I am not sure how to reliably solve the second problem at scale without first solving the first.
Quoted text (1 line)
>* Every node has a (potentially) different mempool, you can't use it to
decide consensus values like the max block size.
I do not suggest a consensus. Depending on which node solves a block the value for next block size will be different. The consensus would be that blocks will adhere to the next block size value transmitted with the current block. It is easy to verify that the consensus is being adhered to once in place.
Quoted text (1 line)
>* Increasing the entropy in a block to make it more unpredictable doesn't
really make sense.
Not a necessary function, just an effect of using a probability-based distribution.
Quoted text (1 line)
>* Bitcoin should be roughly incentive compatible. Your proposal explicits
asks miners to ignore their best interests, and confirm transactions by "priority". What are you going to do if a "malicious" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a consensus requirement? And even if you miners can still sort their mempool by fee, and then order the top 1MB by priority.
I entirely agree with your sentiment that Bitcoin must be incentive compatible. It is necessary.
It is in only miners immediate interest to make the most profitable block from the available transaction pool. As with so many other things, it is necessary to partially ignore short-term gain for long-term benefit. It is in miners and everybody's long-term interest to have a reliable transaction service. A busy transaction service that confirms lots of transactions per hour will become more profitable as demand increases and more users are prepared to pay for priority. As it is there is currently no way to fully scale because of the transaction bandwidth limit and that is problematic. If all valid transactions must eventually confirm then there must be a way to resolve that problem.
Bitcoin deliberately removes traditional scale by ensuring blocks take ten minutes on average to solve, an ingenious idea and, incentive compatible but, fixed block sizes leaves us with a problem to solve when we want to scale.
Quoted text (1 line)
>If you could find a good solution that would allow you to know if miners
were following your rule or not (and thus ignore it if it doesn't) then you wouldn't even need bitcoin in the first place.
I am confident that the math to verify blocks based on the proposal can be developed (and I think it will not be too complex for a mathematician with the relevant experience), however, I am nowhere near experienced enough with probability and statistical analysis to do it. Yes, if Bitcoin doesn't then it might make another great opportunity for an altcoin but I am not even nearly interested in promoting any altcoins.
If not the proposal that I have put forward, then, hopefully, someone can come up with a better solution. The important thing is that the issues are resolved.
Regards, Damian Williamson
------------------------------ *From:* Rhavar <rhavar@protonmail•com> *Sent:* Saturday, 16 December 2017 3:38 AM *To:* Damian Williamson *Cc:* Bitcoin Protocol Discussion *Subject:* Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Quoted text (1 line)
> I understand that there would be technical issues to resolve in
implementation, but, are there no fundamental errors?
Unfortunately your proposal is really fundamentally broken, on a few levels. I think you might need to do a bit more research into how bitcoin works before coming up with such improvements =)
But just some quick notes:
* Every node has a (potentially) different mempool, you can't use it to decide consensus values like the max block size.
* Increasing the entropy in a block to make it more unpredictable doesn't really make sense.
* Bitcoin should be roughly incentive compatible. Your proposal explicits asks miners to ignore their best interests, and confirm transactions by "priority". What are you going to do if a "malicious" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a consensus requirement? And even if you miners can still sort their mempool by fee, and then order the top 1MB by priority.
If you could find a good solution that would allow you to know if miners were following your rule or not (and thus ignore it if it doesn't) then you wouldn't even need bitcoin in the first place.
-Ryan
-------- Original Message -------- Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks Local Time: December 15, 2017 3:42 AM UTC Time: December 15, 2017 9:42 AM From: bitcoin-dev@lists•linuxfoundation•org To: Bitcoin Protocol Discussion <bitcoin-dev@lists•linuxfoundation•org>
I should not take it that the lack of critical feedback to this revised proposal is a glowing endorsement. I understand that there would be technical issues to resolve in implementation, but, are there no fundamental errors?
I suppose that it if is difficult to determine how long a transaction has been waiting in the pool then, each node could simply keep track of when a transaction was first seen. This may have implications for a verify routine, however, for example, if a node was offline, how should it differentiate how long each transaction was waiting in that case? If a node was restarted daily would it always think that all transactions had been waiting in the pool less than one day If each node keeps the current transaction pool in a file and updates it, as transactions are included in blocks and, as new transactions appear in the pool, then that would go some way to alleviate the issue, apart from entirely new nodes. There should be no reason the contents of a transaction pool files cannot be shared without agreement as to the transaction pool between nodes, just as nodes transmit new transactions freely.
It has been questioned why miners could not cheat. For the question of how many transactions to include in a block, I say it is a standoff and miners will conform to the proposal, not wanting to leave transactions with valid fees standing, and, not wanting to shrink the transaction pool. In any case, if miners shrink the transaction pool then I am not immediately concerned since it provides a more efficient service. For the question of including transactions according to the proposal, I say if it is possible to keep track of how long transactions are waiting in the pool so that they can be included on a probability curve then it is possible to verify that blocks conform to the proposal, since the input is a probability, the output should conform to a probability curve.
If someone has the necessary skill, would anyone be willing to develop the math necessary for the proposal?
Regards, Damian Williamson
------------------------------
*From:* bitcoin-dev-bounces@lists•linuxfoundation•org <bitcoin -dev-bounces@lists•linuxfoundation•org> on behalf of Damian Williamson via bitcoin-dev <bitcoin-dev@lists•linuxfoundation•org> *Sent:* Friday, 8 December 2017 8:01 AM *To:* bitcoin-dev@lists•linuxfoundation•org *Subject:* [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
Good afternoon,
The need for this proposal:
We all must learn to admit that transaction bandwidth is still lurking as a serious issue for the operation, reliability, safety, consumer acceptance, uptake and, for the value of Bitcoin.
I recently sent a payment which was not urgent so; I chose three-day target confirmation from the fee recommendation. That transaction has still not confirmed after now more than six days - even waiting twice as long seems quite reasonable to me. That transaction is a valid transaction; it is not rubbish, junk or, spam. Under the current model with transaction bandwidth limitation, the longer a transaction waits, the less likely it is ever to confirm due to rising transaction numbers and being pushed back by transactions with rising fees.
I argue that no transactions are rubbish or junk, only some zero fee transactions might be spam. Having an ever-increasing number of valid transactions that do not confirm as more new transactions with higher fees are created is the opposite of operating a robust, reliable transaction system.
Business cannot operate with a model where transactions may or may not confirm. Even a business choosing a modest fee has no guarantee that their valid transaction will not be shuffled down by new transactions to the realm of never confirming after it is created. Consumers also will not accept this model as Bitcoin expands. If Bitcoin cannot be a reliable payment system for confirmed transactions then consumers, by and large, will simply not accept the model once they understand. Bitcoin will be a dirty payment system, and this will kill the value of Bitcoin.
Under the current system, a minority of transactions will eventually be the lucky few who have fees high enough to escape being pushed down the list.
Once there are more than x transactions (transaction bandwidth limit) every ten minutes, only those choosing twenty-minute confirmation (2 blocks) will have initially at most a fifty percent chance of ever having their payment confirm. Presently, not even using fee recommendations can ensure a sufficiently high fee is paid to ensure transaction confirmation.
I also argue that the current auction model for limited transaction bandwidth is wrong, is not suitable for a reliable transaction system and, is wrong for Bitcoin. All transactions must confirm in due time. Currently, Bitcoin is not a safe way to send payments.
I do not believe that consumers and business are against paying fees, even high fees. What is required is operational reliability.
This great issue needs to be resolved for the safety and reliability of Bitcoin. The time to resolve issues in commerce is before they become great big issues. The time to resolve this issue is now. We must have the foresight to identify and resolve problems before they trip us over. Simply doubling block sizes every so often is reactionary and is not a reliable permanent solution. I have written a BIP proposal for a technical solution but, need your help to write it up to an acceptable standard to be a full BIP.
I have formatted the following with markdown which is human readable so, I hope nobody minds. I have done as much with this proposal as I feel that I am able so far but continue to take your feedback.
# BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks
## The problem: Everybody wants value. Miners want to maximize revenue from fees (and we presume, to minimize block size). Consumers need transaction reliability and, (we presume) want low fees.
The current transaction bandwidth limit is a limiting factor for both. As the operational safety of transactions is limited, so is consumer confidence as they realize the issue and, accordingly, uptake is limited. Fees are artificially inflated due to bandwidth limitations while failing to provide a full confirmation service for all transactions.
Current fee recommendations provide no satisfaction for transaction reliability and, as Bitcoin scales, this will worsen.
Bitcoin must be a fully scalable and reliable service, providing full transaction confirmation for every valid transaction.
The possibility to send a transaction with a fee lower than one that is acceptable to allow eventual transaction confirmation should be removed from the protocol and also from the user interface.
## Solution summary: Provide each transaction with an individual transaction priority each time before choosing transactions to include in the current block, the priority being a function of the fee paid (on a curve), and the time waiting in the transaction pool (also on a curve) out to n days (n=60 ?). The transaction priority to serve as the likelihood of a transaction being included in the current block, and for determining the order in which transactions are tried to see if they will be included.
Use a target block size. Determine the target block size using; current transaction pool size x ( 1 / (144 x n days ) ) = number of transactions to be included in the current block. Broadcast the next target block size with the current block when it is solved so that nodes know the next target block size for the block that they are building on.
The curves used for the priority of transactions would have to be appropriate. Perhaps a mathematician with experience in probability can develop the right formulae. My thinking is a steep curve. I suppose that the probability of all transactions should probably account for a sufficient number of inclusions that the target block size is met although, it may not always be. As a suggestion, consider including some zero fee transactions to pad, highest BTC value first?
**Explanation of the operation of priority:**
Quoted text (1 line)
> If transaction priority is, for example, a number between one (low) and
one-hundred (high) it can be directly understood as the percentage chance in one-hundred of a transaction being included in the block. Using probability or likelihood infers that there is some function of random. If random (100) < transaction priority then the transaction is included.
Quoted text (1 line)
>To break it down further, if both the fee on a curve value and the time
waiting on a curve value are each a number between one and one-hundred, a rudimentary method may be to simply multiply those two numbers, to find the priority number. For example, a middle fee transaction waiting thirty days (if n = 60 days) may have a value of five for each part (yes, just five, the values are on a curve). When multiplied that will give a priority value of twenty-five, or, a twenty-five percent chance at that moment of being included in the block; it will likely be included in one of the next four blocks, getting more likely each chance. If it is still not included then the value of time waiting will be higher, making for more probability. A very low fee transaction would have a value for the fee of one. It would not be until near sixty-days that the particular low fee transaction has a high likelihood of being included in the block.
I am not concerned with low (or high) transaction fees, the primary reason for addressing the issue is to ensure transactional reliability and scalability while having each transaction confirm in due time.
## Pros: * Maximizes transaction reliability. * Fully scalable. * Maximizes possibility for consumer and business uptake. * Maximizes total fees paid per block without reducing reliability; because of reliability, in time confidence and overall uptake are greater; therefore, more transactions. * Market determines fee paid for transaction priority. * Fee recommendations work all the way out to 30 days or greater. * Provides additional block entropy; greater security since there is less probability of predicting the next block.
## Cons: * Could initially lower total transaction fees per block. * Must be first be programmed.
## Solution operation: This is a simplistic view of the operation. The actual operation will need to be determined in a spec for the programmer.
1. Determine the target block size for the current block. 2. Assign a transaction priority to each transaction in the pool. 3. Select transactions to include in the current block using probability in transaction priority order until the target block size is met. 5. Solve block. 6. Broadcast the next target block size with the current block when it is solved. 7. Block is received. 8. Block verification process. 9. Accept/reject block based on verification result. 10. Repeat.
## Closing comments: It may be possible to verify blocks conform to the proposal by showing that the probability for all transactions included in the block statistically conforms to a probability distribution curve, *if* the individual transaction priority can be recreated. I am not that deep into the mathematics; however, it may also be possible to use a similar method to do this just based on the fee, that statistically, the blocks conform to a fee distribution. Any zero fee transactions would have to be ignored. This solution needs a clever mathematician.
I implore, at the very least, that we use some method that validates full transaction reliability and enables scalability of block sizes. If not this proposal, an alternative.
Regards, Damian Williamson