BloomFilter issue with segwit addresses
7 messages, 13 to 15 Apr 2018. Started by Andreas Schildbach.
Andreas Schildbach (2), Jim Posen (2), Jonas Schnelli, Luke Dashjr, Christian Decker
Anton, a developer on the bitcoinj maiing list, recently made me aware [1] of a compatibility issue between segwit and BIP37 (Bloom Filtering).
The issue affects only P2WPKH and the special case of transactions without change outputs (such as when emptying a wallet). In this case, neither inputs not outputs contain any data elements that would cause a match for the filter. The public key, which would match, goes to the witness but not to the input.
My suggestion was to include an OP_RETURN output with a matching public key in such transactions. Anton confirmed that this workaround is indeed working. But of course it nullifies some of the segwit's size improvements.
I wonder if Bitcoin Core would be willing to extend the BIP37 matching rules such that data elements in the witness are also matched against?
[1] https://groups.google.com/d/msg/bitcoinj/SJpLgjowc1I/V7u2BavvAwAJ
Hi Andreas
Thanks for bringing this up and this seems indeed to be suboptimal.
Quoted text (2 lines)
> I wonder if Bitcoin Core would be willing to extend the BIP37 matching > rules such that data elements in the witness are also matched against?
Bitcoin Core is not an identity that can be „willing to extend“ (or reject) a feature. Someone needs to come up with a proposal (pull request).
Maybe an extension for BIP37 would make sense (*meh*). Just inserting the witness data into the bloom filter seems to be an easy solution (CBloomFilter::IsRelevantAndUpdate())
/jonas
Why not add the outpoints owned by the wallet to the filter and watch for those instead of elements in the input script or witness data?
On Fri, Apr 13, 2018 at 12:12 PM, Jonas Schnelli via bitcoin-dev < bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (23 lines)
> Hi Andreas > > Thanks for bringing this up and this seems indeed to be suboptimal. > > > I wonder if Bitcoin Core would be willing to extend the BIP37 matching > > rules such that data elements in the witness are also matched against? > > Bitcoin Core is not an identity that can be „willing to extend“ (or > reject) a feature. > Someone needs to come up with a proposal (pull request). > > Maybe an extension for BIP37 would make sense (*meh*). > Just inserting the witness data into the bloom filter seems to be an easy > solution (CBloomFilter::IsRelevantAndUpdate()) > > /jonas > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > >
As I understand it, the plan is to deprecated and remove BIP37 entirely once BIP158 is implemented and deployed.
In the meantime, Bitcoin Knots supports the MSG_FILTERED_WITNESS_BLOCK extension to download witness data. (Note that light clients currently have no way to verify the witness data is correct.)
As far as matching goes, why not look for the specific COutPoints? That should work already with standard BIP37.
Luke
On Friday 13 April 2018 3:32:15 PM Andreas Schildbach via bitcoin-dev wrote:
Quoted text (23 lines)
> Anton, a developer on the bitcoinj maiing list, recently made me aware > [1] of a compatibility issue between segwit and BIP37 (Bloom Filtering). > > The issue affects only P2WPKH and the special case of transactions > without change outputs (such as when emptying a wallet). In this case, > neither inputs not outputs contain any data elements that would cause a > match for the filter. The public key, which would match, goes to the > witness but not to the input. > > My suggestion was to include an OP_RETURN output with a matching public > key in such transactions. Anton confirmed that this workaround is indeed > working. But of course it nullifies some of the segwit's size improvements. > > I wonder if Bitcoin Core would be willing to extend the BIP37 matching > rules such that data elements in the witness are also matched against? > > > [1] https://groups.google.com/d/msg/bitcoinj/SJpLgjowc1I/V7u2BavvAwAJ > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Note that this would compound the privacy leak that Jonas Nick used to identify address clusters via the bloom filters in one of his publications. By reducing the false positives when matching you can get very detailed clusters. Then again we know that bloom filters aren't good for privacy anyway, so this might be a non-issue.
On Sat, Apr 14, 2018, 00:17 Jim Posen via bitcoin-dev < bitcoin-dev@lists•linuxfoundation•org> wrote:
Quoted text (34 lines)
> Why not add the outpoints owned by the wallet to the filter and watch for > those instead of elements in the input script or witness data? > > On Fri, Apr 13, 2018 at 12:12 PM, Jonas Schnelli via bitcoin-dev < > bitcoin-dev@lists•linuxfoundation•org> wrote: > >> Hi Andreas >> >> Thanks for bringing this up and this seems indeed to be suboptimal. >> >> > I wonder if Bitcoin Core would be willing to extend the BIP37 matching >> > rules such that data elements in the witness are also matched against? >> >> Bitcoin Core is not an identity that can be „willing to extend“ (or >> reject) a feature. >> Someone needs to come up with a proposal (pull request). >> >> Maybe an extension for BIP37 would make sense (*meh*). >> Just inserting the witness data into the bloom filter seems to be an easy >> solution (CBloomFilter::IsRelevantAndUpdate()) >> >> /jonas >> >> _______________________________________________ >> bitcoin-dev mailing list >> bitcoin-dev@lists•linuxfoundation•org >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >> >> > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >
To Christian's point about privacy, I'll take this opportunity to shamelessly review beg on https://github.com/bitcoin/bitcoin/pull/12254, the PR for BIP 158 implementation (but not 157).
On Sat, Apr 14, 2018 at 9:14 AM, Christian Decker < decker.christian@gmail•com> wrote:
Quoted text (44 lines)
> Note that this would compound the privacy leak that Jonas Nick used to > identify address clusters via the bloom filters in one of his publications. > By reducing the false positives when matching you can get very detailed > clusters. Then again we know that bloom filters aren't good for privacy > anyway, so this might be a non-issue. > > On Sat, Apr 14, 2018, 00:17 Jim Posen via bitcoin-dev <bitcoin-dev@lists. > linuxfoundation.org> wrote: > >> Why not add the outpoints owned by the wallet to the filter and watch for >> those instead of elements in the input script or witness data? >> >> On Fri, Apr 13, 2018 at 12:12 PM, Jonas Schnelli via bitcoin-dev < >> bitcoin-dev@lists•linuxfoundation•org> wrote: >> >>> Hi Andreas >>> >>> Thanks for bringing this up and this seems indeed to be suboptimal. >>> >>> > I wonder if Bitcoin Core would be willing to extend the BIP37 matching >>> > rules such that data elements in the witness are also matched against? >>> >>> Bitcoin Core is not an identity that can be „willing to extend“ (or >>> reject) a feature. >>> Someone needs to come up with a proposal (pull request). >>> >>> Maybe an extension for BIP37 would make sense (*meh*). >>> Just inserting the witness data into the bloom filter seems to be an >>> easy solution (CBloomFilter::IsRelevantAndUpdate()) >>> >>> /jonas >>> >>> _______________________________________________ >>> bitcoin-dev mailing list >>> bitcoin-dev@lists•linuxfoundation•org >>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >>> >>> >> _______________________________________________ >> bitcoin-dev mailing list >> bitcoin-dev@lists•linuxfoundation•org >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >> >
Yes, I guess the quicker filter exhaustion must be the reason why bitcoinj doesn't make use of outpoints in filters for standard transactions. I'll look into if I can change that.
On 04/14/2018 06:14 PM, Christian Decker via bitcoin-dev wrote:
Quoted text (54 lines)
> Note that this would compound the privacy leak that Jonas Nick used to > identify address clusters via the bloom filters in one of his > publications. By reducing the false positives when matching you can get > very detailed clusters. Then again we know that bloom filters aren't > good for privacy anyway, so this might be a non-issue. > > On Sat, Apr 14, 2018, 00:17 Jim Posen via bitcoin-dev > <bitcoin-dev@lists•linuxfoundation•org > <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote: > > Why not add the outpoints owned by the wallet to the filter and > watch for those instead of elements in the input script or witness data? > > On Fri, Apr 13, 2018 at 12:12 PM, Jonas Schnelli via bitcoin-dev > <bitcoin-dev@lists•linuxfoundation•org > <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote: > > Hi Andreas > > Thanks for bringing this up and this seems indeed to be suboptimal. > > > I wonder if Bitcoin Core would be willing to extend the BIP37 matching > > rules such that data elements in the witness are also matched against? > > Bitcoin Core is not an identity that can be „willing to extend“ > (or reject) a feature. > Someone needs to come up with a proposal (pull request). > > Maybe an extension for BIP37 would make sense (*meh*). > Just inserting the witness data into the bloom filter seems to > be an easy solution (CBloomFilter::IsRelevantAndUpdate()) > > /jonas > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > <mailto:bitcoin-dev@lists.linuxfoundation.org> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > <mailto:bitcoin-dev@lists.linuxfoundation.org> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > > > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >