Mailing List Administrivia - GPG, Archives, Breakage, TODO, mirrors, etc
7 messages, 27 Jun to 16 Jul 2015. Started by grarpamp.
grarpamp (3), Jeff Garzik, Andy Schroder, Warren Togami, Peter Todd
On Sun, Jun 21, 2015 at 9:14 PM, Frank Flores <frankf44@gmail•com> wrote:
Quoted text (1 line)
> If you're going to go through the trouble of signing your public emails ...
... then you should also demand that the official archives of your favorite lists preserve them and their verifiability in the supposedly canonical reference "mbox" format that they distribute.
On Wed, Jun 24, 2015 at 7:47 AM, Wladimir J. van der Laan <laanwj@gmail•com> wrote:
Quoted text (1 line)
> Subject: [bitcoin-dev] New GPG signing key for Bitcoin Core binary releases
http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June/009045.html http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June.txt.gz
As you can clearly see, both the HTML archives and the "mbox" archives have corrupted this message, it will not verify. Do not try to say this case is trivial, the problem itself is not trivial, it's gratuitous, and it applies to all matching messages... text, code, binary inline... that's dangerous.
Do not try to say this corruption prevents spam, it does not. Spammers simply subscribe to the list and harvest everything efficiently in realtime... no webcrawling overhead, no stale addresses. Obfuscation is futile.
This misfeature needs to be disabled.
On Fri, Jun 19, 2015 at 12:57 AM, Warren Togami Jr. <wtogami@gmail•com> wrote:
Quoted text (2 lines)
> archives will be exported > and imported into the new list server
On Tue, Jun 23, 2015 at 2:45 PM, Andy Schroder <info@andyschroder•com> wrote:
Quoted text (1 line)
> I'm not sure if anyone has submitted a request for gmane
On Sun, Jun 21, 2015 at 5:35 PM, s7r <s7r@sky-ip•org> wrote:
Quoted text (3 lines)
> Do we have all the archives imported? I run several full nodes and > mirrors for open source projects, if you think it's useful, I can > provide a mirror for the mail list archives.
Yes... these other mirrors, archives, analysis, journalism, and interfaces are useful. However, as it is now, there are no useful authoritative sources for them to seed from... they're all corrupt. And any subscribed realtime sources, though nice, are subject to downtime, administrative and other unrecoverable gaps. Your mirror project is a fine idea, you should demand that the pristine historical sources be made publicly available. Not just for you, but for everyone.
On Wed, Jun 17, 2015, grarpamp wrote:
Quoted text (1 line)
> ...
As before, the current "mbox" archives are broken and not useful...
a) They corrupt message data, messages are unverifiable, another example... http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June/009132.html http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June.txt.gz b) They are missing the minimum set of original headers necessary for fully replyable, threadable, sortable, searchable, context preserving and direct use by users MUA's in their local environment: (Date, From, To, Cc, Subject, Message-ID, In-Reply-To, References) c) They do not include attachments (patches, signatures, images, crypto, data) that are absolutely necessary to the context and archives of all lists. Instead they stupidly throw them away to "web links" which results not only in uselessness of the so called "mbox" version, but in many thousands of needless fetches by archive users and indexers. And hours of wasted work attempting to postprocess them into usable form. Valuable content lost from the "mbox" files this June alone: 418 attachment.html 106 attachment.sig 6 attachment.jpe 4 attachment.png 2 attachment.bin d) There appear to be at least 15 instances of unescaped '^From ' in the "mbox". Regeneration with current mailman may fix. One such case is here: http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2014-January/004245.html http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2014-January.txt.gz
Please fix all the above mentioned issues by providing the full raw archives in regularly updated [gzip/7z] mbox format. The internet thanks you :)
Example, compare the "Downloadable version"s here: http://lists.linuxfoundation.org/pipermail/bitcoin-dev/ https://cpunks.org/pipermail/cypherpunks/ https://cpunks.org/pipermail/cypherpunks/2015-February/006820.html
On Tue, Jun 23, 2015 at 2:45 PM, Andy Schroder <info@andyschroder•com> wrote:
Quoted text (1 line)
> Regarding message footers and the subject prefix
Yes, they're also corruptive and space wasting clutter, for and by the clueless :( Both of them should be turned off.
Generally agreed w/ all this.
To preserve digital signatures now and in the future, and make mbox archives actually useful, a minimum modification policy is needed.
On Sat, Jun 27, 2015 at 3:21 PM, grarpamp <grarpamp@gmail•com> wrote:
Quoted text (114 lines)
> On Sun, Jun 21, 2015 at 9:14 PM, Frank Flores <frankf44@gmail•com> wrote: > > If you're going to go through the trouble of signing your public emails > ... > > ... then you should also demand that the official archives of your > favorite lists preserve them and their verifiability in the supposedly > canonical reference "mbox" format that they distribute. > > On Wed, Jun 24, 2015 at 7:47 AM, Wladimir J. van der Laan > <laanwj@gmail•com> wrote: > > Subject: [bitcoin-dev] New GPG signing key for Bitcoin Core binary > releases > > > http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June/009045.html > http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June.txt.gz > > As you can clearly see, both the HTML archives and the "mbox" > archives have corrupted this message, it will not verify. Do not > try to say this case is trivial, the problem itself is not trivial, > it's gratuitous, and it applies to all matching messages... text, > code, binary inline... that's dangerous. > > Do not try to say this corruption prevents spam, it does not. > Spammers simply subscribe to the list and harvest everything > efficiently in realtime... no webcrawling overhead, no stale > addresses. Obfuscation is futile. > > This misfeature needs to be disabled. > > > > On Fri, Jun 19, 2015 at 12:57 AM, Warren Togami Jr. <wtogami@gmail•com> > wrote: > > archives will be exported > > and imported into the new list server > > On Tue, Jun 23, 2015 at 2:45 PM, Andy Schroder <info@andyschroder•com> > wrote: > > I'm not sure if anyone has submitted a request for gmane > > On Sun, Jun 21, 2015 at 5:35 PM, s7r <s7r@sky-ip•org> wrote: > > Do we have all the archives imported? I run several full nodes and > > mirrors for open source projects, if you think it's useful, I can > > provide a mirror for the mail list archives. > > Yes... these other mirrors, archives, analysis, journalism, and > interfaces are useful. However, as it is now, there are no useful > authoritative sources for them to seed from... they're all corrupt. > And any subscribed realtime sources, though nice, are subject to > downtime, administrative and other unrecoverable gaps. Your mirror > project is a fine idea, you should demand that the pristine historical > sources be made publicly available. Not just for you, but for everyone. > > > > On Wed, Jun 17, 2015, grarpamp wrote: > > ... > > As before, the current "mbox" archives are broken and not useful... > > a) They corrupt message data, messages are unverifiable, another example... > > http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June/009132.html > http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June.txt.gz > b) They are missing the minimum set of original headers necessary > for fully replyable, threadable, sortable, searchable, context > preserving and direct use by users MUA's in their local environment: > (Date, From, To, Cc, Subject, Message-ID, In-Reply-To, References) > c) They do not include attachments (patches, signatures, images, crypto, > data) that are absolutely necessary to the context and archives > of all lists. Instead they stupidly throw them away to "web links" > which results not only in uselessness of the so called "mbox" > version, but in many thousands of needless fetches by archive > users and indexers. And hours of wasted work attempting to > postprocess them into usable form. > Valuable content lost from the "mbox" files this June alone: > 418 attachment.html > 106 attachment.sig > 6 attachment.jpe > 4 attachment.png > 2 attachment.bin > d) There appear to be at least 15 instances of unescaped '^From ' > in the "mbox". Regeneration with current mailman may fix. One such > case is here: > > http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2014-January/004245.html > > http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2014-January.txt.gz > > > > Please fix all the above mentioned issues by providing the full raw > archives in regularly updated [gzip/7z] mbox format. The internet > thanks you :) > > Example, compare the "Downloadable version"s here: > http://lists.linuxfoundation.org/pipermail/bitcoin-dev/ > https://cpunks.org/pipermail/cypherpunks/ > https://cpunks.org/pipermail/cypherpunks/2015-February/006820.html > > > > On Tue, Jun 23, 2015 at 2:45 PM, Andy Schroder <info@andyschroder•com> > wrote: > > Regarding message footers and the subject prefix > > Yes, they're also corruptive and space wasting clutter, for and by > the clueless :( Both of them should be turned off. > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >
Although I'd say your writing style is a bit unfriendly and hard to identify/unclear at the specifics of some of your complaints, I do agree with many of your points.
I really have no idea what format the .txt.gz files are in that are on the current archive. If you unzip them, they are not text tiles.
gmane was able to change the subscription of the previous list to the current list. So, anyone who had a newsreader setup, it should be working again. There is some gap in their archive at the beginning of the archive, as well as last week while the list was transitioned. They requested a link to a single mbox file archive that they can use to restore the list archive in it's entirety. Anyone know how we can get this? The reference to another mailing list below (https://cpunks.org/pipermail/cypherpunks/) has such a link on its archive page. It looks like the same interface so maybe that feature can just be turned on? I'm not sure if it gzips the mbox file on the fly whenever an http request is made or if it has some schedule and may be missing a few of the most recent messages?
Also, regarding the footer, if it can't be changed on a per user basis, I'd vote to get rid of it. Anyone with a reasonably powered brain can go to lists.linuxfoundation.org and find a way to get to the bitcoin-dev mailing list page. The information provided in the footer is also included in the email message headers as well. It's also included in the welcome e-mail when someone subscribes to the list.
Andy Schroder
On 06/27/2015 06:21 PM, grarpamp wrote:
Quoted text (100 lines)
> On Sun, Jun 21, 2015 at 9:14 PM, Frank Flores <frankf44@gmail•com> wrote: >> If you're going to go through the trouble of signing your public emails ... > ... then you should also demand that the official archives of your > favorite lists preserve them and their verifiability in the supposedly > canonical reference "mbox" format that they distribute. > > On Wed, Jun 24, 2015 at 7:47 AM, Wladimir J. van der Laan > <laanwj@gmail•com> wrote: >> Subject: [bitcoin-dev] New GPG signing key for Bitcoin Core binary releases > http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June/009045.html > http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June.txt.gz > > As you can clearly see, both the HTML archives and the "mbox" > archives have corrupted this message, it will not verify. Do not > try to say this case is trivial, the problem itself is not trivial, > it's gratuitous, and it applies to all matching messages... text, > code, binary inline... that's dangerous. > > Do not try to say this corruption prevents spam, it does not. > Spammers simply subscribe to the list and harvest everything > efficiently in realtime... no webcrawling overhead, no stale > addresses. Obfuscation is futile. > > This misfeature needs to be disabled. > > > > On Fri, Jun 19, 2015 at 12:57 AM, Warren Togami Jr. <wtogami@gmail•com> wrote: >> archives will be exported >> and imported into the new list server > On Tue, Jun 23, 2015 at 2:45 PM, Andy Schroder <info@andyschroder•com> wrote: >> I'm not sure if anyone has submitted a request for gmane > On Sun, Jun 21, 2015 at 5:35 PM, s7r <s7r@sky-ip•org> wrote: >> Do we have all the archives imported? I run several full nodes and >> mirrors for open source projects, if you think it's useful, I can >> provide a mirror for the mail list archives. > Yes... these other mirrors, archives, analysis, journalism, and > interfaces are useful. However, as it is now, there are no useful > authoritative sources for them to seed from... they're all corrupt. > And any subscribed realtime sources, though nice, are subject to > downtime, administrative and other unrecoverable gaps. Your mirror > project is a fine idea, you should demand that the pristine historical > sources be made publicly available. Not just for you, but for everyone. > > > > On Wed, Jun 17, 2015, grarpamp wrote: >> ... > As before, the current "mbox" archives are broken and not useful... > > a) They corrupt message data, messages are unverifiable, another example... > http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June/009132.html > http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June.txt.gz > b) They are missing the minimum set of original headers necessary > for fully replyable, threadable, sortable, searchable, context > preserving and direct use by users MUA's in their local environment: > (Date, From, To, Cc, Subject, Message-ID, In-Reply-To, References) > c) They do not include attachments (patches, signatures, images, crypto, > data) that are absolutely necessary to the context and archives > of all lists. Instead they stupidly throw them away to "web links" > which results not only in uselessness of the so called "mbox" > version, but in many thousands of needless fetches by archive > users and indexers. And hours of wasted work attempting to > postprocess them into usable form. > Valuable content lost from the "mbox" files this June alone: > 418 attachment.html > 106 attachment.sig > 6 attachment.jpe > 4 attachment.png > 2 attachment.bin > d) There appear to be at least 15 instances of unescaped '^From ' > in the "mbox". Regeneration with current mailman may fix. One such > case is here: > http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2014-January/004245.html > http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2014-January.txt.gz > > > > Please fix all the above mentioned issues by providing the full raw > archives in regularly updated [gzip/7z] mbox format. The internet > thanks you :) > > Example, compare the "Downloadable version"s here: > http://lists.linuxfoundation.org/pipermail/bitcoin-dev/ > https://cpunks.org/pipermail/cypherpunks/ > https://cpunks.org/pipermail/cypherpunks/2015-February/006820.html > > > > On Tue, Jun 23, 2015 at 2:45 PM, Andy Schroder <info@andyschroder•com> wrote: >> Regarding message footers and the subject prefix > Yes, they're also corruptive and space wasting clutter, for and by > the clueless :( Both of them should be turned off. > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > > >
June 28th, 2015 Mailman's obscure_addresses option is now disabled. It has been pretty useless as a spam mitigation measure anyway.
This is a test message to see if mailman continues to break --clearsign GPG signatures for messages that contain an e-mail address like test@example•com .
On Sat, Jun 27, 2015 at 4:54 PM, Jeff Garzik <jgarzik@gmail•com> wrote:
Quoted text (133 lines)
> Generally agreed w/ all this. > > To preserve digital signatures now and in the future, and make mbox > archives actually useful, a minimum modification policy is needed. > > > > On Sat, Jun 27, 2015 at 3:21 PM, grarpamp <grarpamp@gmail•com> wrote: > >> On Sun, Jun 21, 2015 at 9:14 PM, Frank Flores <frankf44@gmail•com> wrote: >> > If you're going to go through the trouble of signing your public emails >> ... >> >> ... then you should also demand that the official archives of your >> favorite lists preserve them and their verifiability in the supposedly >> canonical reference "mbox" format that they distribute. >> >> On Wed, Jun 24, 2015 at 7:47 AM, Wladimir J. van der Laan >> <laanwj@gmail•com> wrote: >> > Subject: [bitcoin-dev] New GPG signing key for Bitcoin Core binary >> releases >> >> >> http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June/009045.html >> http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June.txt.gz >> >> As you can clearly see, both the HTML archives and the "mbox" >> archives have corrupted this message, it will not verify. Do not >> try to say this case is trivial, the problem itself is not trivial, >> it's gratuitous, and it applies to all matching messages... text, >> code, binary inline... that's dangerous. >> >> Do not try to say this corruption prevents spam, it does not. >> Spammers simply subscribe to the list and harvest everything >> efficiently in realtime... no webcrawling overhead, no stale >> addresses. Obfuscation is futile. >> >> This misfeature needs to be disabled. >> >> >> >> On Fri, Jun 19, 2015 at 12:57 AM, Warren Togami Jr. <wtogami@gmail•com> >> wrote: >> > archives will be exported >> > and imported into the new list server >> >> On Tue, Jun 23, 2015 at 2:45 PM, Andy Schroder <info@andyschroder•com> >> wrote: >> > I'm not sure if anyone has submitted a request for gmane >> >> On Sun, Jun 21, 2015 at 5:35 PM, s7r <s7r@sky-ip•org> wrote: >> > Do we have all the archives imported? I run several full nodes and >> > mirrors for open source projects, if you think it's useful, I can >> > provide a mirror for the mail list archives. >> >> Yes... these other mirrors, archives, analysis, journalism, and >> interfaces are useful. However, as it is now, there are no useful >> authoritative sources for them to seed from... they're all corrupt. >> And any subscribed realtime sources, though nice, are subject to >> downtime, administrative and other unrecoverable gaps. Your mirror >> project is a fine idea, you should demand that the pristine historical >> sources be made publicly available. Not just for you, but for everyone. >> >> >> >> On Wed, Jun 17, 2015, grarpamp wrote: >> > ... >> >> As before, the current "mbox" archives are broken and not useful... >> >> a) They corrupt message data, messages are unverifiable, another >> example... >> >> http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June/009132.html >> >> http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June.txt.gz >> b) They are missing the minimum set of original headers necessary >> for fully replyable, threadable, sortable, searchable, context >> preserving and direct use by users MUA's in their local environment: >> (Date, From, To, Cc, Subject, Message-ID, In-Reply-To, References) >> c) They do not include attachments (patches, signatures, images, crypto, >> data) that are absolutely necessary to the context and archives >> of all lists. Instead they stupidly throw them away to "web links" >> which results not only in uselessness of the so called "mbox" >> version, but in many thousands of needless fetches by archive >> users and indexers. And hours of wasted work attempting to >> postprocess them into usable form. >> Valuable content lost from the "mbox" files this June alone: >> 418 attachment.html >> 106 attachment.sig >> 6 attachment.jpe >> 4 attachment.png >> 2 attachment.bin >> d) There appear to be at least 15 instances of unescaped '^From ' >> in the "mbox". Regeneration with current mailman may fix. One such >> case is here: >> >> http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2014-January/004245.html >> >> http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2014-January.txt.gz >> >> >> >> Please fix all the above mentioned issues by providing the full raw >> archives in regularly updated [gzip/7z] mbox format. The internet >> thanks you :) >> >> Example, compare the "Downloadable version"s here: >> http://lists.linuxfoundation.org/pipermail/bitcoin-dev/ >> https://cpunks.org/pipermail/cypherpunks/ >> https://cpunks.org/pipermail/cypherpunks/2015-February/006820.html >> >> >> >> On Tue, Jun 23, 2015 at 2:45 PM, Andy Schroder <info@andyschroder•com> >> wrote: >> > Regarding message footers and the subject prefix >> >> Yes, they're also corruptive and space wasting clutter, for and by >> the clueless :( Both of them should be turned off. >> _______________________________________________ >> bitcoin-dev mailing list >> bitcoin-dev@lists•linuxfoundation•org >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >> > > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation•org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > >
On Sat, Jun 27, 2015 at 04:54:00PM -0700, Jeff Garzik wrote:
Quoted text (4 lines)
> Generally agreed w/ all this. > > To preserve digital signatures now and in the future, and make mbox > archives actually useful, a minimum modification policy is needed.
ACK
I'd support making the raw archives public for archiving as well.
On Sun, Jun 28, 2015 at 10:27 PM, Warren Togami Jr. <wtogami@gmail•com> wrote:
Quoted text (6 lines)
> Mailman's obscure_addresses option is now disabled. It has been pretty > useless as a spam mitigation measure anyway. > > This is a test message to see if mailman continues to break --clearsign GPG > signatures for messages that contain an e-mail address like > test@example•com.
Still broken, in both the web "archives" and the "mbox" archives. http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June/009252.html http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June.txt.gz
Note also, as outlined with other issues in the OP, the below attachment will not appear in the "mbox" archives, thus becoming lost to mirrors and users rightly attempting to use the "mbox" archives as their canonical source for local seeding or MUA access.
Quoted text (1 line)
> will not appear in the "mbox" archives, thus becoming lost to mirrors and
And the attachment link for users of the web "archive" has stripped off the final blank line that was present in the original attachment. Thus it would fail any detached signatures authors may choose to make.
SHA1 (lost.c) = 4c6284510d73031eac16ddac53363137f39499e0 http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-July/009439.html http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150716/63f50962/attachment-0001.bin